Oct 30 04:41:22.813155 ip-10-0-134-105 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 04:41:22.813170 ip-10-0-134-105 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 04:41:22.813179 ip-10-0-134-105 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 04:41:22.813520 ip-10-0-134-105 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 04:41:33.062612 ip-10-0-134-105 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 04:41:33.062633 ip-10-0-134-105 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d28d25c9a7344d3688b31f9ca1158323 -- Oct 30 04:43:22.240295 ip-10-0-134-105 systemd[1]: Starting Kubernetes Kubelet... Oct 30 04:43:22.700165 ip-10-0-134-105 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 04:43:22.700165 ip-10-0-134-105 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 04:43:22.700165 ip-10-0-134-105 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 04:43:22.700165 ip-10-0-134-105 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 04:43:22.707190 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707100 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:22.707190 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707186 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:22.707190 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707192 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:22.707190 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707195 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:22.707190 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707199 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707203 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707206 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707209 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707212 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707215 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707218 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707220 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707223 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707226 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707228 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707231 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707234 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707237 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707240 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707242 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707244 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707247 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707250 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707252 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:22.707387 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707257 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707260 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707263 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707265 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707268 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707270 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707273 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707283 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707286 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707289 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707291 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707294 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707296 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707299 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707302 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707307 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707312 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707315 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707318 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:22.707854 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707321 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707324 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707326 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707329 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707332 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707334 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707337 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707339 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707342 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707344 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707347 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707350 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707353 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707355 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707358 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707360 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707363 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707368 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707372 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:22.708367 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707375 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707378 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707381 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707385 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707388 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707391 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707394 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707397 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707400 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707403 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707406 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707409 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707411 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707414 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707417 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707420 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707422 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707425 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707428 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:22.708807 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707431 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707433 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707437 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707439 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707442 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707444 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707447 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707450 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707452 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707455 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707457 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707460 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707462 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707465 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707467 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707470 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707473 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707476 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707479 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:22.709274 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707482 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707484 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707489 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707492 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707494 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.707497 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708341 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708349 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708352 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708355 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708358 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708361 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708363 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708367 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708369 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708373 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708376 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708379 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708382 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708384 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:22.709764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708387 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708390 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708393 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708395 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708398 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708400 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708403 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708406 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708408 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708411 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708414 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708417 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708420 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708423 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708426 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708428 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708431 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708434 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708437 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:22.710256 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708440 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708442 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708445 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708448 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708450 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708453 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708455 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708458 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708460 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708463 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708466 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708469 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708471 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708474 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708476 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708479 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708482 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708484 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708487 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708489 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:22.710699 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708492 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708494 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708497 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708500 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708503 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708505 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708508 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708511 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708514 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708517 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708521 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708525 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708527 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708530 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708533 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708535 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708538 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708540 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708543 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:22.711226 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708546 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708548 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708551 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708554 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708557 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708560 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708563 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708566 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708568 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708571 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708573 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708576 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708579 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708581 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708585 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708589 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708593 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708596 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708599 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708602 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:22.711678 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708605 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708608 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708611 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708613 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708616 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708619 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708621 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708624 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708626 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708629 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708631 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708635 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708638 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.708640 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709156 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709163 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709166 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709169 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709172 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709175 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:22.712173 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709178 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709180 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709183 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709186 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709189 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709191 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709194 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709197 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709200 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709204 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709208 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709210 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709213 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709216 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709219 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709222 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709224 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709227 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709230 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:22.712659 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709232 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709235 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709237 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709240 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709243 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709245 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709248 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709251 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709253 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709256 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709259 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709262 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709264 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709267 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709270 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709272 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709275 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709278 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709281 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709284 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:22.713125 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709287 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709290 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709292 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709295 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709297 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709300 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709302 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709305 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709307 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709310 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709312 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709315 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709317 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709320 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709322 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709325 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709328 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709333 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709336 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709338 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:22.713589 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709341 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709344 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709346 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709353 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709356 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709358 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709361 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709363 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709366 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709369 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709371 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709374 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709376 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709379 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709382 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709384 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709387 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709389 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709392 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709394 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:22.714065 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709397 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709399 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709402 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709405 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709408 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709411 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709415 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709418 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709421 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709425 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709428 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709431 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709433 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709436 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709438 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709441 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709444 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709447 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709450 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:22.714535 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709452 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.709455 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710459 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710471 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710479 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710484 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710489 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710492 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710496 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710501 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710504 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710507 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710510 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710514 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710517 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710520 2577 flags.go:64] FLAG: --cgroup-root="" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710523 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710526 2577 flags.go:64] FLAG: --client-ca-file="" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710529 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710532 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710536 2577 flags.go:64] FLAG: --cluster-domain="" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710543 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710547 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710551 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 04:43:22.714988 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710555 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710559 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710562 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710565 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710569 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710572 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710576 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710579 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710583 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710586 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710590 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710593 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710596 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710599 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710603 2577 flags.go:64] FLAG: --enable-server="true" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710605 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710610 2577 flags.go:64] FLAG: --event-burst="100" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710613 2577 flags.go:64] FLAG: --event-qps="50" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710616 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710620 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710623 2577 flags.go:64] FLAG: --eviction-hard="" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710627 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710631 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710634 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710637 2577 flags.go:64] FLAG: --eviction-soft="" Oct 30 04:43:22.715561 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710640 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710643 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710646 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710649 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710652 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710655 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710661 2577 flags.go:64] FLAG: --feature-gates="" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710669 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710672 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710675 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710679 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710682 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710685 2577 flags.go:64] FLAG: --help="false" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710688 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-134-105.ec2.internal" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710693 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710696 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710700 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710703 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710707 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710710 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710713 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710716 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710719 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 04:43:22.716253 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710739 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710744 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710748 2577 flags.go:64] FLAG: --kube-reserved="" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710751 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710754 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710757 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710760 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710763 2577 flags.go:64] FLAG: --lock-file="" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710766 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710769 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710772 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710778 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710781 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710784 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710787 2577 flags.go:64] FLAG: --logging-format="text" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710792 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710795 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710799 2577 flags.go:64] FLAG: --manifest-url="" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710802 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710807 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710810 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710814 2577 flags.go:64] FLAG: --max-pods="110" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710817 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710820 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710824 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 04:43:22.716831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710827 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710830 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710833 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710836 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710844 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710847 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710850 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710853 2577 flags.go:64] FLAG: --pod-cidr="" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710856 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710859 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710862 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710875 2577 flags.go:64] FLAG: --port="10250" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710879 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710882 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-042875b55d28515f0" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710886 2577 flags.go:64] FLAG: --qos-reserved="" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710889 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710892 2577 flags.go:64] FLAG: --register-node="true" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710895 2577 flags.go:64] FLAG: --register-with-taints="" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710899 2577 flags.go:64] FLAG: --registry-burst="10" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710902 2577 flags.go:64] FLAG: --registry-qps="5" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710904 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710908 2577 flags.go:64] FLAG: --reserved-memory="" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710913 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710916 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710919 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710922 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 04:43:22.717525 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710925 2577 flags.go:64] FLAG: --runonce="false" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710928 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710931 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710934 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710937 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710940 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710944 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710947 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710950 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710953 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710956 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710959 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710963 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710966 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710969 2577 flags.go:64] FLAG: --system-cgroups="" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710972 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710978 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710981 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710984 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710987 2577 flags.go:64] FLAG: --tls-min-version="" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710990 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710993 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710996 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.710999 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.711002 2577 flags.go:64] FLAG: --v="2" Oct 30 04:43:22.718153 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.711017 2577 flags.go:64] FLAG: --version="false" Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.711021 2577 flags.go:64] FLAG: --vmodule="" Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.711027 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.711031 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.711990 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.711996 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.711999 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712002 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712018 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712022 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712027 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712032 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712036 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712039 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712043 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712046 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712049 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712052 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712055 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712057 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:22.718764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712060 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712063 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712065 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712068 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712071 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712073 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712076 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712078 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712081 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712084 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712087 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712090 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712093 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712095 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712099 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712103 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712106 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712108 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712111 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:22.719249 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712114 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712116 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712119 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712121 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712124 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712127 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712129 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712132 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712135 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712138 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712140 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712143 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712146 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712148 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712151 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712155 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712159 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712162 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712164 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:22.719694 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712167 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712170 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712172 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712175 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712178 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712180 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712183 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712186 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712190 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712193 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712196 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712199 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712201 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712204 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712206 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712209 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712212 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712215 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712217 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712220 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:22.720206 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712222 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712225 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712229 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712232 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712234 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712237 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712240 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712242 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712245 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712248 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712250 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712253 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712256 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712258 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712261 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712264 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712266 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712269 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712271 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712274 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:22.720685 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712278 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712281 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712284 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712287 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712289 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712292 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712295 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712297 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712300 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712303 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712305 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712308 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.712337 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712373 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712377 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712381 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:22.721177 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712383 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712386 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712389 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712392 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712395 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712397 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712400 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712402 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712405 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712408 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712410 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712413 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712415 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712418 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712421 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712423 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712427 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712430 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712432 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:22.721553 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712436 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712440 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712443 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712445 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712448 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712450 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712453 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712455 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712458 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712460 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712463 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712466 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712468 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712471 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712474 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712477 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712479 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712482 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712484 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:22.722000 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712487 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712490 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712492 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712495 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712497 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712500 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712503 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712507 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712509 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712512 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712515 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712518 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712521 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712524 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712526 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712530 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712532 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712535 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712537 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712540 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:22.722463 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712543 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712546 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712548 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712551 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712554 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712558 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712561 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712564 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712567 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712570 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712573 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712575 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712578 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712581 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712583 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712586 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712588 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712591 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712594 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712598 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:22.722940 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712600 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712603 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712607 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712610 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712613 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712615 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712618 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712620 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712623 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712626 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712628 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712632 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712635 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712638 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712640 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712643 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712645 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712648 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712651 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712653 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:22.723417 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712656 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712659 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712662 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712664 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.712667 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.720182 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.720196 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.720213 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.720218 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720285 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720290 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720295 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720297 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720300 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720303 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720306 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720309 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720311 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720314 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720317 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:22.723878 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720320 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720323 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720325 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720328 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720331 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720333 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720342 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720345 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720348 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720351 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720354 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720357 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720366 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720369 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720372 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720375 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720377 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720380 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720385 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720389 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:22.724361 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720392 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720395 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720398 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720400 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720403 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720406 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720408 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720411 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720414 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720417 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720420 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720422 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720425 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720428 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720430 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720432 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720435 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720438 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720440 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720443 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:22.725642 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720446 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720449 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720451 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720454 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720456 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720466 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720470 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720474 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720477 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720480 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720482 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720485 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720487 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720490 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720492 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720495 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720498 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720501 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720503 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:22.726164 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720506 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720508 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720511 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720513 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720516 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720518 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720521 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720523 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720526 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720529 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720532 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720534 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720537 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720539 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720542 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720545 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720547 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720550 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720552 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:22.726604 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720561 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720564 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720566 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720568 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720571 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720574 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720576 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720579 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720581 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720584 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720586 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720589 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720591 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720594 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720597 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720600 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:22.727167 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720602 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.720630 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720818 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720823 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720827 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720830 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720833 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720836 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720838 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720841 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720844 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720847 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720849 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720852 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720855 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:22.727538 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720857 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720860 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720868 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720871 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720873 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720876 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720878 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720881 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720883 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720886 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720889 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720891 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720894 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720896 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720898 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720901 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720904 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720906 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720909 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720912 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:22.727924 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720914 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720917 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720919 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720922 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720925 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720928 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720930 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720933 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720935 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720938 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720940 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720943 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720946 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720948 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720951 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720959 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720962 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720964 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720967 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:22.728748 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720970 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720972 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720975 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720978 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720980 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720983 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720985 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720988 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720990 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720993 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720996 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.720998 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721001 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721003 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721023 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721025 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721032 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721035 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721038 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721040 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:22.729473 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721043 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721045 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721048 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721052 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721056 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721059 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721062 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721065 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721067 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721078 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721081 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721083 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721086 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721089 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721091 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721094 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721097 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721099 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721102 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:22.729949 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721104 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721107 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721111 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721115 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721117 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721120 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721123 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721126 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721128 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721133 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721135 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721138 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721140 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721143 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721162 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.721187 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 04:43:22.730431 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721235 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721239 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721242 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721245 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721247 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721250 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721252 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721261 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721264 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721267 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721272 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721276 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721279 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721282 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721284 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721287 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721290 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721292 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721295 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:43:22.730814 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721297 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721300 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721303 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721305 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721308 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721311 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721313 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721317 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721320 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721324 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721334 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721338 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721341 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721343 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721346 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721349 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721352 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721355 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721357 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:43:22.731406 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721360 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721362 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721370 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721374 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721378 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721380 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721383 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721386 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721388 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721391 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721393 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721397 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721399 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721402 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721404 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721407 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721410 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721412 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721415 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:43:22.731859 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721417 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721421 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721424 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721426 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721429 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721432 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721434 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721437 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721440 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721442 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721445 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721447 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721450 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721453 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721455 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721458 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721467 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721472 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721474 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721477 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:43:22.732332 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721480 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721482 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721485 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721488 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721490 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721493 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721495 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721498 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721501 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721503 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721506 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721508 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721511 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721514 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721517 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721520 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721522 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721525 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721528 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:43:22.732793 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721531 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721534 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721536 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721539 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721541 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721548 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721551 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721554 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721556 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:22.721559 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.721617 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.724002 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.725050 2577 server.go:1014] "Starting client certificate rotation" Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.725182 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.725223 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.729513 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 04:43:22.733244 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.733034 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 04:43:22.743100 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.743082 2577 log.go:25] "Validated CRI v1 runtime API" Oct 30 04:43:22.746680 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.746661 2577 log.go:25] "Validated CRI v1 image API" Oct 30 04:43:22.747479 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.747454 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 04:43:22.753616 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.753596 2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80d5f8cf-d243-42b4-b064-f8c778cdff95:/dev/nvme0n1p3 a8d79c02-9e52-4928-80f5-5de84d21161a:/dev/nvme0n1p4] Oct 30 04:43:22.753688 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.753613 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 04:43:22.760051 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.759930 2577 manager.go:217] Machine: {Timestamp:2026-10-30 04:43:22.758510058 +0000 UTC m=+0.379538364 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101014 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2207bdb34566e6b0457703ffad80de SystemUUID:ec2207bd-b345-66e6-b045-7703ffad80de BootID:d28d25c9-a734-4d36-88b3-1f9ca1158323 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cc:4c:04:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cc:4c:04:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:98:67:8d:cc:7a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 04:43:22.760051 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.760050 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 04:43:22.760159 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.760125 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 04:43:22.761537 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.761497 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 04:43:22.761681 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.761540 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 04:43:22.762443 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.762433 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 04:43:22.762476 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.762445 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 04:43:22.762476 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.762464 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 04:43:22.762532 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.762476 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 04:43:22.763532 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.763521 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 04:43:22.767649 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.767637 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 04:43:22.767762 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.767754 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 04:43:22.768574 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.768565 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 30 04:43:22.768606 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.768578 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 04:43:22.768606 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.768588 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 04:43:22.768606 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.768596 2577 kubelet.go:405] "Adding apiserver pod source" Oct 30 04:43:22.768606 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.768604 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 04:43:22.768977 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.768963 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 04:43:22.771313 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.771298 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 04:43:22.776951 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.776939 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 04:43:22.777887 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.777875 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 04:43:22.777933 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.777904 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 04:43:22.778084 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.778073 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 04:43:22.778119 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.778088 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 04:43:22.778119 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.778095 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 04:43:22.778119 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.778101 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 04:43:22.778119 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.778107 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 04:43:22.778119 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.778113 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 04:43:22.778830 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.778819 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 04:43:22.778870 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.778831 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 04:43:22.779059 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.779040 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 04:43:22.779122 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.779074 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 04:43:22.779122 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.779091 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 04:43:22.779122 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.779108 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 04:43:22.780519 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.780501 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 04:43:22.780595 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.780528 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 04:43:22.785970 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.785954 2577 server.go:1263] "Started kubelet" Oct 30 04:43:22.786145 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.786086 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 04:43:22.786235 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.786166 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 04:43:22.786422 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.786042 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 04:43:22.786776 ip-10-0-134-105 systemd[1]: Started Kubernetes Kubelet. Oct 30 04:43:22.787636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.787616 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 04:43:22.788102 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.788088 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 30 04:43:22.791636 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:22.791617 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 04:43:22.796237 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.796219 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 04:43:22.796323 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.796244 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 04:43:22.796371 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:22.796321 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-105.ec2.internal\" not found" Oct 30 04:43:22.797215 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.797191 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 04:43:22.797215 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.797193 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 04:43:22.797364 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.797226 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 04:43:22.797364 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.797199 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 04:43:22.797364 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.797262 2577 factory.go:55] Registering systemd factory Oct 30 04:43:22.797364 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.797273 2577 factory.go:223] Registration of the systemd container factory successfully Oct 30 04:43:22.797364 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.797322 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 30 04:43:22.797364 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.797330 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 30 04:43:22.797688 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.797469 2577 factory.go:153] Registering CRI-O factory Oct 30 04:43:22.797688 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.797479 2577 factory.go:223] Registration of the crio container factory successfully Oct 30 04:43:22.797688 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.797496 2577 factory.go:103] Registering Raw factory Oct 30 04:43:22.797688 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.797510 2577 manager.go:1199] Started watching for new ooms in manager Oct 30 04:43:22.798566 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.798549 2577 manager.go:319] Starting recovery of all containers Oct 30 04:43:22.806404 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.806384 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 04:43:22.806524 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.806429 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pjv9p" Oct 30 04:43:22.808661 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.807797 2577 manager.go:324] Recovery completed Oct 30 04:43:22.811900 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.811879 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pjv9p" Oct 30 04:43:22.812491 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.812453 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-105.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 04:43:22.812584 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:22.812546 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 04:43:22.813876 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:22.812458 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-105.ec2.internal.18e3350c1e2ff059 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-105.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-105.ec2.internal,},FirstTimestamp:2026-10-30 04:43:22.785927257 +0000 UTC m=+0.406955563,LastTimestamp:2026-10-30 04:43:22.785927257 +0000 UTC m=+0.406955563,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-105.ec2.internal,}" Oct 30 04:43:22.814597 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.814584 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:22.816426 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.816409 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:22.816499 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.816441 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:22.816499 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.816455 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:22.816981 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.816969 2577 cpu_manager.go:226] "Starting" policy="none" Oct 30 04:43:22.817061 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.816980 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 04:43:22.817061 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.816994 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 04:43:22.817061 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.816973 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:22.818415 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.818398 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:22.818482 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.818423 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:22.818482 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.818434 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:22.818930 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.818913 2577 policy_none.go:50] "Start" Oct 30 04:43:22.818991 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.818938 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 04:43:22.818991 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.818951 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 04:43:22.820927 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.820913 2577 policy_none.go:44] "Start" Oct 30 04:43:22.827979 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:22.827959 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 04:43:22.828091 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:22.828026 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 04:43:22.828091 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:22.828036 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 04:43:22.850949 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.850922 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.854615 2577 manager.go:341] "Starting Device Plugin manager" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:22.854645 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.854658 2577 server.go:87] "Starting device plugin registration server" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.854859 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.854877 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.855083 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.855147 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.855155 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:22.855615 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:22.855655 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-105.ec2.internal\" not found" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.955557 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.956503 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.956529 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.956543 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:22.958780 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.956584 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-105.ec2.internal" Oct 30 04:43:22.967457 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.967433 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 04:43:22.968587 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.968569 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 04:43:22.968676 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.968595 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 04:43:22.968676 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.968611 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 04:43:22.968676 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:22.968641 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 04:43:22.976782 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.976766 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-105.ec2.internal" Oct 30 04:43:22.976856 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:22.976786 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-105.ec2.internal\": node \"ip-10-0-134-105.ec2.internal\" not found" Oct 30 04:43:22.986719 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:22.986703 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:23.018333 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:23.018309 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-105.ec2.internal\" not found" Oct 30 04:43:23.069682 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.069657 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal"] Oct 30 04:43:23.069778 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.069735 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:23.070730 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.070717 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:23.070804 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.070744 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:23.070804 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.070771 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:23.073035 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.073024 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:23.073179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.073165 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.073231 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.073199 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:23.073792 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.073778 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:23.073875 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.073803 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:23.073875 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.073813 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:23.073875 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.073778 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:23.074045 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.073885 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:23.074045 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.073905 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:23.075921 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.075904 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.075978 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.075932 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:43:23.076734 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.076719 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:43:23.076825 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.076747 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:43:23.076825 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.076761 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:43:23.095447 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:23.095430 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-105.ec2.internal\" not found" node="ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.097403 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:23.097386 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-105.ec2.internal\" not found" node="ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.099036 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.099002 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06a30a33e1bd990d029556ff52059236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" (UID: \"06a30a33e1bd990d029556ff52059236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.099121 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.099082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06a30a33e1bd990d029556ff52059236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" (UID: \"06a30a33e1bd990d029556ff52059236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.099170 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.099134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/624730a70c2b0e3a128e1c5af9dbe16d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-105.ec2.internal\" (UID: \"624730a70c2b0e3a128e1c5af9dbe16d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.118360 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:23.118340 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-105.ec2.internal\" not found" Oct 30 04:43:23.200051 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.200025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06a30a33e1bd990d029556ff52059236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" (UID: \"06a30a33e1bd990d029556ff52059236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.200170 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.200060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06a30a33e1bd990d029556ff52059236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" (UID: \"06a30a33e1bd990d029556ff52059236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.200170 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.200084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/624730a70c2b0e3a128e1c5af9dbe16d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-105.ec2.internal\" (UID: \"624730a70c2b0e3a128e1c5af9dbe16d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.200170 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.200113 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06a30a33e1bd990d029556ff52059236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" (UID: \"06a30a33e1bd990d029556ff52059236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.200170 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.200117 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06a30a33e1bd990d029556ff52059236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" (UID: \"06a30a33e1bd990d029556ff52059236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.200170 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.200130 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/624730a70c2b0e3a128e1c5af9dbe16d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-105.ec2.internal\" (UID: \"624730a70c2b0e3a128e1c5af9dbe16d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.219042 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:23.218976 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-105.ec2.internal\" not found" Oct 30 04:43:23.319802 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:23.319761 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-105.ec2.internal\" not found" Oct 30 04:43:23.400214 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.400183 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.400319 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:23.400196 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" Oct 30 04:43:23.420357 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:23.420214 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-105.ec2.internal\" not found" Oct 30 04:43:23.520738 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:23.520706 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-105.ec2.internal\" not found" Oct 30 04:43:24.935645 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:24.935624 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-105.ec2.internal\" not found" Oct 30 04:43:25.036156 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:25.036121 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-105.ec2.internal\" not found" Oct 30 04:43:25.039277 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:25.039259 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 04:43:25.039421 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:25.039404 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 04:43:25.132800 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:25.132733 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 04:38:24 +0000 UTC" deadline="2028-06-14 13:01:54.161187713 +0000 UTC" Oct 30 04:43:25.132800 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:25.132762 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14240h18m29.028430071s" Oct 30 04:43:25.136882 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:25.136865 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-105.ec2.internal\" not found" Oct 30 04:43:25.214891 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:25.214862 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:25.283282 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:25.283254 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624730a70c2b0e3a128e1c5af9dbe16d.slice/crio-3f15a33a8778626d8aabfcd1354c427a12b4cb5f27c67fae30f68f00d5837ba9 WatchSource:0}: Error finding container 3f15a33a8778626d8aabfcd1354c427a12b4cb5f27c67fae30f68f00d5837ba9: Status 404 returned error can't find the container with id 3f15a33a8778626d8aabfcd1354c427a12b4cb5f27c67fae30f68f00d5837ba9 Oct 30 04:43:25.283632 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:25.283617 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a30a33e1bd990d029556ff52059236.slice/crio-beff413d95e8fa1eb732fbc49c2ffa9fc0d72f7c2ef2f9570f882e9849fe81c7 WatchSource:0}: Error finding container beff413d95e8fa1eb732fbc49c2ffa9fc0d72f7c2ef2f9570f882e9849fe81c7: Status 404 returned error can't find the container with id beff413d95e8fa1eb732fbc49c2ffa9fc0d72f7c2ef2f9570f882e9849fe81c7 Oct 30 04:43:25.289098 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:25.289083 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:43:25.295301 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:25.295282 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:25.311482 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:25.311463 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:43:25.334898 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:25.334878 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 04:43:25.335922 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:25.335910 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" Oct 30 04:43:25.356101 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:25.356080 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 04:43:25.470838 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:25.470774 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:26.084396 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.084369 2577 apiserver.go:52] "Watching apiserver" Oct 30 04:43:26.112538 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.112509 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 04:43:26.127539 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.127516 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 04:43:26.127871 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.127849 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-crggn","openshift-multus/network-metrics-daemon-kcmnj","openshift-network-diagnostics/network-check-target-bblhz","openshift-network-operator/iptables-alerter-vnplr","kube-system/global-pull-secret-syncer-hcgqp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal","openshift-multus/multus-49x8w","kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal","openshift-dns/node-resolver-xhn9f","openshift-ovn-kubernetes/ovnkube-node-tk9mv","openshift-cluster-node-tuning-operator/tuned-dlpz8","openshift-image-registry/node-ca-dl5rx","kube-system/konnectivity-agent-cqv2d"] Oct 30 04:43:26.130969 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.130621 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:43:26.132929 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.132903 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:43:26.135405 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.135138 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:26.135405 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.135217 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:26.137629 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.137330 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:43:26.139702 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.139684 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:26.139798 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.139755 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:26.142205 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.141836 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.144305 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.144232 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:43:26.146954 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.146926 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.147313 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.147094 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 04:43:26.147313 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.147101 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 04:43:26.147313 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.147166 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tg42c\"" Oct 30 04:43:26.147707 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.147689 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 04:43:26.147974 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.147813 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45bnw\"" Oct 30 04:43:26.147974 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.147946 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 04:43:26.148207 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.148190 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 04:43:26.148969 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.148948 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:26.149060 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.149020 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:26.151893 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.151874 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.155472 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.154429 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.156760 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.156739 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.162068 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.162046 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 04:43:26.162160 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.162094 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t7h75" Oct 30 04:43:26.212667 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.212646 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 04:43:26.224265 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-ovnkube-script-lib\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.224386 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224312 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-modprobe-d\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.224386 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224343 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:26.224509 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224425 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7115245a-cbc9-4180-9709-2a6e590bb170-agent-certs\") pod \"konnectivity-agent-cqv2d\" (UID: \"7115245a-cbc9-4180-9709-2a6e590bb170\") " pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:43:26.224509 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7115245a-cbc9-4180-9709-2a6e590bb170-konnectivity-ca\") pod \"konnectivity-agent-cqv2d\" (UID: \"7115245a-cbc9-4180-9709-2a6e590bb170\") " pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:43:26.224509 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-cni-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.224509 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224503 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-cni-bin\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.224704 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-ovnkube-config\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.224704 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-kubelet-config\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:26.224704 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224574 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-cni-netd\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.224704 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224593 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chwz8\" (UniqueName: \"kubernetes.io/projected/09846780-4fb0-474a-bd15-9b272dc00772-kube-api-access-chwz8\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.224704 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-var-lib-kubelet\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.224704 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224636 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c1d23d7-96c3-4406-9354-860812b8d2bb-serviceca\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:43:26.224704 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224660 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-device-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.224704 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-system-cni-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.224704 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-multus-certs\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224726 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs5xk\" (UniqueName: \"kubernetes.io/projected/26dd286e-348a-4378-be3d-9687ba409fc4-kube-api-access-hs5xk\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-slash\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-var-lib-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224776 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-kubelet\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-kubelet-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224807 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-run\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c1d23d7-96c3-4406-9354-860812b8d2bb-host\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224846 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-hostroot\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysctl-d\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-lib-modules\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-registration-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-system-cni-dir\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-systemd\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.224990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-host\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.225113 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225009 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-cnibin\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225029 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225054 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-conf-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-kubelet\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgbkv\" (UniqueName: \"kubernetes.io/projected/67f6fc33-45c9-4579-ab82-039cb764b155-kube-api-access-tgbkv\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-systemd\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225240 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-sys\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-iptables-alerter-script\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59632489-f9ac-44e1-ae2a-7a119c0ece48-hosts-file\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-socket-dir-parent\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225375 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26dd286e-348a-4378-be3d-9687ba409fc4-multus-daemon-config\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225410 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225455 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-tuned\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zsw\" (UniqueName: \"kubernetes.io/projected/926ec2b0-7229-4297-8af9-bcd48ceab12a-kube-api-access-p9zsw\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twfl6\" (UniqueName: \"kubernetes.io/projected/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-kube-api-access-twfl6\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-dbus\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:26.225873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225587 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-run-ovn-kubernetes\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225625 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-k8s-cni-cncf-io\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-etc-selinux\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225688 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-sys-fs\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-cni-binary-copy\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-systemd-units\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-env-overrides\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt74j\" (UniqueName: \"kubernetes.io/projected/a0780f4f-55b4-49c6-80a1-4026cc98f858-kube-api-access-mt74j\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225913 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rb56\" (UniqueName: \"kubernetes.io/projected/59632489-f9ac-44e1-ae2a-7a119c0ece48-kube-api-access-5rb56\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.225943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-cni-bin\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/09846780-4fb0-474a-bd15-9b272dc00772-ovn-node-metrics-cert\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysctl-conf\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226076 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/926ec2b0-7229-4297-8af9-bcd48ceab12a-tmp\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvbs\" (UniqueName: \"kubernetes.io/projected/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-kube-api-access-xgvbs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:26.226636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59632489-f9ac-44e1-ae2a-7a119c0ece48-tmp-dir\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226217 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsv6f\" (UniqueName: \"kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f\") pod \"network-check-target-bblhz\" (UID: \"1c23f69d-0026-4202-884b-38a3d4bbe87d\") " pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-etc-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-log-socket\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226272 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysconfig\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-kubernetes\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226314 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-host-slash\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-cnibin\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226423 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-netns\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-node-log\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226474 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-cni-multus\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-827bt\" (UniqueName: \"kubernetes.io/projected/9c1d23d7-96c3-4406-9354-860812b8d2bb-kube-api-access-827bt\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-socket-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26dd286e-348a-4378-be3d-9687ba409fc4-cni-binary-copy\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-etc-kubernetes\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.227374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-run-netns\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.228142 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.226621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-ovn\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.247134 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247109 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 04:43:26.247271 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247146 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 04:43:26.247271 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247262 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 04:43:26.247392 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247341 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhd5l\"" Oct 30 04:43:26.247452 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247401 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bpk6m\"" Oct 30 04:43:26.247452 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247417 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 04:43:26.247452 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247434 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:26.247599 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247441 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:43:26.247599 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247476 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 04:43:26.247599 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247483 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jzfzk\"" Oct 30 04:43:26.247599 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247486 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tdqqv\"" Oct 30 04:43:26.247599 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247523 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:26.247599 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247532 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q24mq\"" Oct 30 04:43:26.247599 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247590 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 04:43:26.247925 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247613 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 04:43:26.247925 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247625 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 04:43:26.247925 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247627 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 04:43:26.247925 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247737 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 04:43:26.247925 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247753 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 04:43:26.247925 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.247845 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 04:43:26.250600 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.250580 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 04:43:26.250705 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.250589 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 04:43:26.250705 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.250610 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 04:43:26.250705 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.250614 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 04:43:26.260723 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.260686 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 04:43:26.260723 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.260720 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g88vf\"" Oct 30 04:43:26.260880 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.260733 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bngnh\"" Oct 30 04:43:26.260880 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.260741 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 04:43:26.260880 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.260828 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 04:43:26.285968 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.285947 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:43:26.288442 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.288398 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" event={"ID":"06a30a33e1bd990d029556ff52059236","Type":"ContainerStarted","Data":"beff413d95e8fa1eb732fbc49c2ffa9fc0d72f7c2ef2f9570f882e9849fe81c7"} Oct 30 04:43:26.289767 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.289741 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" event={"ID":"624730a70c2b0e3a128e1c5af9dbe16d","Type":"ContainerStarted","Data":"3f15a33a8778626d8aabfcd1354c427a12b4cb5f27c67fae30f68f00d5837ba9"} Oct 30 04:43:26.327341 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327300 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26dd286e-348a-4378-be3d-9687ba409fc4-cni-binary-copy\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.327498 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-etc-kubernetes\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.327498 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327397 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-run-netns\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.327498 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327415 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-ovn\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.327498 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327464 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-ovnkube-script-lib\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.327681 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-modprobe-d\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.327681 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:26.327681 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7115245a-cbc9-4180-9709-2a6e590bb170-agent-certs\") pod \"konnectivity-agent-cqv2d\" (UID: \"7115245a-cbc9-4180-9709-2a6e590bb170\") " pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:43:26.327681 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7115245a-cbc9-4180-9709-2a6e590bb170-konnectivity-ca\") pod \"konnectivity-agent-cqv2d\" (UID: \"7115245a-cbc9-4180-9709-2a6e590bb170\") " pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:43:26.327681 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-cni-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.327681 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-cni-bin\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.327681 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327618 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-ovnkube-config\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.327681 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-kubelet-config\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:26.327681 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-cni-netd\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.327681 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwz8\" (UniqueName: \"kubernetes.io/projected/09846780-4fb0-474a-bd15-9b272dc00772-kube-api-access-chwz8\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-var-lib-kubelet\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c1d23d7-96c3-4406-9354-860812b8d2bb-serviceca\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-device-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-system-cni-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-multus-certs\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5xk\" (UniqueName: \"kubernetes.io/projected/26dd286e-348a-4378-be3d-9687ba409fc4-kube-api-access-hs5xk\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327838 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-slash\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-var-lib-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-kubelet\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-kubelet-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-run\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.327985 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c1d23d7-96c3-4406-9354-860812b8d2bb-host\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-hostroot\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysctl-d\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.328071 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-lib-modules\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-registration-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-system-cni-dir\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-systemd\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-host\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-cnibin\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-conf-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-kubelet\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbkv\" (UniqueName: \"kubernetes.io/projected/67f6fc33-45c9-4579-ab82-039cb764b155-kube-api-access-tgbkv\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-systemd\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-sys\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-iptables-alerter-script\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59632489-f9ac-44e1-ae2a-7a119c0ece48-hosts-file\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328410 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-socket-dir-parent\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328438 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26dd286e-348a-4378-be3d-9687ba409fc4-multus-daemon-config\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328470 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.328686 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328475 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-slash\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-tuned\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zsw\" (UniqueName: \"kubernetes.io/projected/926ec2b0-7229-4297-8af9-bcd48ceab12a-kube-api-access-p9zsw\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twfl6\" (UniqueName: \"kubernetes.io/projected/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-kube-api-access-twfl6\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-dbus\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-run-ovn-kubernetes\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-k8s-cni-cncf-io\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-etc-selinux\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-sys-fs\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-cni-binary-copy\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328846 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-systemd-units\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-env-overrides\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt74j\" (UniqueName: \"kubernetes.io/projected/a0780f4f-55b4-49c6-80a1-4026cc98f858-kube-api-access-mt74j\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rb56\" (UniqueName: \"kubernetes.io/projected/59632489-f9ac-44e1-ae2a-7a119c0ece48-kube-api-access-5rb56\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.328982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-cni-bin\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.329131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/09846780-4fb0-474a-bd15-9b272dc00772-ovn-node-metrics-cert\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysctl-conf\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/926ec2b0-7229-4297-8af9-bcd48ceab12a-tmp\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329131 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvbs\" (UniqueName: \"kubernetes.io/projected/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-kube-api-access-xgvbs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329160 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26dd286e-348a-4378-be3d-9687ba409fc4-cni-binary-copy\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59632489-f9ac-44e1-ae2a-7a119c0ece48-tmp-dir\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv6f\" (UniqueName: \"kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f\") pod \"network-check-target-bblhz\" (UID: \"1c23f69d-0026-4202-884b-38a3d4bbe87d\") " pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329273 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-etc-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-log-socket\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysconfig\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329406 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-kubernetes\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-host-slash\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-cnibin\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329492 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59632489-f9ac-44e1-ae2a-7a119c0ece48-tmp-dir\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329509 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-netns\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-node-log\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.329609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329578 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-var-lib-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.330050 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329646 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-cni-multus\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.330050 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-827bt\" (UniqueName: \"kubernetes.io/projected/9c1d23d7-96c3-4406-9354-860812b8d2bb-kube-api-access-827bt\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:43:26.330050 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-socket-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.330050 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.329920 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-socket-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.330050 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.330003 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-etc-kubernetes\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.330185 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.330055 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-run-netns\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.330185 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.330094 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-kubelet\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.330185 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.330096 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-ovn\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.330185 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.330151 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-kubelet-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.330316 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.330299 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-run\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.330348 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.330331 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-multus-certs\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.330400 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.330349 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-host-slash\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:43:26.330400 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.330383 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.330400 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.330387 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c1d23d7-96c3-4406-9354-860812b8d2bb-host\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:43:26.330481 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.330445 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-cni-netd\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.330481 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.330441 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-node-log\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.330481 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.330458 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-var-lib-kubelet\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.331116 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.331081 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysctl-d\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.331487 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.331464 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysconfig\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.331606 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.331585 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-ovnkube-script-lib\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.331646 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.331608 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.331722 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.331707 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-kubernetes\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.331884 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.331868 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-cni-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.331914 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.331897 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7115245a-cbc9-4180-9709-2a6e590bb170-konnectivity-ca\") pod \"konnectivity-agent-cqv2d\" (UID: \"7115245a-cbc9-4180-9709-2a6e590bb170\") " pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:43:26.331949 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.331933 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-cni-bin\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.332002 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.331984 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 04:43:26.332194 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.332179 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-cni-multus\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.333460 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.332458 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-ovnkube-config\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.333460 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.332552 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c1d23d7-96c3-4406-9354-860812b8d2bb-serviceca\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:43:26.333460 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.332556 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-netns\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.333460 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.332599 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-device-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.333460 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.332888 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.333806 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.333535 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-kubelet\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.333806 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.333575 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysctl-conf\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.333806 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.333594 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-modprobe-d\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.333806 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.333709 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-hostroot\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.333806 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.333772 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-cnibin\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.333806 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.333789 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-sys-fs\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.334058 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.333865 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:26.334058 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.333934 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret podName:0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:26.833914648 +0000 UTC m=+3.140592481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret") pod "global-pull-secret-syncer-hcgqp" (UID: "0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c") : object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:26.334058 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.333969 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-dbus\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:26.334058 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.334056 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:26.334234 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.334115 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs podName:9f09ff3d-66bc-46bc-b404-9a45acd58e1d nodeName:}" failed. No retries permitted until 2026-10-30 04:43:26.83409288 +0000 UTC m=+3.140770702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs") pod "network-metrics-daemon-kcmnj" (UID: "9f09ff3d-66bc-46bc-b404-9a45acd58e1d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:26.334295 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.334264 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-run-ovn-kubernetes\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.334372 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.334342 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-system-cni-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.334529 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.334408 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-log-socket\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.334529 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.334457 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-kubelet-config\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:26.334979 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.334947 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-cni-binary-copy\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.335130 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.335099 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.335219 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.335170 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-etc-selinux\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.335219 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.335177 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-systemd-units\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.335219 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.335205 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-cni-bin\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.335428 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.335239 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-k8s-cni-cncf-io\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.335585 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.335565 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-env-overrides\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.335905 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.335736 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-etc-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.336146 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336081 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-socket-dir-parent\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.336266 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336144 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-lib-modules\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.336266 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336248 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-cnibin\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.336421 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-sys\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.336421 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336330 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-systemd\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.336421 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336345 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-host\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.336421 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336345 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26dd286e-348a-4378-be3d-9687ba409fc4-multus-daemon-config\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.336635 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336441 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59632489-f9ac-44e1-ae2a-7a119c0ece48-hosts-file\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:43:26.336635 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336451 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-conf-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.336635 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336467 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-system-cni-dir\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.336635 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336508 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-registration-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.336635 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336542 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-systemd\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.336972 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336943 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.337053 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.336956 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-iptables-alerter-script\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:43:26.338590 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.338569 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/09846780-4fb0-474a-bd15-9b272dc00772-ovn-node-metrics-cert\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.338714 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.338687 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7115245a-cbc9-4180-9709-2a6e590bb170-agent-certs\") pod \"konnectivity-agent-cqv2d\" (UID: \"7115245a-cbc9-4180-9709-2a6e590bb170\") " pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:43:26.338784 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.338569 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-tuned\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.340119 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.340098 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/926ec2b0-7229-4297-8af9-bcd48ceab12a-tmp\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.350715 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.350688 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twfl6\" (UniqueName: \"kubernetes.io/projected/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-kube-api-access-twfl6\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:43:26.355629 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.355267 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 04:43:26.355629 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.355291 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 04:43:26.355629 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.355304 2577 projected.go:196] Error preparing data for projected volume kube-api-access-hsv6f for pod openshift-network-diagnostics/network-check-target-bblhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:26.355629 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.355377 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f podName:1c23f69d-0026-4202-884b-38a3d4bbe87d nodeName:}" failed. No retries permitted until 2026-10-30 04:43:26.855348576 +0000 UTC m=+3.162026400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv6f" (UniqueName: "kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f") pod "network-check-target-bblhz" (UID: "1c23f69d-0026-4202-884b-38a3d4bbe87d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:26.359835 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.359812 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbkv\" (UniqueName: \"kubernetes.io/projected/67f6fc33-45c9-4579-ab82-039cb764b155-kube-api-access-tgbkv\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.359928 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.359908 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rb56\" (UniqueName: \"kubernetes.io/projected/59632489-f9ac-44e1-ae2a-7a119c0ece48-kube-api-access-5rb56\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:43:26.360008 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.359989 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt74j\" (UniqueName: \"kubernetes.io/projected/a0780f4f-55b4-49c6-80a1-4026cc98f858-kube-api-access-mt74j\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.360057 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.360009 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwz8\" (UniqueName: \"kubernetes.io/projected/09846780-4fb0-474a-bd15-9b272dc00772-kube-api-access-chwz8\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.360112 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.360070 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zsw\" (UniqueName: \"kubernetes.io/projected/926ec2b0-7229-4297-8af9-bcd48ceab12a-kube-api-access-p9zsw\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.360482 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.360437 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-827bt\" (UniqueName: \"kubernetes.io/projected/9c1d23d7-96c3-4406-9354-860812b8d2bb-kube-api-access-827bt\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:43:26.362032 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.362013 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5xk\" (UniqueName: \"kubernetes.io/projected/26dd286e-348a-4378-be3d-9687ba409fc4-kube-api-access-hs5xk\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.362959 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.362942 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvbs\" (UniqueName: \"kubernetes.io/projected/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-kube-api-access-xgvbs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:26.444128 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.444094 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:43:26.451231 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.451204 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:43:26.459764 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:26.459729 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59632489_f9ac_44e1_ae2a_7a119c0ece48.slice/crio-d6fc85b3354aa0e1abff0bdf7a15c82db297a48cf3d3f04848ba68a164859593 WatchSource:0}: Error finding container d6fc85b3354aa0e1abff0bdf7a15c82db297a48cf3d3f04848ba68a164859593: Status 404 returned error can't find the container with id d6fc85b3354aa0e1abff0bdf7a15c82db297a48cf3d3f04848ba68a164859593 Oct 30 04:43:26.460297 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:26.460266 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1d23d7_96c3_4406_9354_860812b8d2bb.slice/crio-607d87808f6a84531e3056456f575abf85f0232df5f0bf4bf2fd221099254bb8 WatchSource:0}: Error finding container 607d87808f6a84531e3056456f575abf85f0232df5f0bf4bf2fd221099254bb8: Status 404 returned error can't find the container with id 607d87808f6a84531e3056456f575abf85f0232df5f0bf4bf2fd221099254bb8 Oct 30 04:43:26.461821 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.461799 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:43:26.474155 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.474132 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:43:26.476295 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.476270 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:43:26.477064 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:26.477037 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27bfe4b2_2f4e_46ec_be37_830a9f5c20f5.slice/crio-21151e3e52f1a5214af61f0bdff5bd7d88ec8013b5800cb2d503debd9411a1bf WatchSource:0}: Error finding container 21151e3e52f1a5214af61f0bdff5bd7d88ec8013b5800cb2d503debd9411a1bf: Status 404 returned error can't find the container with id 21151e3e52f1a5214af61f0bdff5bd7d88ec8013b5800cb2d503debd9411a1bf Oct 30 04:43:26.483848 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.482927 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-49x8w" Oct 30 04:43:26.483848 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:26.483232 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0780f4f_55b4_49c6_80a1_4026cc98f858.slice/crio-01904cd114e8b175c949ee66fb71fa5825b062d6badf76ccc7454c0a6b6caf8b WatchSource:0}: Error finding container 01904cd114e8b175c949ee66fb71fa5825b062d6badf76ccc7454c0a6b6caf8b: Status 404 returned error can't find the container with id 01904cd114e8b175c949ee66fb71fa5825b062d6badf76ccc7454c0a6b6caf8b Oct 30 04:43:26.484900 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:26.484877 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7115245a_cbc9_4180_9709_2a6e590bb170.slice/crio-11757ef7a495bf3d5004d99cceeffdb9a0372397be64ad23063b602881d7afed WatchSource:0}: Error finding container 11757ef7a495bf3d5004d99cceeffdb9a0372397be64ad23063b602881d7afed: Status 404 returned error can't find the container with id 11757ef7a495bf3d5004d99cceeffdb9a0372397be64ad23063b602881d7afed Oct 30 04:43:26.491712 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:26.491688 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26dd286e_348a_4378_be3d_9687ba409fc4.slice/crio-797a00f752240d1f11f68df576f507f691f0d0077e1efa40a17313303abcf7a8 WatchSource:0}: Error finding container 797a00f752240d1f11f68df576f507f691f0d0077e1efa40a17313303abcf7a8: Status 404 returned error can't find the container with id 797a00f752240d1f11f68df576f507f691f0d0077e1efa40a17313303abcf7a8 Oct 30 04:43:26.510137 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.510115 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:43:26.512307 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.512283 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:43:26.523523 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:26.523497 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f6fc33_45c9_4579_ab82_039cb764b155.slice/crio-874dbe75e9771ce3c6b53301f52922ee6650dd91dfa8d697b6494b974bef508f WatchSource:0}: Error finding container 874dbe75e9771ce3c6b53301f52922ee6650dd91dfa8d697b6494b974bef508f: Status 404 returned error can't find the container with id 874dbe75e9771ce3c6b53301f52922ee6650dd91dfa8d697b6494b974bef508f Oct 30 04:43:26.524221 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:26.524199 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926ec2b0_7229_4297_8af9_bcd48ceab12a.slice/crio-5243f483cf03c70b783346c21a888e68837b775480784ee18e7d3aff0616e69c WatchSource:0}: Error finding container 5243f483cf03c70b783346c21a888e68837b775480784ee18e7d3aff0616e69c: Status 404 returned error can't find the container with id 5243f483cf03c70b783346c21a888e68837b775480784ee18e7d3aff0616e69c Oct 30 04:43:26.524399 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.524380 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:26.534730 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:43:26.534702 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09846780_4fb0_474a_bd15_9b272dc00772.slice/crio-98f9342100e860dc012653575b82ea804606200ee5eedbc2b7cd9f3f44e5a9f7 WatchSource:0}: Error finding container 98f9342100e860dc012653575b82ea804606200ee5eedbc2b7cd9f3f44e5a9f7: Status 404 returned error can't find the container with id 98f9342100e860dc012653575b82ea804606200ee5eedbc2b7cd9f3f44e5a9f7 Oct 30 04:43:26.935456 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.934989 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv6f\" (UniqueName: \"kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f\") pod \"network-check-target-bblhz\" (UID: \"1c23f69d-0026-4202-884b-38a3d4bbe87d\") " pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:26.935456 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.935028 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:26.935456 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:26.935074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:26.935456 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.935148 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:26.935456 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.935150 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 04:43:26.935456 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.935173 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 04:43:26.935456 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.935183 2577 projected.go:196] Error preparing data for projected volume kube-api-access-hsv6f for pod openshift-network-diagnostics/network-check-target-bblhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:26.935456 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.935195 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs podName:9f09ff3d-66bc-46bc-b404-9a45acd58e1d nodeName:}" failed. No retries permitted until 2026-10-30 04:43:27.935182177 +0000 UTC m=+4.241860000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs") pod "network-metrics-daemon-kcmnj" (UID: "9f09ff3d-66bc-46bc-b404-9a45acd58e1d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:26.935456 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.935222 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f podName:1c23f69d-0026-4202-884b-38a3d4bbe87d nodeName:}" failed. No retries permitted until 2026-10-30 04:43:27.935210317 +0000 UTC m=+4.241888140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv6f" (UniqueName: "kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f") pod "network-check-target-bblhz" (UID: "1c23f69d-0026-4202-884b-38a3d4bbe87d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:26.935456 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.935292 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:26.935456 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:26.935377 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret podName:0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:27.935330246 +0000 UTC m=+4.242008074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret") pod "global-pull-secret-syncer-hcgqp" (UID: "0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c") : object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:27.165521 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.165400 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 04:38:26 +0000 UTC" deadline="2028-06-27 23:38:24.051427442 +0000 UTC" Oct 30 04:43:27.165521 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.165460 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14562h54m56.88597147s" Oct 30 04:43:27.289599 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.289343 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:27.290165 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:27.290131 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:27.305305 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.304302 2577 generic.go:356] "Generic (PLEG): container finished" podID="06a30a33e1bd990d029556ff52059236" containerID="533f78e171659dc9892081b34f0e02f39a7c2f944825b4283cd641a2f833924e" exitCode=0 Oct 30 04:43:27.305305 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.304411 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" event={"ID":"06a30a33e1bd990d029556ff52059236","Type":"ContainerDied","Data":"533f78e171659dc9892081b34f0e02f39a7c2f944825b4283cd641a2f833924e"} Oct 30 04:43:27.325100 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.325073 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" event={"ID":"624730a70c2b0e3a128e1c5af9dbe16d","Type":"ContainerStarted","Data":"31680165c4989cc3d966a511d10b3dd8a93ddd4b3b16a44686e168248c7e204c"} Oct 30 04:43:27.339780 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.339748 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" event={"ID":"926ec2b0-7229-4297-8af9-bcd48ceab12a","Type":"ContainerStarted","Data":"5243f483cf03c70b783346c21a888e68837b775480784ee18e7d3aff0616e69c"} Oct 30 04:43:27.345754 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.345683 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crggn" event={"ID":"67f6fc33-45c9-4579-ab82-039cb764b155","Type":"ContainerStarted","Data":"874dbe75e9771ce3c6b53301f52922ee6650dd91dfa8d697b6494b974bef508f"} Oct 30 04:43:27.369408 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.369376 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-49x8w" event={"ID":"26dd286e-348a-4378-be3d-9687ba409fc4","Type":"ContainerStarted","Data":"797a00f752240d1f11f68df576f507f691f0d0077e1efa40a17313303abcf7a8"} Oct 30 04:43:27.386159 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.386070 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" event={"ID":"a0780f4f-55b4-49c6-80a1-4026cc98f858","Type":"ContainerStarted","Data":"01904cd114e8b175c949ee66fb71fa5825b062d6badf76ccc7454c0a6b6caf8b"} Oct 30 04:43:27.405308 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.405257 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vnplr" event={"ID":"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5","Type":"ContainerStarted","Data":"21151e3e52f1a5214af61f0bdff5bd7d88ec8013b5800cb2d503debd9411a1bf"} Oct 30 04:43:27.413689 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.413656 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dl5rx" event={"ID":"9c1d23d7-96c3-4406-9354-860812b8d2bb","Type":"ContainerStarted","Data":"607d87808f6a84531e3056456f575abf85f0232df5f0bf4bf2fd221099254bb8"} Oct 30 04:43:27.421700 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.421628 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" event={"ID":"09846780-4fb0-474a-bd15-9b272dc00772","Type":"ContainerStarted","Data":"98f9342100e860dc012653575b82ea804606200ee5eedbc2b7cd9f3f44e5a9f7"} Oct 30 04:43:27.443430 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.443319 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cqv2d" event={"ID":"7115245a-cbc9-4180-9709-2a6e590bb170","Type":"ContainerStarted","Data":"11757ef7a495bf3d5004d99cceeffdb9a0372397be64ad23063b602881d7afed"} Oct 30 04:43:27.456078 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.455994 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xhn9f" event={"ID":"59632489-f9ac-44e1-ae2a-7a119c0ece48","Type":"ContainerStarted","Data":"d6fc85b3354aa0e1abff0bdf7a15c82db297a48cf3d3f04848ba68a164859593"} Oct 30 04:43:27.942622 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.942582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:27.942793 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.942655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:27.942793 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:27.942692 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv6f\" (UniqueName: \"kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f\") pod \"network-check-target-bblhz\" (UID: \"1c23f69d-0026-4202-884b-38a3d4bbe87d\") " pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:27.942912 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:27.942874 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 04:43:27.942912 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:27.942893 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 04:43:27.942912 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:27.942906 2577 projected.go:196] Error preparing data for projected volume kube-api-access-hsv6f for pod openshift-network-diagnostics/network-check-target-bblhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:27.943044 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:27.942970 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f podName:1c23f69d-0026-4202-884b-38a3d4bbe87d nodeName:}" failed. No retries permitted until 2026-10-30 04:43:29.942951874 +0000 UTC m=+6.249629700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv6f" (UniqueName: "kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f") pod "network-check-target-bblhz" (UID: "1c23f69d-0026-4202-884b-38a3d4bbe87d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:27.943044 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:27.943041 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:27.943165 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:27.943076 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret podName:0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:29.943063987 +0000 UTC m=+6.249742012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret") pod "global-pull-secret-syncer-hcgqp" (UID: "0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c") : object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:27.943165 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:27.943130 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:27.943165 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:27.943159 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs podName:9f09ff3d-66bc-46bc-b404-9a45acd58e1d nodeName:}" failed. No retries permitted until 2026-10-30 04:43:29.943148594 +0000 UTC m=+6.249826419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs") pod "network-metrics-daemon-kcmnj" (UID: "9f09ff3d-66bc-46bc-b404-9a45acd58e1d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:28.166399 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:28.166319 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 04:38:26 +0000 UTC" deadline="2028-03-28 21:18:19.825599542 +0000 UTC" Oct 30 04:43:28.166399 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:28.166396 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12376h34m51.659209014s" Oct 30 04:43:28.285785 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:28.285711 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:28.285950 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:28.285852 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:28.286254 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:28.286232 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:28.286372 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:28.286339 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:28.470446 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:28.470411 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" event={"ID":"06a30a33e1bd990d029556ff52059236","Type":"ContainerStarted","Data":"502a31a50bed9b0ecfbdfd1e39e1b75e7be15f3cd5e6ee783c46d991950ce6e5"} Oct 30 04:43:28.510634 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:28.510575 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" podStartSLOduration=3.510557698 podStartE2EDuration="3.510557698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:27.381724741 +0000 UTC m=+3.688402583" watchObservedRunningTime="2026-10-30 04:43:28.510557698 +0000 UTC m=+4.817235532" Oct 30 04:43:29.283840 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:29.283806 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:29.284303 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:29.283937 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:29.960855 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:29.960825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:29.961006 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:29.960886 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:29.961006 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:29.960911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv6f\" (UniqueName: \"kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f\") pod \"network-check-target-bblhz\" (UID: \"1c23f69d-0026-4202-884b-38a3d4bbe87d\") " pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:29.961006 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:29.960957 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:29.961234 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:29.961020 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 04:43:29.961234 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:29.961032 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 04:43:29.961234 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:29.961034 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret podName:0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:33.961012876 +0000 UTC m=+10.267690702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret") pod "global-pull-secret-syncer-hcgqp" (UID: "0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c") : object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:29.961234 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:29.961042 2577 projected.go:196] Error preparing data for projected volume kube-api-access-hsv6f for pod openshift-network-diagnostics/network-check-target-bblhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:29.961234 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:29.961077 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f podName:1c23f69d-0026-4202-884b-38a3d4bbe87d nodeName:}" failed. No retries permitted until 2026-10-30 04:43:33.961067047 +0000 UTC m=+10.267744858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv6f" (UniqueName: "kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f") pod "network-check-target-bblhz" (UID: "1c23f69d-0026-4202-884b-38a3d4bbe87d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:29.961234 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:29.961150 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:29.961234 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:29.961186 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs podName:9f09ff3d-66bc-46bc-b404-9a45acd58e1d nodeName:}" failed. No retries permitted until 2026-10-30 04:43:33.961174668 +0000 UTC m=+10.267852479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs") pod "network-metrics-daemon-kcmnj" (UID: "9f09ff3d-66bc-46bc-b404-9a45acd58e1d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:30.286168 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:30.286097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:30.286618 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:30.286232 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:30.286618 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:30.286281 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:30.286618 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:30.286323 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:30.506380 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:30.506032 2577 generic.go:356] "Generic (PLEG): container finished" podID="67f6fc33-45c9-4579-ab82-039cb764b155" containerID="431f31410288dea05606ae57fd0b3a8a9ee3b42826d691e36d734f38e10746b9" exitCode=0 Oct 30 04:43:30.506380 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:30.506140 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crggn" event={"ID":"67f6fc33-45c9-4579-ab82-039cb764b155","Type":"ContainerDied","Data":"431f31410288dea05606ae57fd0b3a8a9ee3b42826d691e36d734f38e10746b9"} Oct 30 04:43:30.515296 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:30.514779 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cqv2d" event={"ID":"7115245a-cbc9-4180-9709-2a6e590bb170","Type":"ContainerStarted","Data":"bbecbf8cc761e4892d8160634dadac9ab291e050d04a54093c817a6791b35c36"} Oct 30 04:43:30.577170 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:30.577013 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" podStartSLOduration=5.576995183 podStartE2EDuration="5.576995183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:28.511181203 +0000 UTC m=+4.817859036" watchObservedRunningTime="2026-10-30 04:43:30.576995183 +0000 UTC m=+6.883673017" Oct 30 04:43:30.602138 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:30.601047 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cqv2d" podStartSLOduration=3.257955265 podStartE2EDuration="6.601027247s" totalImagesPullingTime="3.343071982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:30.600909612 +0000 UTC m=+6.907587473" watchObservedRunningTime="2026-10-30 04:43:30.601027247 +0000 UTC m=+6.907705082" Oct 30 04:43:31.283874 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:31.283798 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:31.284058 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:31.283922 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:31.521039 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:31.520939 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dl5rx" event={"ID":"9c1d23d7-96c3-4406-9354-860812b8d2bb","Type":"ContainerStarted","Data":"be4564ddbbf8748fd4bf3a951a0ca047853b15685dfcf3469f0b5d17a19ef10c"} Oct 30 04:43:31.689542 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:31.689506 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:43:31.690329 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:31.690302 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:43:31.728592 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:31.728327 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dl5rx" podStartSLOduration=3.430157109 podStartE2EDuration="7.728306769s" totalImagesPullingTime="4.29814966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:31.560546098 +0000 UTC m=+7.867223932" watchObservedRunningTime="2026-10-30 04:43:31.728306769 +0000 UTC m=+8.034984599" Oct 30 04:43:32.284545 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:32.284493 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:32.284721 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:32.284630 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:32.285045 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:32.285021 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:32.285152 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:32.285130 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:32.532413 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:32.532383 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" event={"ID":"a0780f4f-55b4-49c6-80a1-4026cc98f858","Type":"ContainerStarted","Data":"6d85cb5359e2bd14b70e4db3b13f1a99e6e592e560b9581f39f470bc346b7201"} Oct 30 04:43:33.283506 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:33.283470 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:33.283690 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:33.283584 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:33.445151 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:33.445100 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 04:43:33.536442 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:33.536371 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 04:43:33.542403 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:33.540450 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" event={"ID":"a0780f4f-55b4-49c6-80a1-4026cc98f858","Type":"ContainerStarted","Data":"915bf8cf073e9f4ea32df4c7fb8a25b3bd0380455d88e889af7418fc98b0a367"} Oct 30 04:43:33.995054 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:33.994999 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:33.995219 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:33.995113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:33.995219 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:33.995160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv6f\" (UniqueName: \"kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f\") pod \"network-check-target-bblhz\" (UID: \"1c23f69d-0026-4202-884b-38a3d4bbe87d\") " pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:33.995345 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:33.995317 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 04:43:33.995345 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:33.995339 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 04:43:33.995491 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:33.995351 2577 projected.go:196] Error preparing data for projected volume kube-api-access-hsv6f for pod openshift-network-diagnostics/network-check-target-bblhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:33.995491 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:33.995429 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f podName:1c23f69d-0026-4202-884b-38a3d4bbe87d nodeName:}" failed. No retries permitted until 2026-10-30 04:43:41.995410575 +0000 UTC m=+18.302088404 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv6f" (UniqueName: "kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f") pod "network-check-target-bblhz" (UID: "1c23f69d-0026-4202-884b-38a3d4bbe87d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:33.995859 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:33.995838 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:33.995922 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:33.995900 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret podName:0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:41.99588216 +0000 UTC m=+18.302559985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret") pod "global-pull-secret-syncer-hcgqp" (UID: "0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c") : object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:33.995977 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:33.995961 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:33.996024 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:33.995994 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs podName:9f09ff3d-66bc-46bc-b404-9a45acd58e1d nodeName:}" failed. No retries permitted until 2026-10-30 04:43:41.995982956 +0000 UTC m=+18.302660782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs") pod "network-metrics-daemon-kcmnj" (UID: "9f09ff3d-66bc-46bc-b404-9a45acd58e1d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:34.174743 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:34.174051 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T04:43:33.445121879Z","UUID":"0a0e2d98-04ba-49f0-99c4-9a0c6a46a7a3","Handler":null,"Name":"","Endpoint":""} Oct 30 04:43:34.182374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:34.179049 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 04:43:34.182374 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:34.179081 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 04:43:34.284542 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:34.284478 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:34.285452 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:34.284653 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:34.285452 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:34.284704 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:34.285452 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:34.285412 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:34.541551 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:34.541518 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-49x8w" event={"ID":"26dd286e-348a-4378-be3d-9687ba409fc4","Type":"ContainerStarted","Data":"20c0abdb7e2a871603f6cdca4b83e2e8ff7ea0fc7946adfb0649fe73233bd05a"} Oct 30 04:43:34.545598 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:34.545570 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xhn9f" event={"ID":"59632489-f9ac-44e1-ae2a-7a119c0ece48","Type":"ContainerStarted","Data":"1022fb883627419e3c199c9a0066d4b8b809f57f89b21a778c5d708ec45ca1e3"} Oct 30 04:43:34.584882 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:34.584648 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-49x8w" podStartSLOduration=3.388306534 podStartE2EDuration="10.584631457s" totalImagesPullingTime="7.196324923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:34.584622505 +0000 UTC m=+10.891300335" watchObservedRunningTime="2026-10-30 04:43:34.584631457 +0000 UTC m=+10.891309291" Oct 30 04:43:34.633465 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:34.629821 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xhn9f" podStartSLOduration=3.003968357 podStartE2EDuration="10.629802193s" totalImagesPullingTime="7.625833836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:34.624397767 +0000 UTC m=+10.931075621" watchObservedRunningTime="2026-10-30 04:43:34.629802193 +0000 UTC m=+10.936480047" Oct 30 04:43:35.284196 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:35.284119 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:35.284327 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:35.284244 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:35.549921 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:35.549889 2577 generic.go:356] "Generic (PLEG): container finished" podID="67f6fc33-45c9-4579-ab82-039cb764b155" containerID="d1a1c49484081fb21fb97726f14a69ad09cca79f7447b726cb38c45ff6a51a01" exitCode=0 Oct 30 04:43:35.550395 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:35.549991 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crggn" event={"ID":"67f6fc33-45c9-4579-ab82-039cb764b155","Type":"ContainerDied","Data":"d1a1c49484081fb21fb97726f14a69ad09cca79f7447b726cb38c45ff6a51a01"} Oct 30 04:43:35.552681 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:35.552642 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" event={"ID":"a0780f4f-55b4-49c6-80a1-4026cc98f858","Type":"ContainerStarted","Data":"8fd3aee702457907b1d33041226d7967f60bc66ba8c155fcca3258cdaa7d4fce"} Oct 30 04:43:35.647955 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:35.647903 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" podStartSLOduration=3.986274628 podStartE2EDuration="11.647883535s" totalImagesPullingTime="7.661608907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:35.647139109 +0000 UTC m=+11.953816944" watchObservedRunningTime="2026-10-30 04:43:35.647883535 +0000 UTC m=+11.954561372" Oct 30 04:43:36.283301 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:36.283233 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:36.283450 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:36.283378 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:36.283721 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:36.283704 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:36.283857 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:36.283827 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:36.556350 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:36.556311 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" event={"ID":"926ec2b0-7229-4297-8af9-bcd48ceab12a","Type":"ContainerStarted","Data":"c164822a9e017450c723374640d93e71dce3d945286c9615462f721e32978aa0"} Oct 30 04:43:36.558625 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:36.558593 2577 generic.go:356] "Generic (PLEG): container finished" podID="67f6fc33-45c9-4579-ab82-039cb764b155" containerID="6f8a17ae5d7ff5663fc2f15bc3879aba530080784a1d329bd36dfc3eb820f259" exitCode=0 Oct 30 04:43:36.558742 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:36.558636 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crggn" event={"ID":"67f6fc33-45c9-4579-ab82-039cb764b155","Type":"ContainerDied","Data":"6f8a17ae5d7ff5663fc2f15bc3879aba530080784a1d329bd36dfc3eb820f259"} Oct 30 04:43:36.560171 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:36.560149 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vnplr" event={"ID":"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5","Type":"ContainerStarted","Data":"4e92d05d178e8731d5af8a3b29bc5e9eb0f8ffcb33c9c10f298c4bddcb338c4d"} Oct 30 04:43:36.596430 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:36.596383 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" podStartSLOduration=3.11034495 podStartE2EDuration="12.596353376s" totalImagesPullingTime="9.486008426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:36.595879554 +0000 UTC m=+12.902557388" watchObservedRunningTime="2026-10-30 04:43:36.596353376 +0000 UTC m=+12.903031208" Oct 30 04:43:36.684519 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:36.684466 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vnplr" podStartSLOduration=5.010163394 podStartE2EDuration="12.684454421s" totalImagesPullingTime="7.674291027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:36.684242345 +0000 UTC m=+12.990920175" watchObservedRunningTime="2026-10-30 04:43:36.684454421 +0000 UTC m=+12.991132253" Oct 30 04:43:37.284151 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:37.284114 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:37.284316 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:37.284238 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:37.564591 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:37.564557 2577 generic.go:356] "Generic (PLEG): container finished" podID="67f6fc33-45c9-4579-ab82-039cb764b155" containerID="81cb4ecfbb1f01f1c805f54d6ebeab984ed9b5822aac2e869eab44a5ac44cd5d" exitCode=0 Oct 30 04:43:37.565137 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:37.564648 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crggn" event={"ID":"67f6fc33-45c9-4579-ab82-039cb764b155","Type":"ContainerDied","Data":"81cb4ecfbb1f01f1c805f54d6ebeab984ed9b5822aac2e869eab44a5ac44cd5d"} Oct 30 04:43:38.285903 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:38.285869 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:38.286097 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:38.286000 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:38.286410 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:38.286388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:38.286517 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:38.286500 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:39.283609 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:39.283410 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:39.284024 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:39.283716 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:39.573385 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:39.573337 2577 generic.go:356] "Generic (PLEG): container finished" podID="67f6fc33-45c9-4579-ab82-039cb764b155" containerID="1b68be4bb51252e2404159f7b94ba9bd7046458ff4b8829a687945fa8dbe17f6" exitCode=0 Oct 30 04:43:39.573549 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:39.573394 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crggn" event={"ID":"67f6fc33-45c9-4579-ab82-039cb764b155","Type":"ContainerDied","Data":"1b68be4bb51252e2404159f7b94ba9bd7046458ff4b8829a687945fa8dbe17f6"} Oct 30 04:43:40.284209 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:40.284175 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:40.284583 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:40.284312 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:40.284583 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:40.284368 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:40.284583 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:40.284446 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:40.577876 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:40.577845 2577 generic.go:356] "Generic (PLEG): container finished" podID="67f6fc33-45c9-4579-ab82-039cb764b155" containerID="5cce97456747db203262d61423489adaf000d8a8679d68a4fb14c9c1bdbf1cd1" exitCode=0 Oct 30 04:43:40.578010 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:40.577892 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crggn" event={"ID":"67f6fc33-45c9-4579-ab82-039cb764b155","Type":"ContainerDied","Data":"5cce97456747db203262d61423489adaf000d8a8679d68a4fb14c9c1bdbf1cd1"} Oct 30 04:43:41.284219 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:41.284052 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:41.284717 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:41.284308 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:41.583487 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:41.583449 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crggn" event={"ID":"67f6fc33-45c9-4579-ab82-039cb764b155","Type":"ContainerStarted","Data":"abf0b7006efb081c7c8b40575037f459cfdf0ec5f6475c1324ceb307a831de6b"} Oct 30 04:43:41.586074 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:41.586055 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 04:43:41.586383 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:41.586351 2577 generic.go:356] "Generic (PLEG): container finished" podID="09846780-4fb0-474a-bd15-9b272dc00772" containerID="8556a6fb16db8931e5624c1a6967a28fa45e22650716d351be70af2a2374ab55" exitCode=1 Oct 30 04:43:41.586431 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:41.586381 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" event={"ID":"09846780-4fb0-474a-bd15-9b272dc00772","Type":"ContainerStarted","Data":"8a98e41dc372ed7c384f542ab1becb80106f441ee53a358d0e3a1ac497ba23cf"} Oct 30 04:43:41.586463 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:41.586435 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" event={"ID":"09846780-4fb0-474a-bd15-9b272dc00772","Type":"ContainerStarted","Data":"2e95cb0be43ed8c6559edf7c3ddf9abef1079f25b5873e0aa57ce8caac9a0f48"} Oct 30 04:43:41.586492 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:41.586466 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" event={"ID":"09846780-4fb0-474a-bd15-9b272dc00772","Type":"ContainerStarted","Data":"0efd7fd6335b20604dd3e03644cc544aae33bac3e28a9930ffd813c9a6aef24b"} Oct 30 04:43:41.586532 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:41.586493 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" event={"ID":"09846780-4fb0-474a-bd15-9b272dc00772","Type":"ContainerStarted","Data":"59fb964e97c4b7c5c9781f89addb9a0f5c7e288af67a2aac0cc93606ef2559d8"} Oct 30 04:43:41.586532 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:41.586521 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" event={"ID":"09846780-4fb0-474a-bd15-9b272dc00772","Type":"ContainerDied","Data":"8556a6fb16db8931e5624c1a6967a28fa45e22650716d351be70af2a2374ab55"} Oct 30 04:43:41.586586 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:41.586550 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" event={"ID":"09846780-4fb0-474a-bd15-9b272dc00772","Type":"ContainerStarted","Data":"5c263e8327f9629afa8f11713bfc4809b79f84d969cc728f9c808e55f63dc91d"} Oct 30 04:43:41.628713 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:41.628668 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-crggn" podStartSLOduration=8.067726639 podStartE2EDuration="17.628656077s" totalImagesPullingTime="9.560929438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:41.627748402 +0000 UTC m=+17.934426235" watchObservedRunningTime="2026-10-30 04:43:41.628656077 +0000 UTC m=+17.935333909" Oct 30 04:43:42.054560 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:42.054535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:42.054735 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:42.054570 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv6f\" (UniqueName: \"kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f\") pod \"network-check-target-bblhz\" (UID: \"1c23f69d-0026-4202-884b-38a3d4bbe87d\") " pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:42.054735 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:42.054602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:42.054735 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:42.054690 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:42.054735 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:42.054695 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 04:43:42.054735 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:42.054711 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 04:43:42.054735 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:42.054720 2577 projected.go:196] Error preparing data for projected volume kube-api-access-hsv6f for pod openshift-network-diagnostics/network-check-target-bblhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:42.054967 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:42.054740 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret podName:0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:58.054728335 +0000 UTC m=+34.361406159 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret") pod "global-pull-secret-syncer-hcgqp" (UID: "0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c") : object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:42.054967 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:42.054758 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f podName:1c23f69d-0026-4202-884b-38a3d4bbe87d nodeName:}" failed. No retries permitted until 2026-10-30 04:43:58.054746473 +0000 UTC m=+34.361424283 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv6f" (UniqueName: "kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f") pod "network-check-target-bblhz" (UID: "1c23f69d-0026-4202-884b-38a3d4bbe87d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:42.054967 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:42.054690 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:42.054967 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:42.054787 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs podName:9f09ff3d-66bc-46bc-b404-9a45acd58e1d nodeName:}" failed. No retries permitted until 2026-10-30 04:43:58.05478136 +0000 UTC m=+34.361459171 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs") pod "network-metrics-daemon-kcmnj" (UID: "9f09ff3d-66bc-46bc-b404-9a45acd58e1d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:42.283590 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:42.283559 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:42.283590 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:42.283590 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:42.283776 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:42.283665 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:42.283776 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:42.283737 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:43.283372 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:43.283334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:43.283769 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:43.283437 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:43.598568 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:43.598542 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 04:43:43.598879 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:43.598853 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" event={"ID":"09846780-4fb0-474a-bd15-9b272dc00772","Type":"ContainerStarted","Data":"39a0fb8f4886768285bbb08b71b28fd01a4ab3ad7fa39b59c74ca1ab7c6263ee"} Oct 30 04:43:44.284769 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:44.284734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:44.285131 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:44.284811 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:44.285131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:44.284913 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:44.285131 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:44.285029 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:45.283276 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:45.283254 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:45.283400 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:45.283342 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:45.605317 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:45.605293 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 04:43:45.605974 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:45.605651 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" event={"ID":"09846780-4fb0-474a-bd15-9b272dc00772","Type":"ContainerStarted","Data":"497a0459df44e1a5a2cc8c430741f5a5023745393b5b7d2c83e5595da3365ca3"} Oct 30 04:43:45.606020 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:45.605976 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:45.606126 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:45.606111 2577 scope.go:122] "RemoveContainer" containerID="8556a6fb16db8931e5624c1a6967a28fa45e22650716d351be70af2a2374ab55" Oct 30 04:43:45.622063 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:45.622045 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:46.187022 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:46.186991 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xhn9f_59632489-f9ac-44e1-ae2a-7a119c0ece48/dns-node-resolver/0.log" Oct 30 04:43:46.283626 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:46.283603 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:46.283746 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:46.283642 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:46.283798 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:46.283747 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:46.283877 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:46.283853 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:46.613771 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:46.613746 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 04:43:46.614171 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:46.614048 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" event={"ID":"09846780-4fb0-474a-bd15-9b272dc00772","Type":"ContainerStarted","Data":"ad3b24c1eaf497a3fb54f1d0201d5c9be6c189aa26f163d476b566586b762e0d"} Oct 30 04:43:46.614171 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:46.614142 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 04:43:46.614486 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:46.614386 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:46.629404 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:46.629386 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:46.666905 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:46.666866 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" podStartSLOduration=8.430429197 podStartE2EDuration="22.666855442s" totalImagesPullingTime="14.236426245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:46.665937626 +0000 UTC m=+22.972615459" watchObservedRunningTime="2026-10-30 04:43:46.666855442 +0000 UTC m=+22.973533274" Oct 30 04:43:46.735893 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:46.735873 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dl5rx_9c1d23d7-96c3-4406-9354-860812b8d2bb/node-ca/0.log" Oct 30 04:43:47.284179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:47.284151 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:47.284318 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:47.284254 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:47.616728 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:47.616702 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 04:43:48.283522 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:48.283490 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:48.283522 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:48.283512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:48.283774 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:48.283590 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:48.283774 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:48.283696 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:48.618695 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:48.618669 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 04:43:49.284053 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:49.284027 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:49.284189 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:49.284124 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:50.284013 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:50.283981 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:50.284406 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:50.283981 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:50.284406 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:50.284108 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:50.284406 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:50.284148 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:51.283126 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:51.283094 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:51.283281 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:51.283206 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:52.283984 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:52.283951 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:52.284551 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:52.284051 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:52.284551 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:52.284150 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:52.284551 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:52.284275 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:53.283894 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:53.283863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:53.284055 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:53.283963 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:54.284445 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:54.284417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:54.284840 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:54.284498 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:54.284840 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:54.284586 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:54.284840 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:54.284691 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:55.230044 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:55.230007 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:43:55.230294 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:55.230269 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 04:43:55.249635 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:55.249602 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" podUID="09846780-4fb0-474a-bd15-9b272dc00772" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 04:43:55.257972 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:55.257939 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" podUID="09846780-4fb0-474a-bd15-9b272dc00772" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 04:43:55.283792 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:55.283772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:55.283904 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:55.283875 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:56.286148 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:56.286121 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:56.286532 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:56.286220 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:56.286532 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:56.286271 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:56.286532 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:56.286352 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:56.369681 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:56.369654 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:43:56.369850 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:56.369762 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 04:43:56.370177 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:56.370160 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:43:57.283262 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:57.283235 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:57.283456 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:57.283326 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:43:58.074937 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:58.074909 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:58.075440 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:58.074945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv6f\" (UniqueName: \"kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f\") pod \"network-check-target-bblhz\" (UID: \"1c23f69d-0026-4202-884b-38a3d4bbe87d\") " pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:58.075440 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:58.074971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:58.075440 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:58.075067 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:58.075440 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:58.075076 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:58.075440 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:58.075111 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret podName:0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c nodeName:}" failed. No retries permitted until 2026-10-30 04:44:30.075099997 +0000 UTC m=+66.381777809 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret") pod "global-pull-secret-syncer-hcgqp" (UID: "0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c") : object "kube-system"/"original-pull-secret" not registered Oct 30 04:43:58.075440 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:58.075142 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs podName:9f09ff3d-66bc-46bc-b404-9a45acd58e1d nodeName:}" failed. No retries permitted until 2026-10-30 04:44:30.0751268 +0000 UTC m=+66.381804624 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs") pod "network-metrics-daemon-kcmnj" (UID: "9f09ff3d-66bc-46bc-b404-9a45acd58e1d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 04:43:58.075440 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:58.075081 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 04:43:58.075440 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:58.075173 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 04:43:58.075440 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:58.075185 2577 projected.go:196] Error preparing data for projected volume kube-api-access-hsv6f for pod openshift-network-diagnostics/network-check-target-bblhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:58.075440 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:58.075227 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f podName:1c23f69d-0026-4202-884b-38a3d4bbe87d nodeName:}" failed. No retries permitted until 2026-10-30 04:44:30.075215992 +0000 UTC m=+66.381893821 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hsv6f" (UniqueName: "kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f") pod "network-check-target-bblhz" (UID: "1c23f69d-0026-4202-884b-38a3d4bbe87d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 04:43:58.286116 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:58.286088 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:43:58.286274 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:58.286196 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:43:58.286274 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:58.286215 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:43:58.286348 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:58.286288 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:43:58.286348 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:43:58.286324 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:43:58.286425 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:43:58.286390 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:44:00.283237 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:00.283204 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:00.283237 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:00.283227 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:44:00.283789 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:00.283326 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:44:00.283789 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:00.283421 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:44:00.283789 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:00.283485 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:44:00.283789 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:00.283562 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:44:02.283618 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:02.283586 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:44:02.284126 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:02.283683 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:44:02.284126 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:02.283723 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:02.284126 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:02.283812 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:44:02.284126 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:02.283857 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:44:02.284126 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:02.283944 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:44:03.396046 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:03.395910 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hcgqp"] Oct 30 04:44:03.396456 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:03.396084 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:03.396456 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:03.396188 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:44:03.400218 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:03.400188 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kcmnj"] Oct 30 04:44:03.400347 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:03.400322 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:44:03.400482 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:03.400455 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:44:03.400932 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:03.400905 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bblhz"] Oct 30 04:44:03.401216 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:03.401006 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:44:03.401216 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:03.401092 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:44:04.640415 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:04.640335 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xhn9f_59632489-f9ac-44e1-ae2a-7a119c0ece48/dns-node-resolver/0.log" Oct 30 04:44:05.283874 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:05.283836 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:44:05.284042 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:05.283844 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:05.284042 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:05.283937 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:44:05.284042 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:05.283836 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:44:05.284042 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:05.284025 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:44:05.284195 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:05.284100 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:44:05.439632 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:05.439606 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dl5rx_9c1d23d7-96c3-4406-9354-860812b8d2bb/node-ca/0.log" Oct 30 04:44:07.283481 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.283296 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:44:07.283937 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.283296 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:44:07.283937 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:07.283596 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcmnj" podUID="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" Oct 30 04:44:07.283937 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.283313 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:07.283937 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:07.283626 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bblhz" podUID="1c23f69d-0026-4202-884b-38a3d4bbe87d" Oct 30 04:44:07.283937 ip-10-0-134-105 kubenswrapper[2577]: E1030 04:44:07.283700 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcgqp" podUID="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" Oct 30 04:44:07.526170 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.526142 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeReady" Oct 30 04:44:07.526299 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.526272 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 04:44:07.578919 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.578888 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-69jf4"] Oct 30 04:44:07.580644 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.580622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-69jf4" Oct 30 04:44:07.582164 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.582140 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h72cf"] Oct 30 04:44:07.583849 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.583826 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.599391 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.599352 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-g782k\"" Oct 30 04:44:07.599467 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.599436 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 04:44:07.600004 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.599980 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8jf62"] Oct 30 04:44:07.601613 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.601596 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:07.602921 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.602896 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k8wtk\"" Oct 30 04:44:07.603062 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.603039 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 04:44:07.603163 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.603143 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 04:44:07.603237 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.603193 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 04:44:07.603464 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.603430 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 04:44:07.603843 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.603825 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-69jf4"] Oct 30 04:44:07.612625 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.612610 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 04:44:07.623472 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.623448 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8jf62"] Oct 30 04:44:07.629688 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.629668 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h72cf"] Oct 30 04:44:07.653015 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.652994 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5c8z\" (UniqueName: \"kubernetes.io/projected/c669d496-909b-428e-9b02-1ba9b05fd718-kube-api-access-b5c8z\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:07.653118 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.653034 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54628fc4-6d74-4048-ab60-61a643aac200-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.653181 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.653159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54628fc4-6d74-4048-ab60-61a643aac200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.653241 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.653191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbcp\" (UniqueName: \"kubernetes.io/projected/54628fc4-6d74-4048-ab60-61a643aac200-kube-api-access-8wbcp\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.653241 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.653229 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2967a60d-8f1a-4084-ac8b-5c50109aaf3b-cert\") pod \"ingress-canary-69jf4\" (UID: \"2967a60d-8f1a-4084-ac8b-5c50109aaf3b\") " pod="openshift-ingress-canary/ingress-canary-69jf4" Oct 30 04:44:07.653323 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.653311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gdq\" (UniqueName: \"kubernetes.io/projected/2967a60d-8f1a-4084-ac8b-5c50109aaf3b-kube-api-access-22gdq\") pod \"ingress-canary-69jf4\" (UID: \"2967a60d-8f1a-4084-ac8b-5c50109aaf3b\") " pod="openshift-ingress-canary/ingress-canary-69jf4" Oct 30 04:44:07.653392 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.653341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54628fc4-6d74-4048-ab60-61a643aac200-crio-socket\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.653435 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.653416 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c669d496-909b-428e-9b02-1ba9b05fd718-metrics-tls\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:07.653470 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.653440 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c669d496-909b-428e-9b02-1ba9b05fd718-config-volume\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:07.653470 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.653454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c669d496-909b-428e-9b02-1ba9b05fd718-tmp-dir\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:07.653537 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.653470 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54628fc4-6d74-4048-ab60-61a643aac200-data-volume\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.699553 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.699493 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4jtzx\"" Oct 30 04:44:07.699553 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.699514 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 04:44:07.699553 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.699543 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 04:44:07.702994 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.702974 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 04:44:07.754425 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.754384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c669d496-909b-428e-9b02-1ba9b05fd718-metrics-tls\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:07.754425 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.754425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c669d496-909b-428e-9b02-1ba9b05fd718-config-volume\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:07.754646 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.754450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c669d496-909b-428e-9b02-1ba9b05fd718-tmp-dir\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:07.754646 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.754583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54628fc4-6d74-4048-ab60-61a643aac200-data-volume\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.754646 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.754640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5c8z\" (UniqueName: \"kubernetes.io/projected/c669d496-909b-428e-9b02-1ba9b05fd718-kube-api-access-b5c8z\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:07.754790 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.754682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54628fc4-6d74-4048-ab60-61a643aac200-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.754790 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.754731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54628fc4-6d74-4048-ab60-61a643aac200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.754790 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.754756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbcp\" (UniqueName: \"kubernetes.io/projected/54628fc4-6d74-4048-ab60-61a643aac200-kube-api-access-8wbcp\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.754790 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.754782 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2967a60d-8f1a-4084-ac8b-5c50109aaf3b-cert\") pod \"ingress-canary-69jf4\" (UID: \"2967a60d-8f1a-4084-ac8b-5c50109aaf3b\") " pod="openshift-ingress-canary/ingress-canary-69jf4" Oct 30 04:44:07.754978 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.754822 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c669d496-909b-428e-9b02-1ba9b05fd718-tmp-dir\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:07.754978 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.754829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gdq\" (UniqueName: \"kubernetes.io/projected/2967a60d-8f1a-4084-ac8b-5c50109aaf3b-kube-api-access-22gdq\") pod \"ingress-canary-69jf4\" (UID: \"2967a60d-8f1a-4084-ac8b-5c50109aaf3b\") " pod="openshift-ingress-canary/ingress-canary-69jf4" Oct 30 04:44:07.754978 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.754855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54628fc4-6d74-4048-ab60-61a643aac200-crio-socket\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.754978 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.754882 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54628fc4-6d74-4048-ab60-61a643aac200-data-volume\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.755169 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.755051 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54628fc4-6d74-4048-ab60-61a643aac200-crio-socket\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.755401 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.755382 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54628fc4-6d74-4048-ab60-61a643aac200-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.755502 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.755487 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c669d496-909b-428e-9b02-1ba9b05fd718-config-volume\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:07.758648 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.758626 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54628fc4-6d74-4048-ab60-61a643aac200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.758726 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.758629 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c669d496-909b-428e-9b02-1ba9b05fd718-metrics-tls\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:07.758771 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.758720 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2967a60d-8f1a-4084-ac8b-5c50109aaf3b-cert\") pod \"ingress-canary-69jf4\" (UID: \"2967a60d-8f1a-4084-ac8b-5c50109aaf3b\") " pod="openshift-ingress-canary/ingress-canary-69jf4" Oct 30 04:44:07.774891 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.774873 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5c8z\" (UniqueName: \"kubernetes.io/projected/c669d496-909b-428e-9b02-1ba9b05fd718-kube-api-access-b5c8z\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:07.774982 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.774955 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbcp\" (UniqueName: \"kubernetes.io/projected/54628fc4-6d74-4048-ab60-61a643aac200-kube-api-access-8wbcp\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.775177 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.775160 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gdq\" (UniqueName: \"kubernetes.io/projected/2967a60d-8f1a-4084-ac8b-5c50109aaf3b-kube-api-access-22gdq\") pod \"ingress-canary-69jf4\" (UID: \"2967a60d-8f1a-4084-ac8b-5c50109aaf3b\") " pod="openshift-ingress-canary/ingress-canary-69jf4" Oct 30 04:44:07.895427 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.895399 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-69jf4" Oct 30 04:44:07.900735 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.900709 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:07.911874 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:07.911851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:08.229040 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:08.229015 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-69jf4"] Oct 30 04:44:08.231292 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:44:08.231260 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2967a60d_8f1a_4084_ac8b_5c50109aaf3b.slice/crio-4d7c1ca735f996da6f911dcd9020143c9ed5a27763f121c855f15488ee481e48 WatchSource:0}: Error finding container 4d7c1ca735f996da6f911dcd9020143c9ed5a27763f121c855f15488ee481e48: Status 404 returned error can't find the container with id 4d7c1ca735f996da6f911dcd9020143c9ed5a27763f121c855f15488ee481e48 Oct 30 04:44:08.232073 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:08.232050 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8jf62"] Oct 30 04:44:08.239836 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:44:08.239809 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54628fc4_6d74_4048_ab60_61a643aac200.slice/crio-d9e0eefb16820888b841272d737f204b42bd71323e7746efa239184d2a179cd4 WatchSource:0}: Error finding container d9e0eefb16820888b841272d737f204b42bd71323e7746efa239184d2a179cd4: Status 404 returned error can't find the container with id d9e0eefb16820888b841272d737f204b42bd71323e7746efa239184d2a179cd4 Oct 30 04:44:08.240448 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:44:08.240426 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc669d496_909b_428e_9b02_1ba9b05fd718.slice/crio-a75e02faa23c5d4cecec249e6a1e92841cb93dd0a066bc2c95b5b04262d4970f WatchSource:0}: Error finding container a75e02faa23c5d4cecec249e6a1e92841cb93dd0a066bc2c95b5b04262d4970f: Status 404 returned error can't find the container with id a75e02faa23c5d4cecec249e6a1e92841cb93dd0a066bc2c95b5b04262d4970f Oct 30 04:44:08.242456 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:08.242435 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h72cf"] Oct 30 04:44:08.667399 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:08.667343 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8jf62" event={"ID":"c669d496-909b-428e-9b02-1ba9b05fd718","Type":"ContainerStarted","Data":"a75e02faa23c5d4cecec249e6a1e92841cb93dd0a066bc2c95b5b04262d4970f"} Oct 30 04:44:08.669348 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:08.669316 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h72cf" event={"ID":"54628fc4-6d74-4048-ab60-61a643aac200","Type":"ContainerStarted","Data":"4166fe83119ba62d7b0b7e8a8be0f5d0ec24346482deb72e960df6a129bd90fc"} Oct 30 04:44:08.669495 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:08.669420 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h72cf" event={"ID":"54628fc4-6d74-4048-ab60-61a643aac200","Type":"ContainerStarted","Data":"d9e0eefb16820888b841272d737f204b42bd71323e7746efa239184d2a179cd4"} Oct 30 04:44:08.670923 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:08.670899 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-69jf4" event={"ID":"2967a60d-8f1a-4084-ac8b-5c50109aaf3b","Type":"ContainerStarted","Data":"4d7c1ca735f996da6f911dcd9020143c9ed5a27763f121c855f15488ee481e48"} Oct 30 04:44:09.283786 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.283711 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:09.283786 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.283766 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:44:09.283996 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.283711 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:44:09.301519 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.300643 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 04:44:09.301519 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.300739 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 04:44:09.314132 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.313874 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-56dd5\"" Oct 30 04:44:09.314132 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.313974 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 04:44:09.314132 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.314037 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k257x\"" Oct 30 04:44:09.314132 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.314079 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mxnkb\"" Oct 30 04:44:09.314132 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.314105 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 04:44:09.677677 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.677559 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h72cf" event={"ID":"54628fc4-6d74-4048-ab60-61a643aac200","Type":"ContainerStarted","Data":"a486c37ddcda086177fadade66ea1789583f76394e745a029bdf89ca5fb4eee6"} Oct 30 04:44:09.684619 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.684585 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9"] Oct 30 04:44:09.687172 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.687130 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.693618 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.693594 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z4r7p"] Oct 30 04:44:09.695524 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.695504 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.713982 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.713351 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 04:44:09.713982 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.713392 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cswjq\"" Oct 30 04:44:09.713982 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.713504 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:44:09.714177 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.714069 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:44:09.715458 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.714550 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 04:44:09.715458 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.715195 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 04:44:09.718307 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.718289 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 04:44:09.719083 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.719059 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9"] Oct 30 04:44:09.738131 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.738110 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 04:44:09.738280 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.738266 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzdcr\"" Oct 30 04:44:09.738350 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.738303 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 04:44:09.750675 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.750651 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6"] Oct 30 04:44:09.752721 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.752700 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:09.768950 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.768928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.769058 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.768958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32ed1e66-99e0-4714-92d1-1ef6213fed2c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.769058 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.768989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-wtmp\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.769166 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769118 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.769166 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769158 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30011b22-59f1-4361-b961-c5be6de91e88-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:09.769282 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30011b22-59f1-4361-b961-c5be6de91e88-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:09.769282 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769221 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30011b22-59f1-4361-b961-c5be6de91e88-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:09.769282 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769250 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.769405 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769280 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-tls\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.769405 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-root\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.769405 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/32ed1e66-99e0-4714-92d1-1ef6213fed2c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.769580 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd5t4\" (UniqueName: \"kubernetes.io/projected/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-api-access-dd5t4\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.769658 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769636 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdhkb\" (UniqueName: \"kubernetes.io/projected/30011b22-59f1-4361-b961-c5be6de91e88-kube-api-access-zdhkb\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:09.769784 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.769844 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1e09016-1f64-4c16-be6b-ab60b33a13e8-metrics-client-ca\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.769844 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.769945 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-sys\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.769945 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-textfile\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.769945 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.769891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpl2b\" (UniqueName: \"kubernetes.io/projected/b1e09016-1f64-4c16-be6b-ab60b33a13e8-kube-api-access-dpl2b\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.782546 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.782515 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6"] Oct 30 04:44:09.797484 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.797204 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fq9cl\"" Oct 30 04:44:09.797484 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.797384 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 04:44:09.797484 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.797460 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 04:44:09.807879 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.807857 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 04:44:09.871142 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871033 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-wtmp\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.871291 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.871291 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30011b22-59f1-4361-b961-c5be6de91e88-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:09.871291 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30011b22-59f1-4361-b961-c5be6de91e88-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:09.871291 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30011b22-59f1-4361-b961-c5be6de91e88-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:09.871498 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.871498 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-tls\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.871498 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871390 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-root\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.871498 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/32ed1e66-99e0-4714-92d1-1ef6213fed2c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.871693 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd5t4\" (UniqueName: \"kubernetes.io/projected/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-api-access-dd5t4\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.871693 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdhkb\" (UniqueName: \"kubernetes.io/projected/30011b22-59f1-4361-b961-c5be6de91e88-kube-api-access-zdhkb\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:09.871693 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.871693 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1e09016-1f64-4c16-be6b-ab60b33a13e8-metrics-client-ca\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.871693 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.871693 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-sys\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.871693 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871683 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-textfile\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.872010 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpl2b\" (UniqueName: \"kubernetes.io/projected/b1e09016-1f64-4c16-be6b-ab60b33a13e8-kube-api-access-dpl2b\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.872010 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.872010 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.871820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32ed1e66-99e0-4714-92d1-1ef6213fed2c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.874955 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.872689 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32ed1e66-99e0-4714-92d1-1ef6213fed2c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.874955 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.872852 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-wtmp\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.874955 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.873860 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/32ed1e66-99e0-4714-92d1-1ef6213fed2c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.874955 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.873946 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-root\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.874955 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.874018 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-sys\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.874955 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.874522 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-textfile\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.874955 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.874561 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.874955 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.874960 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1e09016-1f64-4c16-be6b-ab60b33a13e8-metrics-client-ca\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.875448 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.875003 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30011b22-59f1-4361-b961-c5be6de91e88-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:09.875619 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.875596 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.877147 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.877105 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30011b22-59f1-4361-b961-c5be6de91e88-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:09.877666 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.877642 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-tls\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.877911 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.877892 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.878409 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.878385 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.879319 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.879291 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.880076 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.880019 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30011b22-59f1-4361-b961-c5be6de91e88-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:09.894316 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.894281 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpl2b\" (UniqueName: \"kubernetes.io/projected/b1e09016-1f64-4c16-be6b-ab60b33a13e8-kube-api-access-dpl2b\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:09.894564 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.894475 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd5t4\" (UniqueName: \"kubernetes.io/projected/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-api-access-dd5t4\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:09.895412 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:09.895382 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdhkb\" (UniqueName: \"kubernetes.io/projected/30011b22-59f1-4361-b961-c5be6de91e88-kube-api-access-zdhkb\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:10.004002 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.003915 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:10.024913 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.024772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:10.053236 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:44:10.053185 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e09016_1f64_4c16_be6b_ab60b33a13e8.slice/crio-bfd57d4846df9970bbc704f67217e30528c7bdef8afbd2d22be99a8ca64cdd67 WatchSource:0}: Error finding container bfd57d4846df9970bbc704f67217e30528c7bdef8afbd2d22be99a8ca64cdd67: Status 404 returned error can't find the container with id bfd57d4846df9970bbc704f67217e30528c7bdef8afbd2d22be99a8ca64cdd67 Oct 30 04:44:10.076585 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.070347 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:10.329661 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.329153 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9"] Oct 30 04:44:10.330912 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:44:10.330878 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ed1e66_99e0_4714_92d1_1ef6213fed2c.slice/crio-01790731e2d0f87cd94e012aa2e97bb2b15c0c9a2685a2f8eb23b2e77030552f WatchSource:0}: Error finding container 01790731e2d0f87cd94e012aa2e97bb2b15c0c9a2685a2f8eb23b2e77030552f: Status 404 returned error can't find the container with id 01790731e2d0f87cd94e012aa2e97bb2b15c0c9a2685a2f8eb23b2e77030552f Oct 30 04:44:10.385188 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.385153 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6"] Oct 30 04:44:10.386136 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:44:10.386096 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30011b22_59f1_4361_b961_c5be6de91e88.slice/crio-2d9ce14a3de9bcef35191c32449e17558f07119f1058e47643822c1818cf7dda WatchSource:0}: Error finding container 2d9ce14a3de9bcef35191c32449e17558f07119f1058e47643822c1818cf7dda: Status 404 returned error can't find the container with id 2d9ce14a3de9bcef35191c32449e17558f07119f1058e47643822c1818cf7dda Oct 30 04:44:10.682273 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.682237 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4r7p" event={"ID":"b1e09016-1f64-4c16-be6b-ab60b33a13e8","Type":"ContainerStarted","Data":"bfd57d4846df9970bbc704f67217e30528c7bdef8afbd2d22be99a8ca64cdd67"} Oct 30 04:44:10.684513 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.684482 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8jf62" event={"ID":"c669d496-909b-428e-9b02-1ba9b05fd718","Type":"ContainerStarted","Data":"b9e99f5c5209178c2eceb70d4299731ad428d7a291bba2547ea56d636614242f"} Oct 30 04:44:10.684648 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.684528 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8jf62" event={"ID":"c669d496-909b-428e-9b02-1ba9b05fd718","Type":"ContainerStarted","Data":"05418142a773d3ce40e37360518b3f2b5618b6a2f557742b7bb034088c6742e5"} Oct 30 04:44:10.684648 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.684587 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:10.687083 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.687047 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h72cf" event={"ID":"54628fc4-6d74-4048-ab60-61a643aac200","Type":"ContainerStarted","Data":"e6de72a0b9f562f136cc1fe5ee70c1cc8e3ebc76fea66859c3e891a13de48593"} Oct 30 04:44:10.688322 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.688296 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" event={"ID":"32ed1e66-99e0-4714-92d1-1ef6213fed2c","Type":"ContainerStarted","Data":"01790731e2d0f87cd94e012aa2e97bb2b15c0c9a2685a2f8eb23b2e77030552f"} Oct 30 04:44:10.690327 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.690301 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" event={"ID":"30011b22-59f1-4361-b961-c5be6de91e88","Type":"ContainerStarted","Data":"c9617280d10c0d09db73687dedbd822e0cb3d37d79c27d6e4df0a92dbaba7226"} Oct 30 04:44:10.690440 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.690369 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" event={"ID":"30011b22-59f1-4361-b961-c5be6de91e88","Type":"ContainerStarted","Data":"e637fa15a5a012187fb02241632017fb71cdb62b583f039528ba4462bc7ce75e"} Oct 30 04:44:10.690440 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.690399 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" event={"ID":"30011b22-59f1-4361-b961-c5be6de91e88","Type":"ContainerStarted","Data":"2d9ce14a3de9bcef35191c32449e17558f07119f1058e47643822c1818cf7dda"} Oct 30 04:44:10.727004 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.726269 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-8jf62" podStartSLOduration=1.91083242 podStartE2EDuration="3.726253s" totalImagesPullingTime="1.81542058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:10.725925908 +0000 UTC m=+47.032603735" watchObservedRunningTime="2026-10-30 04:44:10.726253 +0000 UTC m=+47.032930812" Oct 30 04:44:10.767981 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:10.767927 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h72cf" podStartSLOduration=1.73003554 podStartE2EDuration="3.767911816s" totalImagesPullingTime="2.037876276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:44:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:10.766595078 +0000 UTC m=+47.073272940" watchObservedRunningTime="2026-10-30 04:44:10.767911816 +0000 UTC m=+47.074589649" Oct 30 04:44:11.124005 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.123970 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 04:44:11.128723 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.127655 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.144422 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.144333 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 04:44:11.144563 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.144424 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 04:44:11.144563 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.144472 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 04:44:11.144563 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.144498 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 04:44:11.144736 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.144585 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 04:44:11.144736 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.144587 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 04:44:11.144736 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.144628 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 04:44:11.144736 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.144658 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 04:44:11.144736 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.144704 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s48rw\"" Oct 30 04:44:11.144736 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.144726 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 04:44:11.153128 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.153105 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 04:44:11.181860 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.181834 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkjb7\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-kube-api-access-dkjb7\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.181979 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.181881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-tls-assets\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.181979 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.181907 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.182052 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.182009 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.182085 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.182055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-web-config\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.182117 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.182089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.182145 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.182125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.182191 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.182171 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.182240 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.182207 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.182275 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.182236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.182275 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.182264 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-config-volume\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.182334 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.182305 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.182384 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.182348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-config-out\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.282711 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.282678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.282837 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.282722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.282837 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.282750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-config-volume\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.282837 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.282791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.282837 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.282828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-config-out\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.282996 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.282858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjb7\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-kube-api-access-dkjb7\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.282996 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.282912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-tls-assets\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.282996 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.282935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.282996 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.282963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.282996 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.282992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-web-config\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.283154 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.283021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.283154 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.283049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.283154 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.283088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.284221 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.284189 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.284514 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.284487 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.284900 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.284876 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.286285 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.286262 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-config-out\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.292787 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.292754 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-web-config\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.294718 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.294559 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.294718 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.294597 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-tls-assets\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.294996 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.294970 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.295088 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.295065 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.295140 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.295105 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-config-volume\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.295239 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.295137 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.295950 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.295889 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.304425 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.304404 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkjb7\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-kube-api-access-dkjb7\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.450207 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.450167 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:11.697896 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.697865 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" event={"ID":"30011b22-59f1-4361-b961-c5be6de91e88","Type":"ContainerStarted","Data":"346961dc71d226c7c2b21f47623aedb50a0dca8f8735e12c12b931e0ed3908dd"} Oct 30 04:44:11.699433 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.699403 2577 generic.go:356] "Generic (PLEG): container finished" podID="b1e09016-1f64-4c16-be6b-ab60b33a13e8" containerID="622355d997536302706e7f0e2f72de41bc6e56f87f251e335df2a5a5cc3d1711" exitCode=0 Oct 30 04:44:11.699569 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.699468 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4r7p" event={"ID":"b1e09016-1f64-4c16-be6b-ab60b33a13e8","Type":"ContainerDied","Data":"622355d997536302706e7f0e2f72de41bc6e56f87f251e335df2a5a5cc3d1711"} Oct 30 04:44:11.700886 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.700865 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-69jf4" event={"ID":"2967a60d-8f1a-4084-ac8b-5c50109aaf3b","Type":"ContainerStarted","Data":"60bdb4b7ce1b8d1b6868b5211f8395ba4a554787e06a481edef4e738dc7e8b1d"} Oct 30 04:44:11.742168 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.742127 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" podStartSLOduration=1.642074181 podStartE2EDuration="2.742111811s" totalImagesPullingTime="1.10003763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:11.739449837 +0000 UTC m=+48.046127673" watchObservedRunningTime="2026-10-30 04:44:11.742111811 +0000 UTC m=+48.048789686" Oct 30 04:44:11.783559 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.783504 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 04:44:11.827518 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:11.827460 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-69jf4" podStartSLOduration=1.872506697 podStartE2EDuration="4.82743975s" totalImagesPullingTime="2.954933053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:11.826295476 +0000 UTC m=+48.132973336" watchObservedRunningTime="2026-10-30 04:44:11.82743975 +0000 UTC m=+48.134117598" Oct 30 04:44:12.706178 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:12.706138 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" event={"ID":"32ed1e66-99e0-4714-92d1-1ef6213fed2c","Type":"ContainerStarted","Data":"0044d83d848efa4250a441a73ab1566bc6caa17d41a7e8d11af4bde50a750851"} Oct 30 04:44:12.706656 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:12.706215 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" event={"ID":"32ed1e66-99e0-4714-92d1-1ef6213fed2c","Type":"ContainerStarted","Data":"2ef18281561c5f8b7eacf31d85786670e0061e9eeed9d436002b7a14c7b04182"} Oct 30 04:44:12.706656 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:12.706243 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" event={"ID":"32ed1e66-99e0-4714-92d1-1ef6213fed2c","Type":"ContainerStarted","Data":"3d3475cdab23ecbe302618eeefec987f8e020a3e4a0d419d8ef61151ef00deaf"} Oct 30 04:44:12.707824 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:12.707797 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerStarted","Data":"af9d29a9ea9dc92cd9b476d44afe8091a22404a8f59fbef697d73a00b5fa1da0"} Oct 30 04:44:12.710046 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:12.710017 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4r7p" event={"ID":"b1e09016-1f64-4c16-be6b-ab60b33a13e8","Type":"ContainerStarted","Data":"d15abd63b804777b4c0607ed62962158da6b0911a0fdb7a2c94fa5a8dedd78cc"} Oct 30 04:44:12.710157 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:12.710072 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4r7p" event={"ID":"b1e09016-1f64-4c16-be6b-ab60b33a13e8","Type":"ContainerStarted","Data":"1f569723d829d080b4eb6239a3c068e302e0ebe6fc95dc71c7c8a8f4364b4647"} Oct 30 04:44:12.755803 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:12.755744 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" podStartSLOduration=2.3802093380000002 podStartE2EDuration="3.755725845s" totalImagesPullingTime="1.375516507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:12.754766802 +0000 UTC m=+49.061444639" watchObservedRunningTime="2026-10-30 04:44:12.755725845 +0000 UTC m=+49.062403682" Oct 30 04:44:12.796758 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:12.796700 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z4r7p" podStartSLOduration=2.745444392 podStartE2EDuration="3.796682469s" totalImagesPullingTime="963.586616ms" totalInitContainerRuntime="87.651461ms" isStatefulPod=true podCreationTimestamp="2026-10-30 04:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:12.795561603 +0000 UTC m=+49.102239439" watchObservedRunningTime="2026-10-30 04:44:12.796682469 +0000 UTC m=+49.103360305" Oct 30 04:44:13.714067 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:13.714033 2577 generic.go:356] "Generic (PLEG): container finished" podID="084e8194-89a6-4702-af2a-34df80b46792" containerID="19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22" exitCode=0 Oct 30 04:44:13.714570 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:13.714118 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerDied","Data":"19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22"} Oct 30 04:44:14.161092 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.161053 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d6768cb7d-ssqrt"] Oct 30 04:44:14.163196 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.163175 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.179166 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.179142 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zp4kp\"" Oct 30 04:44:14.179278 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.179222 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 04:44:14.179278 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.179276 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 04:44:14.179396 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.179282 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 04:44:14.181956 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.181937 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 04:44:14.186533 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.186471 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d6768cb7d-ssqrt"] Oct 30 04:44:14.207040 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.207013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9fd93f4-68f8-4711-ad54-410d10af6c35-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.207167 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.207046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn4sc\" (UniqueName: \"kubernetes.io/projected/d9fd93f4-68f8-4711-ad54-410d10af6c35-kube-api-access-nn4sc\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.207167 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.207138 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d9fd93f4-68f8-4711-ad54-410d10af6c35-audit-log\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.207270 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.207191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d9fd93f4-68f8-4711-ad54-410d10af6c35-metrics-server-audit-profiles\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.207323 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.207301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-secret-metrics-server-tls\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.207393 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.207377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-secret-metrics-server-client-certs\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.207457 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.207416 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-client-ca-bundle\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.279743 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.279711 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4oao19m04ptk3\"" Oct 30 04:44:14.307918 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.307884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d9fd93f4-68f8-4711-ad54-410d10af6c35-metrics-server-audit-profiles\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.308074 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.307928 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-secret-metrics-server-tls\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.308074 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.307954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-secret-metrics-server-client-certs\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.308190 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.308079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-client-ca-bundle\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.308190 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.308147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9fd93f4-68f8-4711-ad54-410d10af6c35-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.308190 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.308182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn4sc\" (UniqueName: \"kubernetes.io/projected/d9fd93f4-68f8-4711-ad54-410d10af6c35-kube-api-access-nn4sc\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.308335 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.308307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d9fd93f4-68f8-4711-ad54-410d10af6c35-audit-log\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.308784 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.308758 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d9fd93f4-68f8-4711-ad54-410d10af6c35-audit-log\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.308898 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.308820 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9fd93f4-68f8-4711-ad54-410d10af6c35-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.309146 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.309107 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d9fd93f4-68f8-4711-ad54-410d10af6c35-metrics-server-audit-profiles\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.310688 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.310668 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-secret-metrics-server-client-certs\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.310850 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.310832 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-client-ca-bundle\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.310915 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.310863 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-secret-metrics-server-tls\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.328439 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.328415 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn4sc\" (UniqueName: \"kubernetes.io/projected/d9fd93f4-68f8-4711-ad54-410d10af6c35-kube-api-access-nn4sc\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.473175 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.473087 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:14.781567 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:14.781538 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d6768cb7d-ssqrt"] Oct 30 04:44:14.782482 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:44:14.782454 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9fd93f4_68f8_4711_ad54_410d10af6c35.slice/crio-06f85ee326de087a302fcc9708effe33e27a8bb2c2c69ad6b59f17c0d1573269 WatchSource:0}: Error finding container 06f85ee326de087a302fcc9708effe33e27a8bb2c2c69ad6b59f17c0d1573269: Status 404 returned error can't find the container with id 06f85ee326de087a302fcc9708effe33e27a8bb2c2c69ad6b59f17c0d1573269 Oct 30 04:44:15.725408 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:15.725304 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerStarted","Data":"aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c"} Oct 30 04:44:15.725576 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:15.725440 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerStarted","Data":"055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88"} Oct 30 04:44:15.725576 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:15.725481 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerStarted","Data":"467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123"} Oct 30 04:44:15.725576 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:15.725513 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerStarted","Data":"336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d"} Oct 30 04:44:15.725576 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:15.725539 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerStarted","Data":"79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0"} Oct 30 04:44:15.726629 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:15.726594 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" event={"ID":"d9fd93f4-68f8-4711-ad54-410d10af6c35","Type":"ContainerStarted","Data":"06f85ee326de087a302fcc9708effe33e27a8bb2c2c69ad6b59f17c0d1573269"} Oct 30 04:44:16.703971 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.703939 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 04:44:16.707517 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.707501 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.723783 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.723762 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 04:44:16.723873 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.723812 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 04:44:16.727941 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.727918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728014 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.727966 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728014 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.727987 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728119 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728178 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728120 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728222 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728181 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728260 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728234 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728298 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728353 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728460 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728493 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728478 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728542 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728636 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728623 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728673 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728662 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728712 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-config\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728750 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728796 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728781 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728831 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728822 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728874 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728843 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.728874 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.728865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9w2d\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-kube-api-access-l9w2d\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.733037 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.733013 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerStarted","Data":"7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a"} Oct 30 04:44:16.734515 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.734487 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" event={"ID":"d9fd93f4-68f8-4711-ad54-410d10af6c35","Type":"ContainerStarted","Data":"51e36d7081a830519f675aa2f58cff0108a84ce6101edfc38da22157f06baa67"} Oct 30 04:44:16.736961 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.736937 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 04:44:16.737596 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.737066 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 04:44:16.737596 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.737077 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 04:44:16.737596 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.737115 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hsqv5\"" Oct 30 04:44:16.737596 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.737137 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 04:44:16.737596 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.737144 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 04:44:16.737596 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.737177 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 04:44:16.737596 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.737250 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3fklq5stm461m\"" Oct 30 04:44:16.752545 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.752524 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 04:44:16.753698 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.753681 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 04:44:16.809500 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.809454 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" podStartSLOduration=1.4934881039999999 podStartE2EDuration="2.809442869s" totalImagesPullingTime="1.315954765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:16.808947276 +0000 UTC m=+53.115625109" watchObservedRunningTime="2026-10-30 04:44:16.809442869 +0000 UTC m=+53.116120702" Oct 30 04:44:16.824130 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.824106 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 04:44:16.824204 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.824161 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 04:44:16.824236 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.824224 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 04:44:16.824499 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.824479 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 04:44:16.825277 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.825259 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.829855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.829905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830006 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9w2d\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-kube-api-access-l9w2d\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830228 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830258 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830396 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.833179 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.834046 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-config\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.834046 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.834046 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.830965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.834046 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.832663 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.835295 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.834299 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.835295 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.835006 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.837085 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.836763 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.837085 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.836796 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.837650 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.837471 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.837835 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.837786 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.838817 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.838241 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.838817 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.838444 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.838817 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.838735 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.838817 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.838757 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.838817 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.838764 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.839125 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.838989 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.839125 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.839115 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.839224 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.839200 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.839334 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.839314 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.840007 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.839989 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.840904 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.840885 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.841196 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.841170 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-config\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.854244 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.854223 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9w2d\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-kube-api-access-l9w2d\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:16.865043 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:16.864969 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.735652324 podStartE2EDuration="5.864955987s" totalImagesPullingTime="3.11899343s" totalInitContainerRuntime="10.310233ms" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:16.863687779 +0000 UTC m=+53.170365613" watchObservedRunningTime="2026-10-30 04:44:16.864955987 +0000 UTC m=+53.171633819" Oct 30 04:44:17.031340 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:17.031268 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:17.360128 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:17.360099 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 04:44:17.370145 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:44:17.370116 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc295553_de72_4758_83ad_7123e5e29f03.slice/crio-d0fb8002b53a36d7660e29b9f85de87d3922a1daab2c0190cf339b402a3bddf8 WatchSource:0}: Error finding container d0fb8002b53a36d7660e29b9f85de87d3922a1daab2c0190cf339b402a3bddf8: Status 404 returned error can't find the container with id d0fb8002b53a36d7660e29b9f85de87d3922a1daab2c0190cf339b402a3bddf8 Oct 30 04:44:17.739781 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:17.739696 2577 generic.go:356] "Generic (PLEG): container finished" podID="fc295553-de72-4758-83ad-7123e5e29f03" containerID="f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7" exitCode=0 Oct 30 04:44:17.740247 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:17.739785 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerDied","Data":"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7"} Oct 30 04:44:17.740247 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:17.739866 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerStarted","Data":"d0fb8002b53a36d7660e29b9f85de87d3922a1daab2c0190cf339b402a3bddf8"} Oct 30 04:44:20.702895 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:20.702867 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:20.760069 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:20.759968 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerStarted","Data":"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c"} Oct 30 04:44:20.760215 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:20.760088 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerStarted","Data":"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148"} Oct 30 04:44:22.771066 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:22.771031 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerStarted","Data":"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5"} Oct 30 04:44:22.771544 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:22.771093 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerStarted","Data":"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65"} Oct 30 04:44:22.771544 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:22.771120 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerStarted","Data":"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5"} Oct 30 04:44:22.771544 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:22.771146 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerStarted","Data":"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a"} Oct 30 04:44:22.861051 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:22.861000 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.416092195 podStartE2EDuration="6.86098409s" totalImagesPullingTime="4.428101018s" totalInitContainerRuntime="16.790877ms" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:22.846805917 +0000 UTC m=+59.153483775" watchObservedRunningTime="2026-10-30 04:44:22.86098409 +0000 UTC m=+59.167661922" Oct 30 04:44:25.260533 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:25.260505 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:27.031872 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:27.031842 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:30.160823 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.160773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv6f\" (UniqueName: \"kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f\") pod \"network-check-target-bblhz\" (UID: \"1c23f69d-0026-4202-884b-38a3d4bbe87d\") " pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:44:30.161337 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.160865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:30.161337 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.160899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:44:30.175576 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.175552 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 04:44:30.188935 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.188910 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 04:44:30.189027 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.188933 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 04:44:30.194196 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.194174 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:30.194283 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.194261 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:44:30.196458 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.196439 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 04:44:30.204635 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.204617 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsv6f\" (UniqueName: \"kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f\") pod \"network-check-target-bblhz\" (UID: \"1c23f69d-0026-4202-884b-38a3d4bbe87d\") " pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:44:30.320496 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.320468 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mxnkb\"" Oct 30 04:44:30.326707 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.326686 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:30.333017 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.332998 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k257x\"" Oct 30 04:44:30.339236 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.339216 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:44:30.340296 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.340278 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-56dd5\"" Oct 30 04:44:30.346325 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.346305 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:44:30.626349 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.626318 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hcgqp"] Oct 30 04:44:30.630786 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:44:30.630761 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9fc38f_1ef9_4fbd_bcc2_3172ed84e48c.slice/crio-054fc3bd50256aea17d3928a2ce12dac51b47670a65f4d2465f322447af18ad9 WatchSource:0}: Error finding container 054fc3bd50256aea17d3928a2ce12dac51b47670a65f4d2465f322447af18ad9: Status 404 returned error can't find the container with id 054fc3bd50256aea17d3928a2ce12dac51b47670a65f4d2465f322447af18ad9 Oct 30 04:44:30.639199 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.639177 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kcmnj"] Oct 30 04:44:30.639653 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:44:30.639631 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f09ff3d_66bc_46bc_b404_9a45acd58e1d.slice/crio-fe096d629033c2c73bd8e35a5d9dd965c0d6ff00360f0a66aa127e05b07b0876 WatchSource:0}: Error finding container fe096d629033c2c73bd8e35a5d9dd965c0d6ff00360f0a66aa127e05b07b0876: Status 404 returned error can't find the container with id fe096d629033c2c73bd8e35a5d9dd965c0d6ff00360f0a66aa127e05b07b0876 Oct 30 04:44:30.649978 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.649954 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bblhz"] Oct 30 04:44:30.651195 ip-10-0-134-105 kubenswrapper[2577]: W1030 04:44:30.651175 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c23f69d_0026_4202_884b_38a3d4bbe87d.slice/crio-d5a218012154f5a9cc35276d76618c1a784cb8dbc68bf7c7f0c6e5e7b9ff3452 WatchSource:0}: Error finding container d5a218012154f5a9cc35276d76618c1a784cb8dbc68bf7c7f0c6e5e7b9ff3452: Status 404 returned error can't find the container with id d5a218012154f5a9cc35276d76618c1a784cb8dbc68bf7c7f0c6e5e7b9ff3452 Oct 30 04:44:30.797685 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.797605 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bblhz" event={"ID":"1c23f69d-0026-4202-884b-38a3d4bbe87d","Type":"ContainerStarted","Data":"d5a218012154f5a9cc35276d76618c1a784cb8dbc68bf7c7f0c6e5e7b9ff3452"} Oct 30 04:44:30.798865 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.798838 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kcmnj" event={"ID":"9f09ff3d-66bc-46bc-b404-9a45acd58e1d","Type":"ContainerStarted","Data":"fe096d629033c2c73bd8e35a5d9dd965c0d6ff00360f0a66aa127e05b07b0876"} Oct 30 04:44:30.799913 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:30.799893 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hcgqp" event={"ID":"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c","Type":"ContainerStarted","Data":"054fc3bd50256aea17d3928a2ce12dac51b47670a65f4d2465f322447af18ad9"} Oct 30 04:44:31.065516 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:31.065474 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cqv2d_7115245a-cbc9-4180-9709-2a6e590bb170/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:31.170809 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:31.170773 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cqv2d_7115245a-cbc9-4180-9709-2a6e590bb170/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:31.806788 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:31.806629 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kcmnj" event={"ID":"9f09ff3d-66bc-46bc-b404-9a45acd58e1d","Type":"ContainerStarted","Data":"59d609ecf29c9180d7eece97ce0b7f07779038a439e485f4ef6f61982fb2954e"} Oct 30 04:44:32.813093 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:32.813051 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kcmnj" event={"ID":"9f09ff3d-66bc-46bc-b404-9a45acd58e1d","Type":"ContainerStarted","Data":"0a6f5953d97d354c6a9732ffd3a8dd0fd0312127d961cf38de64b3d46119735a"} Oct 30 04:44:32.887079 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:32.886318 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kcmnj" podStartSLOduration=67.876246234 podStartE2EDuration="1m8.886302413s" totalImagesPullingTime="1.010056179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:32.872535863 +0000 UTC m=+69.179213698" watchObservedRunningTime="2026-10-30 04:44:32.886302413 +0000 UTC m=+69.192980247" Oct 30 04:44:33.818601 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:33.818566 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bblhz" event={"ID":"1c23f69d-0026-4202-884b-38a3d4bbe87d","Type":"ContainerStarted","Data":"e1cf8b7968c134c09706113117fd087eee8ab6394d92728c32c0627d40b490f6"} Oct 30 04:44:33.819034 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:33.818921 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:44:33.858944 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:33.858888 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bblhz" podStartSLOduration=66.808968306 podStartE2EDuration="1m9.858872598s" totalImagesPullingTime="3.049904292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:33.858116923 +0000 UTC m=+70.164794755" watchObservedRunningTime="2026-10-30 04:44:33.858872598 +0000 UTC m=+70.165550432" Oct 30 04:44:34.474319 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:34.474281 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:34.474517 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:34.474345 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:35.405456 ip-10-0-134-105 kubenswrapper[2577]: I1030 04:44:35.405291 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 04:44:35.405468 ip-10-0-134-105 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 04:44:35.421912 ip-10-0-134-105 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 04:44:35.422166 ip-10-0-134-105 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 04:44:35.422448 ip-10-0-134-105 systemd[1]: kubelet.service: Consumed 4.142s CPU time. Oct 30 04:44:35.442152 ip-10-0-134-105 systemd[1]: Starting Kubernetes Kubelet... Oct 30 04:44:35.501581 ip-10-0-134-105 kubenswrapper[6587]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 04:44:35.501581 ip-10-0-134-105 kubenswrapper[6587]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 04:44:35.501581 ip-10-0-134-105 kubenswrapper[6587]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 04:44:35.501581 ip-10-0-134-105 kubenswrapper[6587]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 04:44:35.503819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503738 6587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:44:35.503819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503820 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503824 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503827 6587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503830 6587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503834 6587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503837 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503840 6587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503842 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503845 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503848 6587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503851 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503854 6587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503856 6587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503859 6587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503862 6587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503864 6587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503868 6587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503870 6587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503873 6587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:44:35.503894 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503876 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503879 6587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503881 6587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503884 6587 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503892 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503895 6587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503898 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503901 6587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503903 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503906 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503909 6587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503911 6587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503915 6587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503919 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503922 6587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503925 6587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503928 6587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503930 6587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503933 6587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:44:35.504388 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503936 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503939 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503942 6587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503944 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503947 6587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503949 6587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503952 6587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503955 6587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503958 6587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503960 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503963 6587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503966 6587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503969 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503971 6587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503975 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503977 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503980 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503983 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503985 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:44:35.504858 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503988 6587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503991 6587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503993 6587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503996 6587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.503999 6587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504001 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504004 6587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504006 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504009 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504011 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504014 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504016 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504019 6587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504022 6587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504025 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504027 6587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504031 6587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504033 6587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504036 6587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:44:35.505344 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504039 6587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504041 6587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504044 6587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504048 6587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504051 6587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504054 6587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504057 6587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504060 6587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504063 6587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504065 6587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504068 6587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504071 6587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504074 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504076 6587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504079 6587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504081 6587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504085 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504088 6587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504090 6587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:44:35.505822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504093 6587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504096 6587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504099 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504102 6587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504105 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504107 6587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504110 6587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504112 6587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504115 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504118 6587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504869 6587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504876 6587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504879 6587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504882 6587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504885 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504887 6587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504890 6587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504893 6587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504895 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504898 6587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:44:35.506277 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504901 6587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504904 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504907 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504909 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504912 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504915 6587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504917 6587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504920 6587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504924 6587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504927 6587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504929 6587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504932 6587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504935 6587 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504937 6587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504940 6587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504942 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504945 6587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504948 6587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504950 6587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504953 6587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:44:35.506805 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504956 6587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504959 6587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504962 6587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504965 6587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504967 6587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504970 6587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504972 6587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504976 6587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504981 6587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504984 6587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504987 6587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504989 6587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504992 6587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504995 6587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.504997 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505000 6587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505002 6587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505005 6587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505008 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505011 6587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:44:35.507306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505018 6587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505021 6587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505023 6587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505026 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505028 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505031 6587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505033 6587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505036 6587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505039 6587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505041 6587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505044 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505047 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505049 6587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505052 6587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505054 6587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505057 6587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505061 6587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505064 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505067 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505069 6587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:44:35.507882 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505072 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505075 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505077 6587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505080 6587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505082 6587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505085 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505087 6587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505090 6587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505093 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505095 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505098 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505100 6587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505105 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505108 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505110 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505113 6587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505115 6587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505118 6587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505121 6587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:44:35.508398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505123 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505126 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505129 6587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505132 6587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505134 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505137 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505140 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505142 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505145 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505148 6587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505150 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505153 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505155 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505158 6587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505163 6587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505166 6587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505169 6587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505661 6587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505667 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:44:35.508872 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505670 6587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505673 6587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505676 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505679 6587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505682 6587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505684 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505688 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505691 6587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505694 6587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505696 6587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505699 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505702 6587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505704 6587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505707 6587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505709 6587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505712 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505715 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505718 6587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505720 6587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505722 6587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:44:35.509446 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505725 6587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505728 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505733 6587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505737 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505741 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505743 6587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505746 6587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505749 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505751 6587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505754 6587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505756 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505759 6587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505761 6587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505764 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505767 6587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505770 6587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505772 6587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505775 6587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505778 6587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:44:35.509947 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505780 6587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505782 6587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505785 6587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505788 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505790 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505793 6587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505795 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505798 6587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505800 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505805 6587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505808 6587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505811 6587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505816 6587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505819 6587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505821 6587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505824 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505827 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505830 6587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505833 6587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:44:35.510422 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505836 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505838 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505841 6587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505843 6587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505846 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505849 6587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505851 6587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505854 6587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505857 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505860 6587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505862 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505865 6587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505867 6587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505870 6587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505872 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505875 6587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505878 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505880 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505883 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:44:35.510893 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505885 6587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505888 6587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505890 6587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505893 6587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505897 6587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505899 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505906 6587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505909 6587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505911 6587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505914 6587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505916 6587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505919 6587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505922 6587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505925 6587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505928 6587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505930 6587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505933 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505935 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505938 6587 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505940 6587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:44:35.511353 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505943 6587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505945 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505948 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505950 6587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505953 6587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505955 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.505958 6587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506054 6587 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506069 6587 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506075 6587 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506080 6587 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506084 6587 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506088 6587 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506092 6587 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506097 6587 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506100 6587 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506103 6587 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506107 6587 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506112 6587 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506115 6587 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506118 6587 flags.go:64] FLAG: --cgroup-root="" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506121 6587 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 04:44:35.511868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506124 6587 flags.go:64] FLAG: --client-ca-file="" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506127 6587 flags.go:64] FLAG: --cloud-provider="external" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506131 6587 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506137 6587 flags.go:64] FLAG: --cluster-domain="" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506140 6587 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506143 6587 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506146 6587 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506150 6587 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506154 6587 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506157 6587 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506160 6587 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506163 6587 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506167 6587 flags.go:64] FLAG: --contention-profiling="false" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506170 6587 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506173 6587 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506177 6587 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506180 6587 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506185 6587 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506189 6587 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506192 6587 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506195 6587 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506198 6587 flags.go:64] FLAG: --enable-server="true" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506202 6587 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506205 6587 flags.go:64] FLAG: --event-burst="100" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506209 6587 flags.go:64] FLAG: --event-qps="50" Oct 30 04:44:35.512435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506212 6587 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506215 6587 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506218 6587 flags.go:64] FLAG: --eviction-hard="" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506223 6587 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506227 6587 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506230 6587 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506233 6587 flags.go:64] FLAG: --eviction-soft="" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506236 6587 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506239 6587 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506242 6587 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506246 6587 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506249 6587 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506252 6587 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506255 6587 flags.go:64] FLAG: --feature-gates="" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506260 6587 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506263 6587 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506266 6587 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506269 6587 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506272 6587 flags.go:64] FLAG: --healthz-port="10248" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506275 6587 flags.go:64] FLAG: --help="false" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506279 6587 flags.go:64] FLAG: --hostname-override="ip-10-0-134-105.ec2.internal" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506282 6587 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506285 6587 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 04:44:35.513119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506288 6587 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506293 6587 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506297 6587 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506300 6587 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506303 6587 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506306 6587 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506309 6587 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506312 6587 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506315 6587 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506318 6587 flags.go:64] FLAG: --kube-reserved="" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506321 6587 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506325 6587 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506329 6587 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506332 6587 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506335 6587 flags.go:64] FLAG: --lock-file="" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506338 6587 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506341 6587 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506344 6587 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506349 6587 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506353 6587 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506368 6587 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506371 6587 flags.go:64] FLAG: --logging-format="text" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506374 6587 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506377 6587 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 04:44:35.513915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506380 6587 flags.go:64] FLAG: --manifest-url="" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506383 6587 flags.go:64] FLAG: --manifest-url-header="" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506388 6587 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506391 6587 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506395 6587 flags.go:64] FLAG: --max-pods="110" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506398 6587 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506401 6587 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506404 6587 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506407 6587 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506412 6587 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506415 6587 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506418 6587 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506425 6587 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506428 6587 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506431 6587 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506434 6587 flags.go:64] FLAG: --pod-cidr="" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506437 6587 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506440 6587 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506444 6587 flags.go:64] FLAG: --pods-per-core="0" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506447 6587 flags.go:64] FLAG: --port="10250" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506452 6587 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506455 6587 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-042875b55d28515f0" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506458 6587 flags.go:64] FLAG: --qos-reserved="" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506461 6587 flags.go:64] FLAG: --read-only-port="10255" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506464 6587 flags.go:64] FLAG: --register-node="true" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506467 6587 flags.go:64] FLAG: --register-with-taints="" Oct 30 04:44:35.514548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506471 6587 flags.go:64] FLAG: --registry-burst="10" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506475 6587 flags.go:64] FLAG: --registry-qps="5" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506478 6587 flags.go:64] FLAG: --reserved-cpus="" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506482 6587 flags.go:64] FLAG: --reserved-memory="" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506486 6587 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506489 6587 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506493 6587 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506496 6587 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506499 6587 flags.go:64] FLAG: --runonce="false" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506502 6587 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506505 6587 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506508 6587 flags.go:64] FLAG: --seccomp-default="false" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506511 6587 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506515 6587 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506518 6587 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506522 6587 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506525 6587 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506529 6587 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506532 6587 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506535 6587 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506538 6587 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506541 6587 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506544 6587 flags.go:64] FLAG: --system-cgroups="" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506547 6587 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506552 6587 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 04:44:35.515189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506556 6587 flags.go:64] FLAG: --tls-cert-file="" Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506559 6587 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506564 6587 flags.go:64] FLAG: --tls-min-version="" Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506567 6587 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506570 6587 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506573 6587 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506576 6587 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506579 6587 flags.go:64] FLAG: --v="2" Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506588 6587 flags.go:64] FLAG: --version="false" Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506592 6587 flags.go:64] FLAG: --vmodule="" Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506596 6587 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.506606 6587 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506757 6587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506761 6587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506765 6587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506768 6587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506771 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506774 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506777 6587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506780 6587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506783 6587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506785 6587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506790 6587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:44:35.515819 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506792 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506798 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506802 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506804 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506807 6587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506810 6587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506813 6587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506816 6587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506818 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506821 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506825 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506828 6587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506831 6587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506833 6587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506836 6587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506839 6587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506841 6587 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506844 6587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506847 6587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:44:35.516464 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506849 6587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506852 6587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506855 6587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506858 6587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506861 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506863 6587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506866 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506869 6587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506871 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506874 6587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506877 6587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506879 6587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506883 6587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506886 6587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506889 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506891 6587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506894 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506896 6587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506899 6587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506902 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:44:35.517011 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506905 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506907 6587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506910 6587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506914 6587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506917 6587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506920 6587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506922 6587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506925 6587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506929 6587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506933 6587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506936 6587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506939 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506942 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506945 6587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506948 6587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506952 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506954 6587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506957 6587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506959 6587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:44:35.517745 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506962 6587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506965 6587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506967 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506970 6587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506972 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506977 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506980 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506984 6587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506988 6587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506991 6587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506994 6587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506996 6587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.506999 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507002 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507004 6587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507007 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507010 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507013 6587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507016 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:44:35.518262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507018 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507021 6587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507024 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507026 6587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507029 6587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507031 6587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507034 6587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507036 6587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507039 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507041 6587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507044 6587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507047 6587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507050 6587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507053 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507055 6587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507058 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507060 6587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:44:35.518754 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507063 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.507091 6587 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507124 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507128 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507132 6587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507135 6587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507138 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507140 6587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507143 6587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507146 6587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507148 6587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507151 6587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507155 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507158 6587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507160 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:44:35.519216 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507163 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507166 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507170 6587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507174 6587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507176 6587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507179 6587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507182 6587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507185 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507187 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507190 6587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507193 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507196 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507199 6587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507201 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507204 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507207 6587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507209 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507212 6587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507214 6587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:44:35.519755 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507218 6587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507221 6587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507223 6587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507226 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507229 6587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507231 6587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507234 6587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507237 6587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507240 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507242 6587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507246 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507249 6587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507251 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507254 6587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507257 6587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507260 6587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507262 6587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507265 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507267 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507270 6587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:44:35.520378 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507273 6587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507276 6587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507278 6587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507281 6587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507283 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507287 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507289 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507292 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507294 6587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507297 6587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507301 6587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507303 6587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507307 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507310 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507313 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507316 6587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507318 6587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507321 6587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507323 6587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507326 6587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:44:35.520885 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507329 6587 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507332 6587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507335 6587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507338 6587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507340 6587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507343 6587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507346 6587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507348 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507351 6587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507365 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507368 6587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507371 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507374 6587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507376 6587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507379 6587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507382 6587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507384 6587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507387 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507390 6587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507393 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:44:35.521636 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507396 6587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507399 6587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507401 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507404 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507408 6587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507411 6587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507414 6587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507416 6587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507419 6587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507422 6587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507424 6587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507430 6587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507434 6587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.507437 6587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.511033 6587 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.511049 6587 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.511066 6587 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.511071 6587 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511125 6587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:44:35.522133 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511131 6587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511134 6587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511138 6587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511141 6587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511144 6587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511147 6587 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511150 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511153 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511156 6587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511159 6587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511161 6587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511164 6587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511167 6587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511169 6587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511172 6587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511175 6587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511177 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511180 6587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511183 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511185 6587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:44:35.522628 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511188 6587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511191 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511193 6587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511198 6587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511202 6587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511205 6587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511208 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511212 6587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511215 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511218 6587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511220 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511223 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511226 6587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511229 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511231 6587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511234 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511236 6587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511239 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511242 6587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:44:35.523142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511244 6587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511247 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511249 6587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511252 6587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511255 6587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511257 6587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511260 6587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511262 6587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511265 6587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511268 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511270 6587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511273 6587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511275 6587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511278 6587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511281 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511283 6587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511286 6587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511288 6587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511291 6587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511294 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:44:35.523627 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511297 6587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511299 6587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511306 6587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511309 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511312 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511314 6587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511317 6587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511319 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511322 6587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511324 6587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511327 6587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511329 6587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511332 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511335 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511337 6587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511340 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511343 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511345 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511348 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:44:35.524102 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511350 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511373 6587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511377 6587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511380 6587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511383 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511386 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511388 6587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511391 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511394 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511397 6587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511400 6587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511402 6587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511405 6587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511408 6587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511411 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511413 6587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511417 6587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511420 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511423 6587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:44:35.524566 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511426 6587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511428 6587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511431 6587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511434 6587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511436 6587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511439 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511443 6587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511446 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.511474 6587 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511612 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511616 6587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511619 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511623 6587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511625 6587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511628 6587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511631 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:44:35.525025 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511634 6587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511638 6587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511641 6587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511644 6587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511647 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511650 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511652 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511655 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511658 6587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511661 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511663 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511667 6587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511671 6587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511674 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511677 6587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511680 6587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511682 6587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511685 6587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511687 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:44:35.525463 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511690 6587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511693 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511695 6587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511698 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511700 6587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511703 6587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511705 6587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511708 6587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511710 6587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511713 6587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511716 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511719 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511721 6587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511724 6587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511727 6587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511729 6587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511732 6587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511734 6587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511737 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511740 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:44:35.526057 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511742 6587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511745 6587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511747 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511750 6587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511753 6587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511756 6587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511758 6587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511762 6587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511765 6587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511767 6587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511770 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511772 6587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511775 6587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511777 6587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511780 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511782 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511785 6587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511787 6587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511790 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511792 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:44:35.526695 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511795 6587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511797 6587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511800 6587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511802 6587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511805 6587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511808 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511810 6587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511813 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511815 6587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511818 6587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511820 6587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511823 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511825 6587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511828 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511831 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511833 6587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511836 6587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511839 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511841 6587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:44:35.527398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511844 6587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511846 6587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511849 6587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511852 6587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511854 6587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511857 6587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511860 6587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511862 6587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511865 6587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511867 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511870 6587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511873 6587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511876 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511879 6587 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511882 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511884 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511887 6587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511890 6587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511892 6587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511895 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:44:35.528035 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511898 6587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.511920 6587 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511949 6587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511953 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511956 6587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511960 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511963 6587 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511965 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511968 6587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511971 6587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511974 6587 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511977 6587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511979 6587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511982 6587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511984 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511987 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 04:44:35.528730 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511990 6587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511993 6587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511995 6587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.511998 6587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512001 6587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512003 6587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512008 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512011 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512014 6587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512017 6587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512019 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512022 6587 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512025 6587 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512027 6587 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512030 6587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512032 6587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512035 6587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512037 6587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512040 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512043 6587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 04:44:35.529142 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512045 6587 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512048 6587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512053 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512055 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512058 6587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512060 6587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512063 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512066 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512069 6587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512071 6587 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512074 6587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512076 6587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512079 6587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512082 6587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512085 6587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512087 6587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512090 6587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512092 6587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512096 6587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 04:44:35.529670 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512099 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512102 6587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512104 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512107 6587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512110 6587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512113 6587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512115 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512118 6587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512120 6587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512123 6587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512125 6587 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512128 6587 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512130 6587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512133 6587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512136 6587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512140 6587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512143 6587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512145 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512148 6587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512151 6587 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 04:44:35.530504 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512153 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512156 6587 feature_gate.go:386] unrecognized feature gate: Example Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512159 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512161 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512164 6587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512166 6587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512170 6587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512172 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512175 6587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512178 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512180 6587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512185 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512188 6587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512191 6587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512193 6587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512196 6587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512198 6587 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512203 6587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512206 6587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 04:44:35.531287 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512209 6587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512211 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512214 6587 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512217 6587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512219 6587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512222 6587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512225 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512227 6587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512233 6587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512236 6587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512239 6587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512242 6587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512245 6587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:35.512248 6587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.512298 6587 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.514307 6587 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.514382 6587 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.514600 6587 server.go:1014] "Starting client certificate rotation" Oct 30 04:44:35.532027 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.514740 6587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 04:44:35.532807 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.514791 6587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 04:38:24 +0000 UTC" deadline="2028-06-02 22:53:37.447820009 +0000 UTC" Oct 30 04:44:35.532807 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.514820 6587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13962h9m1.933001382s" Oct 30 04:44:35.532807 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.515053 6587 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 04:44:35.532807 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.516170 6587 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 04:44:35.532807 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.517414 6587 log.go:25] "Validated CRI v1 runtime API" Oct 30 04:44:35.532807 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.519294 6587 log.go:25] "Validated CRI v1 image API" Oct 30 04:44:35.532807 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.520043 6587 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 04:44:35.532807 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.522150 6587 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80d5f8cf-d243-42b4-b064-f8c778cdff95:/dev/nvme0n1p3 a8d79c02-9e52-4928-80f5-5de84d21161a:/dev/nvme0n1p4] Oct 30 04:44:35.533272 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.522170 6587 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01790731e2d0f87cd94e012aa2e97bb2b15c0c9a2685a2f8eb23b2e77030552f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01790731e2d0f87cd94e012aa2e97bb2b15c0c9a2685a2f8eb23b2e77030552f/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01904cd114e8b175c949ee66fb71fa5825b062d6badf76ccc7454c0a6b6caf8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01904cd114e8b175c949ee66fb71fa5825b062d6badf76ccc7454c0a6b6caf8b/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/054fc3bd50256aea17d3928a2ce12dac51b47670a65f4d2465f322447af18ad9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/054fc3bd50256aea17d3928a2ce12dac51b47670a65f4d2465f322447af18ad9/userdata/shm major:0 minor:422 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06f85ee326de087a302fcc9708effe33e27a8bb2c2c69ad6b59f17c0d1573269/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06f85ee326de087a302fcc9708effe33e27a8bb2c2c69ad6b59f17c0d1573269/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11757ef7a495bf3d5004d99cceeffdb9a0372397be64ad23063b602881d7afed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11757ef7a495bf3d5004d99cceeffdb9a0372397be64ad23063b602881d7afed/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21151e3e52f1a5214af61f0bdff5bd7d88ec8013b5800cb2d503debd9411a1bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21151e3e52f1a5214af61f0bdff5bd7d88ec8013b5800cb2d503debd9411a1bf/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d9ce14a3de9bcef35191c32449e17558f07119f1058e47643822c1818cf7dda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d9ce14a3de9bcef35191c32449e17558f07119f1058e47643822c1818cf7dda/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f15a33a8778626d8aabfcd1354c427a12b4cb5f27c67fae30f68f00d5837ba9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f15a33a8778626d8aabfcd1354c427a12b4cb5f27c67fae30f68f00d5837ba9/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d7c1ca735f996da6f911dcd9020143c9ed5a27763f121c855f15488ee481e48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d7c1ca735f996da6f911dcd9020143c9ed5a27763f121c855f15488ee481e48/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/607d87808f6a84531e3056456f575abf85f0232df5f0bf4bf2fd221099254bb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/607d87808f6a84531e3056456f575abf85f0232df5f0bf4bf2fd221099254bb8/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/797a00f752240d1f11f68df576f507f691f0d0077e1efa40a17313303abcf7a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/797a00f752240d1f11f68df576f507f691f0d0077e1efa40a17313303abcf7a8/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/874dbe75e9771ce3c6b53301f52922ee6650dd91dfa8d697b6494b974bef508f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/874dbe75e9771ce3c6b53301f52922ee6650dd91dfa8d697b6494b974bef508f/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98f9342100e860dc012653575b82ea804606200ee5eedbc2b7cd9f3f44e5a9f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98f9342100e860dc012653575b82ea804606200ee5eedbc2b7cd9f3f44e5a9f7/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a75e02faa23c5d4cecec249e6a1e92841cb93dd0a066bc2c95b5b04262d4970f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a75e02faa23c5d4cecec249e6a1e92841cb93dd0a066bc2c95b5b04262d4970f/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af9d29a9ea9dc92cd9b476d44afe8091a22404a8f59fbef697d73a00b5fa1da0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af9d29a9ea9dc92cd9b476d44afe8091a22404a8f59fbef697d73a00b5fa1da0/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/beff413d95e8fa1eb732fbc49c2ffa9fc0d72f7c2ef2f9570f882e9849fe81c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/beff413d95e8fa1eb732fbc49c2ffa9fc0d72f7c2ef2f9570f882e9849fe81c7/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bfd57d4846df9970bbc704f67217e30528c7bdef8afbd2d22be99a8ca64cdd67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bfd57d4846df9970bbc704f67217e30528c7bdef8afbd2d22be99a8ca64cdd67/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0fb8002b53a36d7660e29b9f85de87d3922a1daab2c0190cf339b402a3bddf8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0fb8002b53a36d7660e29b9f85de87d3922a1daab2c0190cf339b402a3bddf8/userdata/shm major:0 minor:384 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5a218012154f5a9cc35276d76618c1a784cb8dbc68bf7c7f0c6e5e7b9ff3452/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5a218012154f5a9cc35276d76618c1a784cb8dbc68bf7c7f0c6e5e7b9ff3452/userdata/shm major:0 minor:426 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6fc85b3354aa0e1abff0bdf7a15c82db297a48cf3d3f04848ba68a164859593/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6fc85b3354aa0e1abff0bdf7a15c82db297a48cf3d3f04848ba68a164859593/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9e0eefb16820888b841272d737f204b42bd71323e7746efa239184d2a179cd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9e0eefb16820888b841272d737f204b42bd71323e7746efa239184d2a179cd4/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe096d629033c2c73bd8e35a5d9dd965c0d6ff00360f0a66aa127e05b07b0876/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe096d629033c2c73bd8e35a5d9dd965c0d6ff00360f0a66aa127e05b07b0876/userdata/shm major:0 minor:424 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volume-subpaths/web-config/alertmanager/9 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volume-subpaths/web-config/config-reloader/6 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~empty-dir/config-out major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~projected/kube-api-access-dkjb7:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~projected/kube-api-access-dkjb7 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~projected/tls-assets major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/config-volume major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/web-config major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09846780-4fb0-474a-bd15-9b272dc00772/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/09846780-4fb0-474a-bd15-9b272dc00772/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09846780-4fb0-474a-bd15-9b272dc00772/volumes/kubernetes.io~projected/kube-api-access-chwz8:{mountpoint:/var/lib/kubelet/pods/09846780-4fb0-474a-bd15-9b272dc00772/volumes/kubernetes.io~projected/kube-api-access-chwz8 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09846780-4fb0-474a-bd15-9b272dc00772/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/09846780-4fb0-474a-bd15-9b272dc00772/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c23f69d-0026-4202-884b-38a3d4bbe87d/volumes/kubernetes.io~projected/kube-api-access-hsv6f:{mountpoint:/var/lib/kubelet/pods/1c23f69d-0026-4202-884b-38a3d4bbe87d/volumes/kubernetes.io~projected/kube-api-access-hsv6f major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26dd286e-348a-4378-be3d-9687ba409fc4/volumes/kubernetes.io~projected/kube-api-access-hs5xk:{mountpoint:/var/lib/kubelet/pods/26dd286e-348a-4378-be3d-9687ba409fc4/volumes/kubernetes.io~projected/kube-api-access-hs5xk major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5/volumes/kubernetes.io~projected/kube-api-access-twfl6:{mountpoint:/var/lib/kubelet/pods/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5/volumes/kubernetes.io~projected/kube-api-access-twfl6 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2967a60d-8f1a-4084-ac8b-5c50109aaf3b/volumes/kubernetes.io~projected/kube-api-access-22gdq:{mountpoint:/var/lib/kubelet/pods/2967a60d-8f1a-4084-ac8b-5c50109aaf3b/volumes/kubernetes.io~projected/kube-api-access-22gdq major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2967a60d-8f1a-4084-ac8b-5c50109aaf3b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2967a60d-8f1a-4084-ac8b-5c50109aaf3b/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30011b22-59f1-4361-b961-c5be6de91e88/volumes/kubernetes.io~projected/kube-api-access-zdhkb:{mountpoint:/var/lib/kubelet/pods/30011b22-59f1-4361-b961-c5be6de91e88/volumes/kubernetes.io~projected/kube-api-access-zdhkb major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30011b22-59f1-4361-b961-c5be6de91e88/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/30011b22-59f1-4361-b961-c5be6de91e88/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30011b22-59f1-4361-b961-c5be6de91e88/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/30011b22-59f1-4361-b961-c5be6de91e88/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32ed1e66-99e0-4714-92d1-1ef6213fed2c/volumes/kubernetes.io~projected/kube-api-access-dd5t4:{mountpoint:/var/lib/kubelet/pods/32ed1e66-99e0-4714-92d1-1ef6213fed2c/volumes/kubernetes.io~projected/kube-api-access-dd5t4 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32ed1e66-99e0-4714-92d1-1ef6213fed2c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/32ed1e66-99e0-4714-92d1-1ef6213fed2c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32ed1e66-99e0-4714-92d1-1ef6213fed2c/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/32ed1e66-99e0-4714-92d1-1ef6213fed2c/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54628fc4-6d74-4048-ab60-61a643aac200/volumes/kubernetes.io~projected/kube-api-access-8wbcp:{mountpoint:/var/lib/kubelet/pods/54628fc4-6d74-4048-ab60-61a643aac200/volumes/kubernetes.io~projected/kube-api-access-8wbcp major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54628fc4-6d74-4048-ab60-61a643aac200/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/54628fc4-6d74-4048-ab60-61a643aac200/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59632489-f9ac-44e1-ae2a-7a119c0ece48/volumes/kubernetes.io~projected/kube-api-access-5rb56:{mountpoint:/var/lib/kubelet/pods/59632489-f9ac-44e1-ae2a-7a119c0ece48/volumes/kubernetes.io~projected/kube-api-access-5rb56 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67f6fc33-45c9-4579-ab82-039cb764b155/volumes/kubernetes.io~projected/kube-api-access-tgbkv:{mountpoint:/var/lib/kubelet/pods/67f6fc33-45c9-4579-ab82-039cb764b155/volumes/kubernetes.io~projected/kube-api-access-tgbkv major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7115245a-cbc9-4180-9709-2a6e590bb170/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7115245a-cbc9-4180-9709-2a6e590bb170/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/926ec2b0-7229-4297-8af9-bcd48ceab12a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/926ec2b0-7229-4297-8af9-bcd48ceab12a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/926ec2b0-7229-4297-8af9-bcd48ceab12a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/926ec2b0-7229-4297-8af9-bcd48ceab12a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/926ec2b0-7229-4297-8af9-bcd48ceab12a/volumes/kubernetes.io~projected/kube-api-access-p9zsw:{mountpoint:/var/lib/kubelet/pods/926ec2b0-7229-4297-8af9-bcd48ceab12a/volumes/kubernetes.io~projected/kube-api-access-p9zsw major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c1d23d7-96c3-4406-9354-860812b8d2bb/volumes/kubernetes.io~projected/kube-api-access-827bt:{mountpoint:/var/lib/kubelet/pods/9c1d23d7-96c3-4406-9354-860812b8d2bb/volumes/kubernetes.io~projected/kube-api-access-827bt major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f09ff3d-66bc-46bc-b404-9a45acd58e1d/volumes/kubernetes.io~projected/kube-api-access-xgvbs:{mountpoint:/var/lib/kubelet/pods/9f09ff3d-66bc-46bc-b404-9a45acd58e1d/volumes/kubernetes.io~projected/kube-api-access-xgvbs major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f09ff3d-66bc-46bc-b404-9a45acd58e1d/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/9f09ff3d-66bc-46bc-b404-9a45acd58e1d/volumes/kubernetes.io~secret/metrics-certs major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0780f4f-55b4-49c6-80a1-4026cc98f858/volumes/kubernetes.io~projected/kube-api-access-mt74j:{mountpoint:/var/lib/kubelet/pods/a0780f4f-55b4-49c6-80a1-4026cc98f858/volumes/kubernetes.io~projected/kube-api-access-mt74j major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1e09016-1f64-4c16-be6b-ab60b33a13e8/volumes/kubernetes.io~projected/kube-api-access-dpl2b:{mountpoint:/var/lib/kubelet/pods/b1e09016-1f64-4c16-be6b-ab60b33a13e8/volumes/kubernetes.io~projected/kube-api-access-dpl2b major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1e09016-1f64-4c16-be6b-ab60b33a13e8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b1e09016-1f64-4c16-be6b-ab60b33a13e8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1e09016-1f64-4c16-be6b-ab60b33a13e8/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/b1e09016-1f64-4c16-be6b-ab60b33a13e8/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c669d496-909b-428e-9b02-1ba9b05fd718/volumes/kubernetes.io~projected/kube-api-access-b5c8z:{mountpoint:/var/lib/kubelet/pods/c669d496-909b-428e-9b02-1ba9b05fd718/volumes/kubernetes.io~projected/kube-api-access-b5c8z major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c669d496-909b-428e-9b02-1ba9b05fd718/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/c669d496-909b-428e-9b02-1ba9b05fd718/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9fd93f4-68f8-4711-ad54-410d10af6c35/volumes/kubernetes.io~projected/kube-api-access-nn4sc:{mountpoint:/var/lib/kubelet/pods/d9fd93f4-68f8-4711-ad54-410d10af6c35/volumes/kubernetes.io~projected/kube-api-access-nn4sc major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9fd93f4-68f8-4711-ad54-410d10af6c35/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/d9fd93f4-68f8-4711-ad54-410d10af6c35/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9fd93f4-68f8-4711-ad54-410d10af6c35/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/d9fd93f4-68f8-4711-ad54-410d10af6c35/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9fd93f4-68f8-4711-ad54-410d10af6c35/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/d9fd93f4-68f8-4711-ad54-410d10af6c35/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volume-subpaths/web-config/config-reloader/2 major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volume-subpaths/web-config/prometheus/15 major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~empty-dir/config-out major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~projected/kube-api-access-l9w2d:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~projected/kube-api-access-l9w2d major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~projected/tls-assets major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/config major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/web-config major:0 minor:381 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ae39e8e88e81012fd42a301f1bc4f45a9374affdda90299d48e5be31f132930f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/42169376f4857b6b161c811877aace509921b0678d8ae12198d7e3593f446231/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1801603b40784d98529bb155b8677ef4ad54c1371b22d635f68a4fb403ec34ea/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ec4abb88ac4548dd1126590573e39fc77e227c1b6185c095b21bee816d1a4238/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/17ffc5bd378e1282931952343a61d031f055476f36b85869167a05465a642f0d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4259876c4df08ca4f61f0ffcec3f226d89aef66c3a5e8de14f1ae20ef0fbc5f2/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/50d7168cc0c077b3c72890f36c4e964efcd3ec35d64b268a591913daad18512c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/09cce26c0c8fe48acd3d3c1e08865a524f3857283b3a25ebd03886030db3f25d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4fc342d5acb551049ed807f063e3ae4fe1cd8ce87d2f357142bc522bd8ea8aa3/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/16d8faa68e501f79374b9506d3ae285da5852ab7296e5fcf7053ef910eebc94e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/b34b0c713db6ac459208f83f713c340f122a2ab07760b45362677339ab0479d8/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/e48517e7545267fbbcff1f7d5689bc24578ab66c00d3d2196380c3ec15ed6df5/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/185c6ca0bd28a2ecff4f78d1e27c945b1b0850a585fe4b7d288f24279efe1828/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e3357d3817fc7ad554e28b1614a8100c2d79511dac1bb9581c8dcb29bbd89556/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c927ff9ad1068f88e943ddee1e27828f62868077ef72859ee8d2b9cc72b4f0c5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/59d99f72a61f15212a051a46731f6ed15171bf4018d3355fda8994391c479538/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3841a25a3ab01b3f4ab02603cd5aa5d6a5bb9e88e846ca937d660ed3e352a411/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b73a74a72473570aa42aae952c391468e814da5f5c79865dcedb7d20b914f959/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/cfe2a92bf0d7c838ba46997f4a5719962a03f53721fe45f1a1ac52e53e58bfbc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/68aaefe149cd2c22b67498103bde20b0d76e7889f357bdb2b64c87cdf43ce8a8/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/da2ca3c74d2eefa1c87659a5b1df7d38022afd214407d75054033ecff01959f9/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/add3eac9c2008110e4f8fe47ab217089cc775cfec78b096a219453a63b650b1b/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/aa5c4f0bbb289c5ba208218302e70f9cbae8825182d1355f9d8da8fd815a0dee/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/d52f65a7728fc042909f14f24073385c06317dec57ca3b4258ed8e1c0e3a5f81/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/3c997c960f2f968eb16a45be80cda5babde94a07d66a68797b37cb60d611bdb8/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/538bae13325fd66aa45cfada3b39a8493ce4f72e9cd40630e2c92d466474994c/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/2e871a10e52c90a46b25bac527550d420a2382641a89a725eb2eb0ced60137a7/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/dd9706ffa8f53a26540ec2ec7f30d2f5d239bd8eaea95a783ac8e9000c891eaf/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/d54d434e860678211c8d12d986a6b0f2500cb5763fad033dc86ab8ff47c62af5/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/869664d2df1e8f0be918995c57609d52872d8f2f10d57e3dabf35679efb5d8c3/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/1bfcf3515abb16da429ddf2f382d7acd489f304b071fddeac5e751a077bad404/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/55daef1009f4caceeeb27714f53f8f97d3e520c12204b26c5d6a348b2dca3465/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/f78e2227f14714c9943ce315380f9d6b8d8e047a00b99761ff6fcb792e4a3784/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/cb029efa6095aaedacf59c66e1a3f9339ed80eb1bac71bca501f26135146ff77/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/9c6526e8ecd81fc6d1e4885feabdd116f286f4450f02a33211a7e8e3567ad1f1/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/05fef7be1120cd6a214ce31ea041ff93721d2fe3af24619ceda296e945525b65/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/ccfb93eb653d2299ed7769ccbf82c69030008ec00dbac042e8c54c0b3180f54a/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/09e467f83900cf4f985e42e1a3b8c2d92b657e7eaed2788cb2c0ac04989b381b/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/7319193f0891d7dd7678f4040c909da18799b5a8fa6465b1ec3b5ce8f836091b/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/3df8662e580c465d9741113b4dd1959de49d18d51d32bb02969a2f84d1440b67/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/9200c1994c88ee2a07a5d1284f1ee850d8f488f71a11d2c85c318047a504153e/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/069c44dffd488b1f49a1f3d6c149aa13def9ccefc55a67c0a83b82e2c92f2617/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/477ebdfbaf6f4ba8d5e534d32ba3a3a4976eaa0266efdb29f74dd28fd75f074a/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/26fa11663f7304b0700c7021fa6337b8dad98abd3179b38323fe9ea879efa1e8/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/58f7ac53565e786654ad4dc586422151e1e9790f6e75e15057a3298e17630dc7/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/db2ed8bec7258ea4d08f1d1ba21efe724210c0a2933caa1143b51eb73b7e689a/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/21ae69986a7fbc1a8d8c3fbf6436c1b4d821572d6295ef01355f05e2aa6a9bbe/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/29dfccf8436e2c0b1d5f5e74944bee5c2859797152af5d0b83525293ec65388e/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/5edba4ba5c2be14686b921fa4964ae751e3400912ac19e619714a7516f80517f/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/4cf49f4b994c2f290c7d7983ec2242c52c53aaa012c904e5c30fa644b8c9a986/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/ed5ec6870d691739bdea5cfd2905a3dd7cccfb7d916c2f89cde107f850945e17/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/7b40ea05eccd96a07f020d6fe161cc473da7bf6ae2f903f870f4841a81dc7c1d/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/ab003689591b39402dca005f3db027cb9570624ecbb438295011cd8aa3bf970f/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/9d659ff49193f1a8a8d656bf54924dc4aa9e508b695c5a2b6a294ec4db696f35/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/63ead67d3a2c2f6fd32eb7656ca18cd03515eb6ee3190da839372c70e1fe82f2/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/620c69bf525d8f6983e4c5b494b73066e11dbe21fe3f43d24977bf5865db5fe8/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/9c807c6486b0d96d46b233e96b6f554fe923e0812698994ce3bfb12ac41af33d/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/fe4cb68b4f79aeaeb9ddbd4ed51719e725381c70acfd9af7d4c185c4042887a6/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/b3a16abe816602e4a87577ee4930382e6e4433b0d8463a2412d49af154603af1/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/2e37a012d6e21a834a928c33d423d1cf56168164141e6aeebbf2c5856b4e2d91/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/c7f05f557fa20e5e290a04b78dd931c192affbafa2c1dec819263172bb7dbd75/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/8ed90745a1c80bd08e0a7801e7293e75c9525d4ec015f3bcc3132e9dfb675efe/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/03f755e529b3a5291f6dc22e5b169c9f9e9f7b36cd84308ea4400ae8c9afcb1e/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/157f894182d6fe5d61327f5a33eafd15175e1b0620d4677496a9ea6e3f9290e5/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/78b9b564200fe322372b7e6ad3f5ef321ede63e03ae4cd4858962ca2024aeb5c/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d905534442a604b6ece109b338db7e1ed0dff18accd1f4b284e736d062c91152/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/f5fa8b7eb6ef698e0c205063f6245966181d6bc8ebeb3ef01761caa7fdc43797/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/42e2c10bdb153f195a48c69785630a7b0ef123bee1e435f21c46971ca102d5b5/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/c30684b4fdd1f160d5dc016a4b6af7b4b1597f0396296ce3c285960f58600131/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/62c6a47f317fb902c2bff62f9e2ce2f7d8cd4b9ed4d33b9c4f2fb8ae7c248d1a/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/4668a0628192ccd0cf950678de8782cadc115e346f1a1b41faaabb53286185f2/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/ef41a8feca0e72d5e59fda90c49b287af7938dc0016c3bd748c9d9ba36f0a1b1/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/4235d550aacd42f892e75f127faa6930011b6937e725d1ef163364597ef2a797/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/1be91381a115462170268a83cff82a7ff08300418e23e4e982b13cafaf323f32/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.529325 6587 manager.go:217] Machine: {Timestamp:2026-10-30 04:44:35.528857106 +0000 UTC m=+0.056623882 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100481 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2207bdb34566e6b0457703ffad80de SystemUUID:ec2207bd-b345-66e6-b045-7703ffad80de BootID:d28d25c9-a734-4d36-88b3-1f9ca1158323 Filesystems:[{Device:/var/lib/kubelet/pods/926ec2b0-7229-4297-8af9-bcd48ceab12a/volumes/kubernetes.io~projected/kube-api-access-p9zsw DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54628fc4-6d74-4048-ab60-61a643aac200/volumes/kubernetes.io~projected/kube-api-access-8wbcp DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:416 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe096d629033c2c73bd8e35a5d9dd965c0d6ff00360f0a66aa127e05b07b0876/userdata/shm DeviceMajor:0 DeviceMinor:424 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f15a33a8778626d8aabfcd1354c427a12b4cb5f27c67fae30f68f00d5837ba9/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11757ef7a495bf3d5004d99cceeffdb9a0372397be64ad23063b602881d7afed/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c669d496-909b-428e-9b02-1ba9b05fd718/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30011b22-59f1-4361-b961-c5be6de91e88/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/30011b22-59f1-4361-b961-c5be6de91e88/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:231 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:287 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67f6fc33-45c9-4579-ab82-039cb764b155/volumes/kubernetes.io~projected/kube-api-access-tgbkv DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a0780f4f-55b4-49c6-80a1-4026cc98f858/volumes/kubernetes.io~projected/kube-api-access-mt74j DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/797a00f752240d1f11f68df576f507f691f0d0077e1efa40a17313303abcf7a8/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98f9342100e860dc012653575b82ea804606200ee5eedbc2b7cd9f3f44e5a9f7/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1e09016-1f64-4c16-be6b-ab60b33a13e8/volumes/kubernetes.io~projected/kube-api-access-dpl2b DeviceMajor:0 DeviceMinor:234 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:285 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af9d29a9ea9dc92cd9b476d44afe8091a22404a8f59fbef697d73a00b5fa1da0/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:375 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:290 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9fd93f4-68f8-4711-ad54-410d10af6c35/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:324 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/beff413d95e8fa1eb732fbc49c2ffa9fc0d72f7c2ef2f9570f882e9849fe81c7/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/26dd286e-348a-4378-be3d-9687ba409fc4/volumes/kubernetes.io~projected/kube-api-access-hs5xk DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/607d87808f6a84531e3056456f575abf85f0232df5f0bf4bf2fd221099254bb8/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32ed1e66-99e0-4714-92d1-1ef6213fed2c/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d9fd93f4-68f8-4711-ad54-410d10af6c35/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:328 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:379 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:381 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2967a60d-8f1a-4084-ac8b-5c50109aaf3b/volumes/kubernetes.io~projected/kube-api-access-22gdq DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:373 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1c23f69d-0026-4202-884b-38a3d4bbe87d/volumes/kubernetes.io~projected/kube-api-access-hsv6f DeviceMajor:0 DeviceMinor:421 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/054fc3bd50256aea17d3928a2ce12dac51b47670a65f4d2465f322447af18ad9/userdata/shm DeviceMajor:0 DeviceMinor:422 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:291 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:382 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09846780-4fb0-474a-bd15-9b272dc00772/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9c1d23d7-96c3-4406-9354-860812b8d2bb/volumes/kubernetes.io~projected/kube-api-access-827bt DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/874dbe75e9771ce3c6b53301f52922ee6650dd91dfa8d697b6494b974bef508f/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b1e09016-1f64-4c16-be6b-ab60b33a13e8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:233 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5a218012154f5a9cc35276d76618c1a784cb8dbc68bf7c7f0c6e5e7b9ff3452/userdata/shm DeviceMajor:0 DeviceMinor:426 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/926ec2b0-7229-4297-8af9-bcd48ceab12a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32ed1e66-99e0-4714-92d1-1ef6213fed2c/volumes/kubernetes.io~projected/kube-api-access-dd5t4 DeviceMajor:0 DeviceMinor:235 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5/volumes/kubernetes.io~projected/kube-api-access-twfl6 DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bfd57d4846df9970bbc704f67217e30528c7bdef8afbd2d22be99a8ca64cdd67/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:376 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1e09016-1f64-4c16-be6b-ab60b33a13e8/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~projected/kube-api-access-dkjb7 DeviceMajor:0 DeviceMinor:278 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~projected/kube-api-access-l9w2d DeviceMajor:0 DeviceMinor:383 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f09ff3d-66bc-46bc-b404-9a45acd58e1d/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:420 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:380 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:381 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6fc85b3354aa0e1abff0bdf7a15c82db297a48cf3d3f04848ba68a164859593/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06f85ee326de087a302fcc9708effe33e27a8bb2c2c69ad6b59f17c0d1573269/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:374 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:381 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01904cd114e8b175c949ee66fb71fa5825b062d6badf76ccc7454c0a6b6caf8b/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a75e02faa23c5d4cecec249e6a1e92841cb93dd0a066bc2c95b5b04262d4970f/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09846780-4fb0-474a-bd15-9b272dc00772/volumes/kubernetes.io~projected/kube-api-access-chwz8 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21151e3e52f1a5214af61f0bdff5bd7d88ec8013b5800cb2d503debd9411a1bf/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54628fc4-6d74-4048-ab60-61a643aac200/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9e0eefb16820888b841272d737f204b42bd71323e7746efa239184d2a179cd4/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:368 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2967a60d-8f1a-4084-ac8b-5c50109aaf3b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c669d496-909b-428e-9b02-1ba9b05fd718/volumes/kubernetes.io~projected/kube-api-access-b5c8z DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d9ce14a3de9bcef35191c32449e17558f07119f1058e47643822c1818cf7dda/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/59632489-f9ac-44e1-ae2a-7a119c0ece48/volumes/kubernetes.io~projected/kube-api-access-5rb56 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9f09ff3d-66bc-46bc-b404-9a45acd58e1d/volumes/kubernetes.io~projected/kube-api-access-xgvbs DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/30011b22-59f1-4361-b961-c5be6de91e88/volumes/kubernetes.io~projected/kube-api-access-zdhkb DeviceMajor:0 DeviceMinor:236 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01790731e2d0f87cd94e012aa2e97bb2b15c0c9a2685a2f8eb23b2e77030552f/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:377 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/926ec2b0-7229-4297-8af9-bcd48ceab12a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d7c1ca735f996da6f911dcd9020143c9ed5a27763f121c855f15488ee481e48/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/32ed1e66-99e0-4714-92d1-1ef6213fed2c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:232 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/7115245a-cbc9-4180-9709-2a6e590bb170/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0fb8002b53a36d7660e29b9f85de87d3922a1daab2c0190cf339b402a3bddf8/userdata/shm DeviceMajor:0 DeviceMinor:384 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:381 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9fd93f4-68f8-4711-ad54-410d10af6c35/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d9fd93f4-68f8-4711-ad54-410d10af6c35/volumes/kubernetes.io~projected/kube-api-access-nn4sc DeviceMajor:0 DeviceMinor:330 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09846780-4fb0-474a-bd15-9b272dc00772/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:378 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01790731e2d0f87 MacAddress:fa:7d:a7:55:30:d6 Speed:10000 Mtu:8901} {Name:054fc3bd50256ae MacAddress:82:60:b9:8b:4d:11 Speed:10000 Mtu:8901} {Name:06f85ee326de087 MacAddress:76:59:74:87:6c:f6 Speed:10000 Mtu:8901} {Name:2d9ce14a3de9bce MacAddress:22:8c:74:5b:47:05 Speed:10000 Mtu:8901} {Name:4d7c1ca735f996d MacAddress:36:d4:ac:9b:9b:a0 Speed:10000 Mtu:8901} {Name:a75e02faa23c5d4 MacAddress:9e:0f:0a:33:90:18 Speed:10000 Mtu:8901} {Name:af9d29a9ea9dc92 MacAddress:0e:7e:ed:a2:d8:91 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cc:4c:04:95 Speed:0 Mtu:9001} {Name:br-int MacAddress:5e:a5:e6:38:59:5c Speed:0 Mtu:8901} {Name:d0fb8002b53a36d MacAddress:e2:31:33:bf:65:9e Speed:10000 Mtu:8901} {Name:d5a218012154f5a MacAddress:e6:59:8c:13:8f:ce Speed:10000 Mtu:8901} {Name:d9e0eefb1682088 MacAddress:2e:bd:02:be:eb:df Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cc:4c:04:95 Speed:0 Mtu:9001} {Name:fe096d629033c2c MacAddress:76:a5:fe:d3:b5:36 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:f2:09:74:a2:9d:e2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d2:98:67:8d:cc:7a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.529737 6587 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.529781 6587 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.530302 6587 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.530347 6587 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.530672 6587 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.530692 6587 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.530706 6587 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.530733 6587 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.530856 6587 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.531443 6587 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.531572 6587 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.531609 6587 kubelet.go:493] "Attempting to sync node with API server" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.531623 6587 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.531634 6587 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.531643 6587 kubelet.go:405] "Adding apiserver pod source" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.531651 6587 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.532300 6587 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.532311 6587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 04:44:35.533734 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.532771 6587 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 04:44:35.535327 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535309 6587 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 04:44:35.535327 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535321 6587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 04:44:35.535496 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535443 6587 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 04:44:35.535635 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535622 6587 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 04:44:35.535695 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535651 6587 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 04:44:35.535743 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535735 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 04:44:35.535777 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535751 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 04:44:35.535777 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535757 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 04:44:35.535777 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535763 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 04:44:35.535777 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535768 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 04:44:35.535777 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535774 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 04:44:35.535969 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535781 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 04:44:35.535969 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535788 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 04:44:35.535969 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535797 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 04:44:35.535969 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535805 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 04:44:35.535969 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535813 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 04:44:35.535969 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535822 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 04:44:35.535969 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535847 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 04:44:35.535969 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.535853 6587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 04:44:35.539313 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.539297 6587 server.go:1263] "Started kubelet" Oct 30 04:44:35.540396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.539421 6587 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 04:44:35.540396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.540072 6587 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 04:44:35.540396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.540153 6587 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 04:44:35.540585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.540534 6587 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 04:44:35.540946 ip-10-0-134-105 systemd[1]: Started Kubernetes Kubelet. Oct 30 04:44:35.542062 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.542027 6587 server.go:317] "Adding debug handlers to kubelet server" Oct 30 04:44:35.547054 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.546719 6587 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 04:44:35.547054 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.546754 6587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 04:44:35.547528 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.547508 6587 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 04:44:35.547528 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.547529 6587 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 04:44:35.547648 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.547570 6587 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 04:44:35.547648 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.547597 6587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 04:38:26 +0000 UTC" deadline="2028-07-26 23:42:28.568452854 +0000 UTC" Oct 30 04:44:35.547648 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.547633 6587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15258h57m53.020822891s" Oct 30 04:44:35.547788 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:35.547662 6587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-105.ec2.internal\" not found" Oct 30 04:44:35.548697 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.548680 6587 factory.go:55] Registering systemd factory Oct 30 04:44:35.548697 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.548698 6587 factory.go:223] Registration of the systemd container factory successfully Oct 30 04:44:35.549177 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549158 6587 factory.go:153] Registering CRI-O factory Oct 30 04:44:35.549261 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549180 6587 factory.go:223] Registration of the crio container factory successfully Oct 30 04:44:35.549261 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549249 6587 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 04:44:35.549386 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549285 6587 factory.go:103] Registering Raw factory Oct 30 04:44:35.549386 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549301 6587 manager.go:1199] Started watching for new ooms in manager Oct 30 04:44:35.549657 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549644 6587 manager.go:319] Starting recovery of all containers Oct 30 04:44:35.549728 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549702 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c669d496-909b-428e-9b02-1ba9b05fd718" volumeName="kubernetes.io/empty-dir/c669d496-909b-428e-9b02-1ba9b05fd718-tmp-dir" seLinuxMountContext="" Oct 30 04:44:35.549760 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549731 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9fd93f4-68f8-4711-ad54-410d10af6c35" volumeName="kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 04:44:35.549760 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549740 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-grpc-tls" seLinuxMountContext="" Oct 30 04:44:35.549760 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549748 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 04:44:35.549760 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549756 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="084e8194-89a6-4702-af2a-34df80b46792" volumeName="kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 04:44:35.549864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549764 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="084e8194-89a6-4702-af2a-34df80b46792" volumeName="kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-main-db" seLinuxMountContext="" Oct 30 04:44:35.549864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549771 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c23f69d-0026-4202-884b-38a3d4bbe87d" volumeName="kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f" seLinuxMountContext="" Oct 30 04:44:35.549864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549779 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30011b22-59f1-4361-b961-c5be6de91e88" volumeName="kubernetes.io/secret/30011b22-59f1-4361-b961-c5be6de91e88-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 04:44:35.549864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549799 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32ed1e66-99e0-4714-92d1-1ef6213fed2c" volumeName="kubernetes.io/secret/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 04:44:35.549864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549806 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32ed1e66-99e0-4714-92d1-1ef6213fed2c" volumeName="kubernetes.io/secret/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 04:44:35.549864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549812 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 04:44:35.549864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549819 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-web-config" seLinuxMountContext="" Oct 30 04:44:35.549864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549826 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30011b22-59f1-4361-b961-c5be6de91e88" volumeName="kubernetes.io/configmap/30011b22-59f1-4361-b961-c5be6de91e88-metrics-client-ca" seLinuxMountContext="" Oct 30 04:44:35.549864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549833 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="084e8194-89a6-4702-af2a-34df80b46792" volumeName="kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-config-volume" seLinuxMountContext="" Oct 30 04:44:35.549864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549845 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27bfe4b2-2f4e-46ec-be37-830a9f5c20f5" volumeName="kubernetes.io/projected/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-kube-api-access-twfl6" seLinuxMountContext="" Oct 30 04:44:35.549864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549851 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1e09016-1f64-4c16-be6b-ab60b33a13e8" volumeName="kubernetes.io/projected/b1e09016-1f64-4c16-be6b-ab60b33a13e8-kube-api-access-dpl2b" seLinuxMountContext="" Oct 30 04:44:35.549864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549859 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9fd93f4-68f8-4711-ad54-410d10af6c35" volumeName="kubernetes.io/configmap/d9fd93f4-68f8-4711-ad54-410d10af6c35-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 04:44:35.549864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549867 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-config-out" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549874 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549881 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="084e8194-89a6-4702-af2a-34df80b46792" volumeName="kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549888 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67f6fc33-45c9-4579-ab82-039cb764b155" volumeName="kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549895 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67f6fc33-45c9-4579-ab82-039cb764b155" volumeName="kubernetes.io/projected/67f6fc33-45c9-4579-ab82-039cb764b155-kube-api-access-tgbkv" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549903 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c1d23d7-96c3-4406-9354-860812b8d2bb" volumeName="kubernetes.io/projected/9c1d23d7-96c3-4406-9354-860812b8d2bb-kube-api-access-827bt" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549909 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1e09016-1f64-4c16-be6b-ab60b33a13e8" volumeName="kubernetes.io/configmap/b1e09016-1f64-4c16-be6b-ab60b33a13e8-metrics-client-ca" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549919 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1e09016-1f64-4c16-be6b-ab60b33a13e8" volumeName="kubernetes.io/empty-dir/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-textfile" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549927 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549935 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549943 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="084e8194-89a6-4702-af2a-34df80b46792" volumeName="kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-tls-assets" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549953 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54628fc4-6d74-4048-ab60-61a643aac200" volumeName="kubernetes.io/empty-dir/54628fc4-6d74-4048-ab60-61a643aac200-data-volume" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549960 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1e09016-1f64-4c16-be6b-ab60b33a13e8" volumeName="kubernetes.io/configmap/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549968 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c669d496-909b-428e-9b02-1ba9b05fd718" volumeName="kubernetes.io/configmap/c669d496-909b-428e-9b02-1ba9b05fd718-config-volume" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549975 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.549993 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-tls-assets" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550001 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09846780-4fb0-474a-bd15-9b272dc00772" volumeName="kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-ovnkube-config" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550011 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2967a60d-8f1a-4084-ac8b-5c50109aaf3b" volumeName="kubernetes.io/secret/2967a60d-8f1a-4084-ac8b-5c50109aaf3b-cert" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550018 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="084e8194-89a6-4702-af2a-34df80b46792" volumeName="kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-cluster-tls-config" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550028 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32ed1e66-99e0-4714-92d1-1ef6213fed2c" volumeName="kubernetes.io/empty-dir/32ed1e66-99e0-4714-92d1-1ef6213fed2c-volume-directive-shadow" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550035 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54628fc4-6d74-4048-ab60-61a643aac200" volumeName="kubernetes.io/secret/54628fc4-6d74-4048-ab60-61a643aac200-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550044 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1e09016-1f64-4c16-be6b-ab60b33a13e8" volumeName="kubernetes.io/secret/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-tls" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550051 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9fd93f4-68f8-4711-ad54-410d10af6c35" volumeName="kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550059 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550066 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="084e8194-89a6-4702-af2a-34df80b46792" volumeName="kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-kube-api-access-dkjb7" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550073 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26dd286e-348a-4378-be3d-9687ba409fc4" volumeName="kubernetes.io/configmap/26dd286e-348a-4378-be3d-9687ba409fc4-cni-binary-copy" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550080 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59632489-f9ac-44e1-ae2a-7a119c0ece48" volumeName="kubernetes.io/empty-dir/59632489-f9ac-44e1-ae2a-7a119c0ece48-tmp-dir" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550088 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7115245a-cbc9-4180-9709-2a6e590bb170" volumeName="kubernetes.io/secret/7115245a-cbc9-4180-9709-2a6e590bb170-agent-certs" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550096 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550103 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550110 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30011b22-59f1-4361-b961-c5be6de91e88" volumeName="kubernetes.io/secret/30011b22-59f1-4361-b961-c5be6de91e88-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550118 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32ed1e66-99e0-4714-92d1-1ef6213fed2c" volumeName="kubernetes.io/configmap/32ed1e66-99e0-4714-92d1-1ef6213fed2c-metrics-client-ca" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550126 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1e09016-1f64-4c16-be6b-ab60b33a13e8" volumeName="kubernetes.io/secret/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550133 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9fd93f4-68f8-4711-ad54-410d10af6c35" volumeName="kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-client-ca-bundle" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550140 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-config" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550148 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26dd286e-348a-4378-be3d-9687ba409fc4" volumeName="kubernetes.io/projected/26dd286e-348a-4378-be3d-9687ba409fc4-kube-api-access-hs5xk" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550155 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32ed1e66-99e0-4714-92d1-1ef6213fed2c" volumeName="kubernetes.io/configmap/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550171 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32ed1e66-99e0-4714-92d1-1ef6213fed2c" volumeName="kubernetes.io/projected/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-api-access-dd5t4" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550180 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54628fc4-6d74-4048-ab60-61a643aac200" volumeName="kubernetes.io/projected/54628fc4-6d74-4048-ab60-61a643aac200-kube-api-access-8wbcp" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550196 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67f6fc33-45c9-4579-ab82-039cb764b155" volumeName="kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550204 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7115245a-cbc9-4180-9709-2a6e590bb170" volumeName="kubernetes.io/configmap/7115245a-cbc9-4180-9709-2a6e590bb170-konnectivity-ca" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550211 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c669d496-909b-428e-9b02-1ba9b05fd718" volumeName="kubernetes.io/projected/c669d496-909b-428e-9b02-1ba9b05fd718-kube-api-access-b5c8z" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550218 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9fd93f4-68f8-4711-ad54-410d10af6c35" volumeName="kubernetes.io/configmap/d9fd93f4-68f8-4711-ad54-410d10af6c35-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 04:44:35.550210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550229 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="084e8194-89a6-4702-af2a-34df80b46792" volumeName="kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550237 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09846780-4fb0-474a-bd15-9b272dc00772" volumeName="kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-env-overrides" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550245 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59632489-f9ac-44e1-ae2a-7a119c0ece48" volumeName="kubernetes.io/projected/59632489-f9ac-44e1-ae2a-7a119c0ece48-kube-api-access-5rb56" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550253 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="084e8194-89a6-4702-af2a-34df80b46792" volumeName="kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-config-out" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550261 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="084e8194-89a6-4702-af2a-34df80b46792" volumeName="kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550268 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09846780-4fb0-474a-bd15-9b272dc00772" volumeName="kubernetes.io/secret/09846780-4fb0-474a-bd15-9b272dc00772-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550275 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54628fc4-6d74-4048-ab60-61a643aac200" volumeName="kubernetes.io/configmap/54628fc4-6d74-4048-ab60-61a643aac200-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550282 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0780f4f-55b4-49c6-80a1-4026cc98f858" volumeName="kubernetes.io/projected/a0780f4f-55b4-49c6-80a1-4026cc98f858-kube-api-access-mt74j" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550289 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-db" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550297 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09846780-4fb0-474a-bd15-9b272dc00772" volumeName="kubernetes.io/projected/09846780-4fb0-474a-bd15-9b272dc00772-kube-api-access-chwz8" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550304 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c1d23d7-96c3-4406-9354-860812b8d2bb" volumeName="kubernetes.io/configmap/9c1d23d7-96c3-4406-9354-860812b8d2bb-serviceca" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550311 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9fd93f4-68f8-4711-ad54-410d10af6c35" volumeName="kubernetes.io/projected/d9fd93f4-68f8-4711-ad54-410d10af6c35-kube-api-access-nn4sc" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550319 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550327 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550337 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550344 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09846780-4fb0-474a-bd15-9b272dc00772" volumeName="kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-ovnkube-script-lib" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550351 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67f6fc33-45c9-4579-ab82-039cb764b155" volumeName="kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-cni-binary-copy" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550376 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="926ec2b0-7229-4297-8af9-bcd48ceab12a" volumeName="kubernetes.io/empty-dir/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-tuned" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550383 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" volumeName="kubernetes.io/projected/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-kube-api-access-xgvbs" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550390 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f09ff3d-66bc-46bc-b404-9a45acd58e1d" volumeName="kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550398 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c669d496-909b-428e-9b02-1ba9b05fd718" volumeName="kubernetes.io/secret/c669d496-909b-428e-9b02-1ba9b05fd718-metrics-tls" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550406 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9fd93f4-68f8-4711-ad54-410d10af6c35" volumeName="kubernetes.io/empty-dir/d9fd93f4-68f8-4711-ad54-410d10af6c35-audit-log" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550412 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550420 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="084e8194-89a6-4702-af2a-34df80b46792" volumeName="kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550431 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="084e8194-89a6-4702-af2a-34df80b46792" volumeName="kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-web-config" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550441 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c" volumeName="kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550449 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2967a60d-8f1a-4084-ac8b-5c50109aaf3b" volumeName="kubernetes.io/projected/2967a60d-8f1a-4084-ac8b-5c50109aaf3b-kube-api-access-22gdq" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550456 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27bfe4b2-2f4e-46ec-be37-830a9f5c20f5" volumeName="kubernetes.io/configmap/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-iptables-alerter-script" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550464 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="926ec2b0-7229-4297-8af9-bcd48ceab12a" volumeName="kubernetes.io/projected/926ec2b0-7229-4297-8af9-bcd48ceab12a-kube-api-access-p9zsw" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550471 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc295553-de72-4758-83ad-7123e5e29f03" volumeName="kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-kube-api-access-l9w2d" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550478 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="084e8194-89a6-4702-af2a-34df80b46792" volumeName="kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-metrics-client-ca" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550485 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26dd286e-348a-4378-be3d-9687ba409fc4" volumeName="kubernetes.io/configmap/26dd286e-348a-4378-be3d-9687ba409fc4-multus-daemon-config" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550493 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30011b22-59f1-4361-b961-c5be6de91e88" volumeName="kubernetes.io/projected/30011b22-59f1-4361-b961-c5be6de91e88-kube-api-access-zdhkb" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550500 6587 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="926ec2b0-7229-4297-8af9-bcd48ceab12a" volumeName="kubernetes.io/empty-dir/926ec2b0-7229-4297-8af9-bcd48ceab12a-tmp" seLinuxMountContext="" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550509 6587 reconstruct.go:97] "Volume reconstruction finished" Oct 30 04:44:35.551771 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.550514 6587 reconciler.go:29] "Reconciler: start to sync state" Oct 30 04:44:35.571819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.571785 6587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 04:44:35.572068 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.572046 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 04:44:35.573130 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.573109 6587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 04:44:35.573130 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.573129 6587 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 04:44:35.573303 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.573149 6587 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 04:44:35.573303 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:35.573203 6587 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 04:44:35.576663 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.576641 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 04:44:35.593316 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.593289 6587 generic.go:356] "Generic (PLEG): container finished" podID="67f6fc33-45c9-4579-ab82-039cb764b155" containerID="5cce97456747db203262d61423489adaf000d8a8679d68a4fb14c9c1bdbf1cd1" exitCode=0 Oct 30 04:44:35.593423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.593316 6587 generic.go:356] "Generic (PLEG): container finished" podID="67f6fc33-45c9-4579-ab82-039cb764b155" containerID="1b68be4bb51252e2404159f7b94ba9bd7046458ff4b8829a687945fa8dbe17f6" exitCode=0 Oct 30 04:44:35.593423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.593333 6587 generic.go:356] "Generic (PLEG): container finished" podID="67f6fc33-45c9-4579-ab82-039cb764b155" containerID="81cb4ecfbb1f01f1c805f54d6ebeab984ed9b5822aac2e869eab44a5ac44cd5d" exitCode=0 Oct 30 04:44:35.593423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.593339 6587 generic.go:356] "Generic (PLEG): container finished" podID="67f6fc33-45c9-4579-ab82-039cb764b155" containerID="6f8a17ae5d7ff5663fc2f15bc3879aba530080784a1d329bd36dfc3eb820f259" exitCode=0 Oct 30 04:44:35.593423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.593344 6587 generic.go:356] "Generic (PLEG): container finished" podID="67f6fc33-45c9-4579-ab82-039cb764b155" containerID="d1a1c49484081fb21fb97726f14a69ad09cca79f7447b726cb38c45ff6a51a01" exitCode=0 Oct 30 04:44:35.593423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.593349 6587 generic.go:356] "Generic (PLEG): container finished" podID="67f6fc33-45c9-4579-ab82-039cb764b155" containerID="431f31410288dea05606ae57fd0b3a8a9ee3b42826d691e36d734f38e10746b9" exitCode=0 Oct 30 04:44:35.594715 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.594692 6587 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:44:35.594815 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.594778 6587 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 04:44:35.594815 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.594780 6587 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:44:35.604386 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.604340 6587 generic.go:356] "Generic (PLEG): container finished" podID="fc295553-de72-4758-83ad-7123e5e29f03" containerID="f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7" exitCode=0 Oct 30 04:44:35.607945 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.607911 6587 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:44:35.608684 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.608662 6587 generic.go:356] "Generic (PLEG): container finished" podID="084e8194-89a6-4702-af2a-34df80b46792" containerID="19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22" exitCode=0 Oct 30 04:44:35.616234 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.616218 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 04:44:35.616559 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.616534 6587 generic.go:356] "Generic (PLEG): container finished" podID="09846780-4fb0-474a-bd15-9b272dc00772" containerID="8556a6fb16db8931e5624c1a6967a28fa45e22650716d351be70af2a2374ab55" exitCode=1 Oct 30 04:44:35.620554 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.620530 6587 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 04:44:35.624845 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.624820 6587 generic.go:356] "Generic (PLEG): container finished" podID="06a30a33e1bd990d029556ff52059236" containerID="533f78e171659dc9892081b34f0e02f39a7c2f944825b4283cd641a2f833924e" exitCode=0 Oct 30 04:44:35.630453 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.630431 6587 generic.go:356] "Generic (PLEG): container finished" podID="b1e09016-1f64-4c16-be6b-ab60b33a13e8" containerID="622355d997536302706e7f0e2f72de41bc6e56f87f251e335df2a5a5cc3d1711" exitCode=0 Oct 30 04:44:35.660203 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.660152 6587 manager.go:324] Recovery completed Oct 30 04:44:35.673633 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:35.673609 6587 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 04:44:35.682256 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.682224 6587 cpu_manager.go:226] "Starting" policy="none" Oct 30 04:44:35.682256 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.682246 6587 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 04:44:35.682408 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.682267 6587 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 04:44:35.682497 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.682478 6587 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 04:44:35.682571 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.682498 6587 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 04:44:35.682571 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.682525 6587 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 04:44:35.682571 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.682533 6587 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 04:44:35.682571 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.682541 6587 policy_none.go:50] "Start" Oct 30 04:44:35.682571 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.682567 6587 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 04:44:35.682777 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.682579 6587 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 04:44:35.684116 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.684101 6587 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 04:44:35.684189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.684122 6587 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 04:44:35.684189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.684133 6587 policy_none.go:44] "Start" Oct 30 04:44:35.689072 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.689045 6587 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 04:44:35.692229 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.692214 6587 manager.go:341] "Starting Device Plugin manager" Oct 30 04:44:35.692793 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:35.692237 6587 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 04:44:35.692793 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.692792 6587 server.go:87] "Starting device plugin registration server" Oct 30 04:44:35.693047 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.693033 6587 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 04:44:35.693126 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.693043 6587 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 04:44:35.693180 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.693139 6587 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 04:44:35.693276 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.693244 6587 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 04:44:35.693334 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.693288 6587 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 04:44:35.693334 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.693296 6587 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 04:44:35.693453 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.693327 6587 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T04:44:35.693271527Z","UUID":"26ca16ca-55f6-48cf-a104-ca61ec943d83","Handler":null,"Name":"","Endpoint":""} Oct 30 04:44:35.694720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.694700 6587 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 04:44:35.694797 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.694740 6587 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 04:44:35.694882 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:35.694865 6587 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 04:44:35.794515 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.794468 6587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 04:44:35.796677 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.796654 6587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 04:44:35.796800 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.796689 6587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 04:44:35.796800 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.796703 6587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 04:44:35.796800 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.796757 6587 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-105.ec2.internal" Oct 30 04:44:35.831072 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.831049 6587 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-105.ec2.internal" Oct 30 04:44:35.831223 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.831129 6587 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-105.ec2.internal" Oct 30 04:44:35.874456 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.874404 6587 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal"] Oct 30 04:44:35.875026 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.874989 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" event={"ID":"06a30a33e1bd990d029556ff52059236","Type":"ContainerStarted","Data":"502a31a50bed9b0ecfbdfd1e39e1b75e7be15f3cd5e6ee783c46d991950ce6e5"} Oct 30 04:44:35.875026 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.875025 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" event={"ID":"06a30a33e1bd990d029556ff52059236","Type":"ContainerDied","Data":"533f78e171659dc9892081b34f0e02f39a7c2f944825b4283cd641a2f833924e"} Oct 30 04:44:35.875186 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.875036 6587 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:44:35.875186 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.875144 6587 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" Oct 30 04:44:35.875186 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.875040 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" event={"ID":"06a30a33e1bd990d029556ff52059236","Type":"ContainerStarted","Data":"beff413d95e8fa1eb732fbc49c2ffa9fc0d72f7c2ef2f9570f882e9849fe81c7"} Oct 30 04:44:35.875310 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.875203 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" event={"ID":"624730a70c2b0e3a128e1c5af9dbe16d","Type":"ContainerStarted","Data":"31680165c4989cc3d966a511d10b3dd8a93ddd4b3b16a44686e168248c7e204c"} Oct 30 04:44:35.875310 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.875232 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" event={"ID":"624730a70c2b0e3a128e1c5af9dbe16d","Type":"ContainerStarted","Data":"3f15a33a8778626d8aabfcd1354c427a12b4cb5f27c67fae30f68f00d5837ba9"} Oct 30 04:44:35.905423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.905404 6587 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 04:44:35.905508 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:35.905443 6587 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-105.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" Oct 30 04:44:35.905961 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.905949 6587 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 04:44:35.905997 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:35.905973 6587 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:44:35.997310 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.997256 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/624730a70c2b0e3a128e1c5af9dbe16d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-105.ec2.internal\" (UID: \"624730a70c2b0e3a128e1c5af9dbe16d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" Oct 30 04:44:35.997310 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.997280 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06a30a33e1bd990d029556ff52059236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" (UID: \"06a30a33e1bd990d029556ff52059236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:44:35.997310 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:35.997304 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06a30a33e1bd990d029556ff52059236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" (UID: \"06a30a33e1bd990d029556ff52059236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:44:36.097854 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.097830 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06a30a33e1bd990d029556ff52059236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" (UID: \"06a30a33e1bd990d029556ff52059236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:44:36.097951 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.097855 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06a30a33e1bd990d029556ff52059236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" (UID: \"06a30a33e1bd990d029556ff52059236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:44:36.097951 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.097873 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/624730a70c2b0e3a128e1c5af9dbe16d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-105.ec2.internal\" (UID: \"624730a70c2b0e3a128e1c5af9dbe16d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" Oct 30 04:44:36.097951 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.097912 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/624730a70c2b0e3a128e1c5af9dbe16d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-105.ec2.internal\" (UID: \"624730a70c2b0e3a128e1c5af9dbe16d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" Oct 30 04:44:36.097951 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.097941 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06a30a33e1bd990d029556ff52059236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" (UID: \"06a30a33e1bd990d029556ff52059236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:44:36.098081 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.097941 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06a30a33e1bd990d029556ff52059236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" (UID: \"06a30a33e1bd990d029556ff52059236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:44:36.531999 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.531961 6587 apiserver.go:52] "Watching apiserver" Oct 30 04:44:36.606425 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.606401 6587 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 04:44:36.606996 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.606974 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0","openshift-ovn-kubernetes/ovnkube-node-tk9mv","kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal","openshift-network-operator/iptables-alerter-vnplr","openshift-monitoring/metrics-server-7d6768cb7d-ssqrt","openshift-multus/multus-49x8w","openshift-insights/insights-runtime-extractor-h72cf","kube-system/global-pull-secret-syncer-hcgqp","openshift-multus/network-metrics-daemon-kcmnj","openshift-dns/node-resolver-xhn9f","openshift-network-diagnostics/network-check-target-bblhz","openshift-ingress-canary/ingress-canary-69jf4","openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9","openshift-image-registry/node-ca-dl5rx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm","openshift-monitoring/node-exporter-z4r7p","openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6","openshift-dns/dns-default-8jf62","openshift-multus/multus-additional-cni-plugins-crggn","openshift-cluster-node-tuning-operator/tuned-dlpz8","openshift-monitoring/prometheus-k8s-0","kube-system/konnectivity-agent-cqv2d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal"] Oct 30 04:44:36.627663 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.627637 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 04:44:36.627798 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.627698 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 04:44:36.627798 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.627708 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 04:44:36.627798 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.627759 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 04:44:36.627916 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.627796 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 04:44:36.628060 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628037 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:44:36.628127 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628048 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mxnkb\"" Oct 30 04:44:36.628265 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628247 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45bnw\"" Oct 30 04:44:36.628265 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628262 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 04:44:36.628471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628277 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 04:44:36.628471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628279 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 04:44:36.628471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628292 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 04:44:36.628471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628303 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 04:44:36.628471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628332 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jzfzk\"" Oct 30 04:44:36.628471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628335 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 04:44:36.628471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628345 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-56dd5\"" Oct 30 04:44:36.628471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628394 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 04:44:36.628471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628396 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 04:44:36.628471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628405 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q24mq\"" Oct 30 04:44:36.628471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628443 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 04:44:36.628800 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628490 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 04:44:36.628800 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.628548 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bngnh\"" Oct 30 04:44:36.634331 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.634313 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 04:44:36.634912 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.634893 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bpk6m\"" Oct 30 04:44:36.635143 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.635131 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 04:44:36.635272 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.635255 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 04:44:36.635335 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.635258 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 04:44:36.635335 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.635260 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 04:44:36.636591 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.636570 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:44:36.636692 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.636597 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g88vf\"" Oct 30 04:44:36.637076 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.637060 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:44:36.637145 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.637123 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 04:44:36.637416 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.637380 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 04:44:36.637416 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.637402 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 04:44:36.637416 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.637413 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 04:44:36.637577 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.637429 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 04:44:36.637577 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.637455 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 04:44:36.637577 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.637460 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 04:44:36.637577 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.637425 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 04:44:36.637577 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.637466 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 04:44:36.637577 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.637520 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 04:44:36.637813 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.637630 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhd5l\"" Oct 30 04:44:36.637813 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.637673 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4jtzx\"" Oct 30 04:44:36.638288 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.638271 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 04:44:36.638428 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.638408 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 04:44:36.638532 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.638435 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 04:44:36.638532 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.638453 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 04:44:36.638656 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.638637 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 04:44:36.639057 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.639037 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 04:44:36.639057 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.639053 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 04:44:36.639208 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.639061 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 04:44:36.639208 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.639080 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k8wtk\"" Oct 30 04:44:36.639208 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.639061 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzdcr\"" Oct 30 04:44:36.639208 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.639101 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 04:44:36.639208 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.639114 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cswjq\"" Oct 30 04:44:36.639208 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.639153 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-g782k\"" Oct 30 04:44:36.639208 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.639169 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tdqqv\"" Oct 30 04:44:36.639208 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.639175 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 04:44:36.640176 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.640163 6587 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" Oct 30 04:44:36.640327 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.640313 6587 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:44:36.640889 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.640871 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 04:44:36.640889 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.640884 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 04:44:36.640995 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.640895 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 04:44:36.640995 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.640878 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 04:44:36.640995 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.640884 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 04:44:36.640995 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.640902 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 04:44:36.640995 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.640928 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 04:44:36.640995 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.640949 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s48rw\"" Oct 30 04:44:36.641209 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641006 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 04:44:36.641209 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641009 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 04:44:36.641209 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641034 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 04:44:36.641209 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641042 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tg42c\"" Oct 30 04:44:36.641209 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641079 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k257x\"" Oct 30 04:44:36.641424 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641329 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 04:44:36.641424 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641348 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zp4kp\"" Oct 30 04:44:36.641424 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641352 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 04:44:36.641424 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641413 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4oao19m04ptk3\"" Oct 30 04:44:36.641610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641426 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 04:44:36.641610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641438 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hsqv5\"" Oct 30 04:44:36.641610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641467 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 04:44:36.641610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641485 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fq9cl\"" Oct 30 04:44:36.641610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641510 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 04:44:36.641610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641529 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 04:44:36.641610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641541 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 04:44:36.641610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641515 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 04:44:36.641610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641580 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 04:44:36.641610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641584 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 04:44:36.641610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641592 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 04:44:36.641610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641612 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 04:44:36.641610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641617 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3fklq5stm461m\"" Oct 30 04:44:36.642069 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641620 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 04:44:36.642069 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641668 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 04:44:36.642069 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641672 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 04:44:36.642069 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641673 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:44:36.642069 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641688 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 04:44:36.642069 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641705 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 04:44:36.642069 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.641727 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 04:44:36.642412 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.642070 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 04:44:36.642550 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.642527 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 04:44:36.642663 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.642648 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 04:44:36.649737 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.649719 6587 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 04:44:36.655920 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.655897 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 04:44:36.656250 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.656232 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 04:44:36.658263 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.658245 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 04:44:36.663247 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.663219 6587 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 04:44:36.663322 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:36.663294 6587 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-105.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-105.ec2.internal" Oct 30 04:44:36.663419 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.663403 6587 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 04:44:36.663473 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:36.663460 6587 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-105.ec2.internal" Oct 30 04:44:36.700411 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700385 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-var-lib-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.700521 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700423 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-cnibin\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.700521 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700448 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-etc-kubernetes\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.700587 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700513 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rb56\" (UniqueName: \"kubernetes.io/projected/59632489-f9ac-44e1-ae2a-7a119c0ece48-kube-api-access-5rb56\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:44:36.700587 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700552 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d9fd93f4-68f8-4711-ad54-410d10af6c35-metrics-server-audit-profiles\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.700587 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700581 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.700674 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700605 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-system-cni-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.700709 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700686 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c669d496-909b-428e-9b02-1ba9b05fd718-tmp-dir\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:36.700756 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700737 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.700803 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700767 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c669d496-909b-428e-9b02-1ba9b05fd718-tmp-dir\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:36.700851 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700823 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-lib-modules\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.701201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700862 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d9fd93f4-68f8-4711-ad54-410d10af6c35-metrics-server-audit-profiles\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.701201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700944 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-run-ovn-kubernetes\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.701201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.700977 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:44:36.701201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701000 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-systemd-units\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.701201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701034 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.701201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701042 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-cni-netd\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.701201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701067 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32ed1e66-99e0-4714-92d1-1ef6213fed2c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:36.701201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701107 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-systemd\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.701201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701144 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-ovn\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.701201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701168 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-metrics-certs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:44:36.701201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701185 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701227 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-827bt\" (UniqueName: \"kubernetes.io/projected/9c1d23d7-96c3-4406-9354-860812b8d2bb-kube-api-access-827bt\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701248 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32ed1e66-99e0-4714-92d1-1ef6213fed2c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701272 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54628fc4-6d74-4048-ab60-61a643aac200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701386 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701391 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-config-volume\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701430 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpl2b\" (UniqueName: \"kubernetes.io/projected/b1e09016-1f64-4c16-be6b-ab60b33a13e8-kube-api-access-dpl2b\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701543 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54628fc4-6d74-4048-ab60-61a643aac200-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701552 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701577 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-kubelet-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701600 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701618 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-systemd\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701629 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-config-volume\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701645 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:36.701720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701689 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-host-slash\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701748 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c669d496-909b-428e-9b02-1ba9b05fd718-metrics-tls\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701764 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-cnibin\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701820 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701826 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701851 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701863 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701929 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701935 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c669d496-909b-428e-9b02-1ba9b05fd718-metrics-tls\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.701964 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-config-out\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702004 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysconfig\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702027 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-config-out\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702034 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/32ed1e66-99e0-4714-92d1-1ef6213fed2c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702089 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c669d496-909b-428e-9b02-1ba9b05fd718-config-volume\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702102 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702117 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-secret-metrics-server-client-certs\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702142 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-client-ca-bundle\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702167 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702172 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/32ed1e66-99e0-4714-92d1-1ef6213fed2c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702191 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7115245a-cbc9-4180-9709-2a6e590bb170-konnectivity-ca\") pod \"konnectivity-agent-cqv2d\" (UID: \"7115245a-cbc9-4180-9709-2a6e590bb170\") " pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702218 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1e09016-1f64-4c16-be6b-ab60b33a13e8-metrics-client-ca\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702244 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-netns\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702270 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59632489-f9ac-44e1-ae2a-7a119c0ece48-hosts-file\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702290 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c669d496-909b-428e-9b02-1ba9b05fd718-config-volume\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702294 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-secret-metrics-server-tls\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702338 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702389 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zsw\" (UniqueName: \"kubernetes.io/projected/926ec2b0-7229-4297-8af9-bcd48ceab12a-kube-api-access-p9zsw\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702418 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-hostroot\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702432 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-client-ca-bundle\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.702431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702446 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59632489-f9ac-44e1-ae2a-7a119c0ece48-tmp-dir\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702471 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d9fd93f4-68f8-4711-ad54-410d10af6c35-audit-log\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702497 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702514 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-secret-metrics-server-tls\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702527 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702553 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-dbus\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702575 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c1d23d7-96c3-4406-9354-860812b8d2bb-host\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702579 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702598 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-cni-bin\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702412 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d9fd93f4-68f8-4711-ad54-410d10af6c35-secret-metrics-server-client-certs\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702621 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-kubelet\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702690 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702747 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7115245a-cbc9-4180-9709-2a6e590bb170-konnectivity-ca\") pod \"konnectivity-agent-cqv2d\" (UID: \"7115245a-cbc9-4180-9709-2a6e590bb170\") " pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702828 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702928 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5c8z\" (UniqueName: \"kubernetes.io/projected/c669d496-909b-428e-9b02-1ba9b05fd718-kube-api-access-b5c8z\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702966 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702971 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.702992 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703018 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysctl-conf\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703047 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwz8\" (UniqueName: \"kubernetes.io/projected/09846780-4fb0-474a-bd15-9b272dc00772-kube-api-access-chwz8\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703073 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-socket-dir-parent\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703098 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-cni-multus\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703109 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1e09016-1f64-4c16-be6b-ab60b33a13e8-metrics-client-ca\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703125 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9fd93f4-68f8-4711-ad54-410d10af6c35-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703150 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703160 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59632489-f9ac-44e1-ae2a-7a119c0ece48-tmp-dir\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703175 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703199 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703223 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7115245a-cbc9-4180-9709-2a6e590bb170-agent-certs\") pod \"konnectivity-agent-cqv2d\" (UID: \"7115245a-cbc9-4180-9709-2a6e590bb170\") " pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703250 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703273 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/926ec2b0-7229-4297-8af9-bcd48ceab12a-tmp\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703296 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gdq\" (UniqueName: \"kubernetes.io/projected/2967a60d-8f1a-4084-ac8b-5c50109aaf3b-kube-api-access-22gdq\") pod \"ingress-canary-69jf4\" (UID: \"2967a60d-8f1a-4084-ac8b-5c50109aaf3b\") " pod="openshift-ingress-canary/ingress-canary-69jf4" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703316 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9fd93f4-68f8-4711-ad54-410d10af6c35-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703321 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703372 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-ovnkube-script-lib\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703409 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c1d23d7-96c3-4406-9354-860812b8d2bb-serviceca\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703438 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjb7\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-kube-api-access-dkjb7\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703467 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-k8s-cni-cncf-io\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703494 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703519 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-textfile\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703550 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703582 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-etc-selinux\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703606 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703620 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-ovnkube-script-lib\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703632 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54628fc4-6d74-4048-ab60-61a643aac200-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703657 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703202 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d9fd93f4-68f8-4711-ad54-410d10af6c35-audit-log\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703661 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703609 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703717 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703785 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/926ec2b0-7229-4297-8af9-bcd48ceab12a-tmp\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.703808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703843 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703863 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703900 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54628fc4-6d74-4048-ab60-61a643aac200-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703954 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-textfile\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703986 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.703999 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7115245a-cbc9-4180-9709-2a6e590bb170-agent-certs\") pod \"konnectivity-agent-cqv2d\" (UID: \"7115245a-cbc9-4180-9709-2a6e590bb170\") " pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704062 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30011b22-59f1-4361-b961-c5be6de91e88-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704114 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704148 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704175 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704182 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-cni-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704221 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26dd286e-348a-4378-be3d-9687ba409fc4-cni-binary-copy\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704275 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-conf-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704296 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30011b22-59f1-4361-b961-c5be6de91e88-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704312 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704402 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvbs\" (UniqueName: \"kubernetes.io/projected/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-kube-api-access-xgvbs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704430 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704477 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26dd286e-348a-4378-be3d-9687ba409fc4-cni-binary-copy\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704482 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-ovnkube-config\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704513 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704596 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704608 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704625 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdhkb\" (UniqueName: \"kubernetes.io/projected/30011b22-59f1-4361-b961-c5be6de91e88-kube-api-access-zdhkb\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704695 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c1d23d7-96c3-4406-9354-860812b8d2bb-serviceca\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704733 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn4sc\" (UniqueName: \"kubernetes.io/projected/d9fd93f4-68f8-4711-ad54-410d10af6c35-kube-api-access-nn4sc\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704744 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-ovnkube-config\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704844 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.704946 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-run\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705083 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705143 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd5t4\" (UniqueName: \"kubernetes.io/projected/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-api-access-dd5t4\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705307 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26dd286e-348a-4378-be3d-9687ba409fc4-multus-daemon-config\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705340 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-registration-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705404 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-tls\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705422 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv6f\" (UniqueName: \"kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f\") pod \"network-check-target-bblhz\" (UID: \"1c23f69d-0026-4202-884b-38a3d4bbe87d\") " pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705425 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705438 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-node-log\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705471 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-iptables-alerter-script\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705502 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30011b22-59f1-4361-b961-c5be6de91e88-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705566 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705568 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26dd286e-348a-4378-be3d-9687ba409fc4-multus-daemon-config\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705595 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-cni-binary-copy\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705621 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705636 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5xk\" (UniqueName: \"kubernetes.io/projected/26dd286e-348a-4378-be3d-9687ba409fc4-kube-api-access-hs5xk\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705653 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-config\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705716 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-slash\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705748 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-run-netns\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705774 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-cni-bin\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705798 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-socket-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705822 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-sys\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705844 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-var-lib-kubelet\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705719 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30011b22-59f1-4361-b961-c5be6de91e88-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705855 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-iptables-alerter-script\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705898 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-tls\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.705907 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2967a60d-8f1a-4084-ac8b-5c50109aaf3b-cert\") pod \"ingress-canary-69jf4\" (UID: \"2967a60d-8f1a-4084-ac8b-5c50109aaf3b\") " pod="openshift-ingress-canary/ingress-canary-69jf4" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706036 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-cni-binary-copy\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706085 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706177 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-config\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706320 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-kubernetes\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.706286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706326 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67f6fc33-45c9-4579-ab82-039cb764b155-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706376 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-host\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706544 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706586 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706593 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2967a60d-8f1a-4084-ac8b-5c50109aaf3b-cert\") pod \"ingress-canary-69jf4\" (UID: \"2967a60d-8f1a-4084-ac8b-5c50109aaf3b\") " pod="openshift-ingress-canary/ingress-canary-69jf4" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706641 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt74j\" (UniqueName: \"kubernetes.io/projected/a0780f4f-55b4-49c6-80a1-4026cc98f858-kube-api-access-mt74j\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706680 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysctl-d\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706694 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-original-pull-secret\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706707 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-root\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706759 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706792 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-wtmp\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706820 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706837 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706837 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706884 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-tls-assets\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706892 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.706927 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-etc-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707034 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707040 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twfl6\" (UniqueName: \"kubernetes.io/projected/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-kube-api-access-twfl6\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707062 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-tls-assets\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707111 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707200 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9w2d\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-kube-api-access-l9w2d\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707243 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54628fc4-6d74-4048-ab60-61a643aac200-crio-socket\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707261 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54628fc4-6d74-4048-ab60-61a643aac200-data-volume\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707284 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-modprobe-d\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707298 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-sys\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707316 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707332 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-env-overrides\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707436 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbkv\" (UniqueName: \"kubernetes.io/projected/67f6fc33-45c9-4579-ab82-039cb764b155-kube-api-access-tgbkv\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707476 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707489 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54628fc4-6d74-4048-ab60-61a643aac200-data-volume\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707535 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-web-config\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707593 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/09846780-4fb0-474a-bd15-9b272dc00772-env-overrides\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707641 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-tuned\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707667 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-log-socket\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707694 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/09846780-4fb0-474a-bd15-9b272dc00772-ovn-node-metrics-cert\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707722 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-tuned\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707694 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707739 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-system-cni-dir\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707785 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707846 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbcp\" (UniqueName: \"kubernetes.io/projected/54628fc4-6d74-4048-ab60-61a643aac200-kube-api-access-8wbcp\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707879 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-kubelet-config\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707884 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-web-config\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707905 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-kubelet\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707937 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30011b22-59f1-4361-b961-c5be6de91e88-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707967 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-multus-certs\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.707995 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.708024 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/09846780-4fb0-474a-bd15-9b272dc00772-ovn-node-metrics-cert\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.708034 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-device-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.708025 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.708058 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-sys-fs\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.708153 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30011b22-59f1-4361-b961-c5be6de91e88-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:36.708567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.708200 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.718799 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.718775 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rb56\" (UniqueName: \"kubernetes.io/projected/59632489-f9ac-44e1-ae2a-7a119c0ece48-kube-api-access-5rb56\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:44:36.719500 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.719474 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpl2b\" (UniqueName: \"kubernetes.io/projected/b1e09016-1f64-4c16-be6b-ab60b33a13e8-kube-api-access-dpl2b\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.719940 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.719924 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-827bt\" (UniqueName: \"kubernetes.io/projected/9c1d23d7-96c3-4406-9354-860812b8d2bb-kube-api-access-827bt\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:44:36.721003 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.720986 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zsw\" (UniqueName: \"kubernetes.io/projected/926ec2b0-7229-4297-8af9-bcd48ceab12a-kube-api-access-p9zsw\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.721114 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.721099 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5c8z\" (UniqueName: \"kubernetes.io/projected/c669d496-909b-428e-9b02-1ba9b05fd718-kube-api-access-b5c8z\") pod \"dns-default-8jf62\" (UID: \"c669d496-909b-428e-9b02-1ba9b05fd718\") " pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:36.722009 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.721990 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gdq\" (UniqueName: \"kubernetes.io/projected/2967a60d-8f1a-4084-ac8b-5c50109aaf3b-kube-api-access-22gdq\") pod \"ingress-canary-69jf4\" (UID: \"2967a60d-8f1a-4084-ac8b-5c50109aaf3b\") " pod="openshift-ingress-canary/ingress-canary-69jf4" Oct 30 04:44:36.722423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.722407 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwz8\" (UniqueName: \"kubernetes.io/projected/09846780-4fb0-474a-bd15-9b272dc00772-kube-api-access-chwz8\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.722508 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.722475 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvbs\" (UniqueName: \"kubernetes.io/projected/9f09ff3d-66bc-46bc-b404-9a45acd58e1d-kube-api-access-xgvbs\") pod \"network-metrics-daemon-kcmnj\" (UID: \"9f09ff3d-66bc-46bc-b404-9a45acd58e1d\") " pod="openshift-multus/network-metrics-daemon-kcmnj" Oct 30 04:44:36.722967 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.722951 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdhkb\" (UniqueName: \"kubernetes.io/projected/30011b22-59f1-4361-b961-c5be6de91e88-kube-api-access-zdhkb\") pod \"openshift-state-metrics-7bd7489957-s8lx6\" (UID: \"30011b22-59f1-4361-b961-c5be6de91e88\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s8lx6" Oct 30 04:44:36.723232 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.723216 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn4sc\" (UniqueName: \"kubernetes.io/projected/d9fd93f4-68f8-4711-ad54-410d10af6c35-kube-api-access-nn4sc\") pod \"metrics-server-7d6768cb7d-ssqrt\" (UID: \"d9fd93f4-68f8-4711-ad54-410d10af6c35\") " pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:36.723579 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.723564 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsv6f\" (UniqueName: \"kubernetes.io/projected/1c23f69d-0026-4202-884b-38a3d4bbe87d-kube-api-access-hsv6f\") pod \"network-check-target-bblhz\" (UID: \"1c23f69d-0026-4202-884b-38a3d4bbe87d\") " pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:44:36.724515 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.724495 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5xk\" (UniqueName: \"kubernetes.io/projected/26dd286e-348a-4378-be3d-9687ba409fc4-kube-api-access-hs5xk\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.724691 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.724671 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd5t4\" (UniqueName: \"kubernetes.io/projected/32ed1e66-99e0-4714-92d1-1ef6213fed2c-kube-api-access-dd5t4\") pod \"kube-state-metrics-5dc8d7d89b-8bgh9\" (UID: \"32ed1e66-99e0-4714-92d1-1ef6213fed2c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8bgh9" Oct 30 04:44:36.724984 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.724970 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt74j\" (UniqueName: \"kubernetes.io/projected/a0780f4f-55b4-49c6-80a1-4026cc98f858-kube-api-access-mt74j\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.725038 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.725023 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twfl6\" (UniqueName: \"kubernetes.io/projected/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-kube-api-access-twfl6\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:44:36.725754 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.725735 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbkv\" (UniqueName: \"kubernetes.io/projected/67f6fc33-45c9-4579-ab82-039cb764b155-kube-api-access-tgbkv\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.725842 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.725770 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbcp\" (UniqueName: \"kubernetes.io/projected/54628fc4-6d74-4048-ab60-61a643aac200-kube-api-access-8wbcp\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:36.726201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.726183 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9w2d\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-kube-api-access-l9w2d\") pod \"prometheus-k8s-0\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:36.726263 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.726213 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkjb7\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-kube-api-access-dkjb7\") pod \"alertmanager-main-0\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:36.808762 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808739 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-kubelet-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.808901 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808768 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-systemd\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.808901 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808784 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-host-slash\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:44:36.808901 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808798 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-cnibin\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.808901 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808815 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysconfig\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.808901 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808833 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-netns\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.808901 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808856 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59632489-f9ac-44e1-ae2a-7a119c0ece48-hosts-file\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:44:36.808901 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808872 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-systemd\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.808901 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808898 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-cnibin\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808877 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27bfe4b2-2f4e-46ec-be37-830a9f5c20f5-host-slash\") pod \"iptables-alerter-vnplr\" (UID: \"27bfe4b2-2f4e-46ec-be37-830a9f5c20f5\") " pod="openshift-network-operator/iptables-alerter-vnplr" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808880 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-hostroot\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808926 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59632489-f9ac-44e1-ae2a-7a119c0ece48-hosts-file\") pod \"node-resolver-xhn9f\" (UID: \"59632489-f9ac-44e1-ae2a-7a119c0ece48\") " pod="openshift-dns/node-resolver-xhn9f" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808945 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-dbus\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808970 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c1d23d7-96c3-4406-9354-860812b8d2bb-host\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808926 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-hostroot\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808916 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysconfig\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808875 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-kubelet-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809015 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c1d23d7-96c3-4406-9354-860812b8d2bb-host\") pod \"node-ca-dl5rx\" (UID: \"9c1d23d7-96c3-4406-9354-860812b8d2bb\") " pod="openshift-image-registry/node-ca-dl5rx" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809010 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-dbus\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808904 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-netns\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.808994 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-cni-bin\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809025 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-cni-bin\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809055 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-kubelet\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809073 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysctl-conf\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809100 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-kubelet\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809131 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-socket-dir-parent\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809147 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-cni-multus\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809168 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-k8s-cni-cncf-io\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809174 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysctl-conf\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809184 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-etc-selinux\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809197 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-socket-dir-parent\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809210 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-cni-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.809195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809215 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-var-lib-cni-multus\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809227 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-conf-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809250 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809253 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-k8s-cni-cncf-io\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809265 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-run\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809269 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-etc-selinux\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809279 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-cni-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809298 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-multus-conf-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809313 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-run\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809316 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809325 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-registration-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809353 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-node-log\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809393 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-registration-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809405 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-slash\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809429 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-run-netns\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809444 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-node-log\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809455 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-slash\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809452 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-cni-bin\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809478 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-run-netns\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809479 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-socket-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809484 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-cni-bin\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809502 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-sys\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809508 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-socket-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809520 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-var-lib-kubelet\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809530 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-sys\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809536 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-kubernetes\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809550 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-host\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809560 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-var-lib-kubelet\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809567 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysctl-d\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809593 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-root\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809603 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-kubernetes\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809621 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-wtmp\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809629 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-root\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809641 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-etc-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809645 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-host\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809656 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-sysctl-d\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809661 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54628fc4-6d74-4048-ab60-61a643aac200-crio-socket\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809681 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-modprobe-d\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809686 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-etc-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809696 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-sys\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809658 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-node-exporter-wtmp\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809695 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54628fc4-6d74-4048-ab60-61a643aac200-crio-socket\") pod \"insights-runtime-extractor-h72cf\" (UID: \"54628fc4-6d74-4048-ab60-61a643aac200\") " pod="openshift-insights/insights-runtime-extractor-h72cf" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809711 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809729 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-log-socket\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809738 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-etc-modprobe-d\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809745 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809739 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1e09016-1f64-4c16-be6b-ab60b33a13e8-sys\") pod \"node-exporter-z4r7p\" (UID: \"b1e09016-1f64-4c16-be6b-ab60b33a13e8\") " pod="openshift-monitoring/node-exporter-z4r7p" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809750 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-system-cni-dir\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809770 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-system-cni-dir\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809769 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-log-socket\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809780 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-kubelet-config\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809795 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-kubelet\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809835 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-multus-certs\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809848 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c-kubelet-config\") pod \"global-pull-secret-syncer-hcgqp\" (UID: \"0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c\") " pod="kube-system/global-pull-secret-syncer-hcgqp" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809854 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-device-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809871 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-kubelet\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809889 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-host-run-multus-certs\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809893 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-device-dir\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809902 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-sys-fs\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809924 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-var-lib-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809941 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-cnibin\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809955 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-etc-kubernetes\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809976 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809984 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67f6fc33-45c9-4579-ab82-039cb764b155-cnibin\") pod \"multus-additional-cni-plugins-crggn\" (UID: \"67f6fc33-45c9-4579-ab82-039cb764b155\") " pod="openshift-multus/multus-additional-cni-plugins-crggn" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809996 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a0780f4f-55b4-49c6-80a1-4026cc98f858-sys-fs\") pod \"aws-ebs-csi-driver-node-67crm\" (UID: \"a0780f4f-55b4-49c6-80a1-4026cc98f858\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.809999 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-system-cni-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810008 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-etc-kubernetes\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810018 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-var-lib-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810067 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-lib-modules\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810096 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-run-ovn-kubernetes\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810116 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-openvswitch\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810121 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-systemd-units\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810132 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-run-ovn-kubernetes\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810166 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/926ec2b0-7229-4297-8af9-bcd48ceab12a-lib-modules\") pod \"tuned-dlpz8\" (UID: \"926ec2b0-7229-4297-8af9-bcd48ceab12a\") " pod="openshift-cluster-node-tuning-operator/tuned-dlpz8" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810168 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26dd286e-348a-4378-be3d-9687ba409fc4-system-cni-dir\") pod \"multus-49x8w\" (UID: \"26dd286e-348a-4378-be3d-9687ba409fc4\") " pod="openshift-multus/multus-49x8w" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810201 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-systemd-units\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.810185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810206 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-cni-netd\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.812200 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810242 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-systemd\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.812200 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810270 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-ovn\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.812200 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810282 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-systemd\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.812200 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810293 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-host-cni-netd\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:36.812200 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:36.810325 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/09846780-4fb0-474a-bd15-9b272dc00772-run-ovn\") pod \"ovnkube-node-tk9mv\" (UID: \"09846780-4fb0-474a-bd15-9b272dc00772\") " pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:37.382792 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:37.382763 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:38.109652 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:38.109624 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8jf62" Oct 30 04:44:39.452236 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:39.452205 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" Oct 30 04:44:41.907576 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:41.907545 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cqv2d" Oct 30 04:44:42.709374 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:42.709336 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:44:43.272482 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:43.272456 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:43.617302 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:43.617270 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:44:43.663863 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:43.663839 6587 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 04:44:51.377191 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.377118 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 04:44:51.377732 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.377555 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="alertmanager" containerID="cri-o://79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0" gracePeriod=120 Oct 30 04:44:51.377732 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.377573 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="kube-rbac-proxy-metric" containerID="cri-o://aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c" gracePeriod=120 Oct 30 04:44:51.377732 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.377607 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="config-reloader" containerID="cri-o://336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d" gracePeriod=120 Oct 30 04:44:51.377732 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.377638 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="prom-label-proxy" containerID="cri-o://7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a" gracePeriod=120 Oct 30 04:44:51.377732 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.377664 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="kube-rbac-proxy" containerID="cri-o://055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88" gracePeriod=120 Oct 30 04:44:51.378013 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.377576 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="kube-rbac-proxy-web" containerID="cri-o://467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123" gracePeriod=120 Oct 30 04:44:51.695458 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.695377 6587 generic.go:356] "Generic (PLEG): container finished" podID="084e8194-89a6-4702-af2a-34df80b46792" containerID="7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a" exitCode=0 Oct 30 04:44:51.695458 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.695401 6587 generic.go:356] "Generic (PLEG): container finished" podID="084e8194-89a6-4702-af2a-34df80b46792" containerID="055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88" exitCode=0 Oct 30 04:44:51.695458 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.695411 6587 generic.go:356] "Generic (PLEG): container finished" podID="084e8194-89a6-4702-af2a-34df80b46792" containerID="336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d" exitCode=0 Oct 30 04:44:51.695458 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.695421 6587 generic.go:356] "Generic (PLEG): container finished" podID="084e8194-89a6-4702-af2a-34df80b46792" containerID="79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0" exitCode=0 Oct 30 04:44:51.695458 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.695444 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerDied","Data":"7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a"} Oct 30 04:44:51.695718 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.695507 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerDied","Data":"055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88"} Oct 30 04:44:51.695718 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.695535 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerDied","Data":"336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d"} Oct 30 04:44:51.695718 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:51.695560 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerDied","Data":"79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0"} Oct 30 04:44:52.619112 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.619089 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.644867 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.644838 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-cluster-tls-config\") pod \"084e8194-89a6-4702-af2a-34df80b46792\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " Oct 30 04:44:52.644997 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.644888 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-main-tls\") pod \"084e8194-89a6-4702-af2a-34df80b46792\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " Oct 30 04:44:52.644997 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.644923 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy\") pod \"084e8194-89a6-4702-af2a-34df80b46792\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " Oct 30 04:44:52.644997 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.644951 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-tls-assets\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-tls-assets\") pod \"084e8194-89a6-4702-af2a-34df80b46792\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " Oct 30 04:44:52.645172 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.645006 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-trusted-ca-bundle\") pod \"084e8194-89a6-4702-af2a-34df80b46792\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " Oct 30 04:44:52.645172 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.645039 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-kube-api-access-dkjb7\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-kube-api-access-dkjb7\") pod \"084e8194-89a6-4702-af2a-34df80b46792\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " Oct 30 04:44:52.645172 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.645072 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-web-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-web-config\") pod \"084e8194-89a6-4702-af2a-34df80b46792\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " Oct 30 04:44:52.645172 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.645118 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-web\") pod \"084e8194-89a6-4702-af2a-34df80b46792\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " Oct 30 04:44:52.645394 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.645155 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-main-db\") pod \"084e8194-89a6-4702-af2a-34df80b46792\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " Oct 30 04:44:52.645394 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.645233 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-metric\") pod \"084e8194-89a6-4702-af2a-34df80b46792\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " Oct 30 04:44:52.645394 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.645261 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-config-out\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-config-out\") pod \"084e8194-89a6-4702-af2a-34df80b46792\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " Oct 30 04:44:52.645394 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.645288 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-metrics-client-ca\") pod \"084e8194-89a6-4702-af2a-34df80b46792\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " Oct 30 04:44:52.645394 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.645314 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-config-volume\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-config-volume\") pod \"084e8194-89a6-4702-af2a-34df80b46792\" (UID: \"084e8194-89a6-4702-af2a-34df80b46792\") " Oct 30 04:44:52.645658 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.645537 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-trusted-ca-bundle" pod "084e8194-89a6-4702-af2a-34df80b46792" (UID: "084e8194-89a6-4702-af2a-34df80b46792"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:52.645941 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.645912 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-main-db" pod "084e8194-89a6-4702-af2a-34df80b46792" (UID: "084e8194-89a6-4702-af2a-34df80b46792"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:44:52.648620 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.648577 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-metrics-client-ca" pod "084e8194-89a6-4702-af2a-34df80b46792" (UID: "084e8194-89a6-4702-af2a-34df80b46792"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:52.649405 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.649335 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-main-tls" pod "084e8194-89a6-4702-af2a-34df80b46792" (UID: "084e8194-89a6-4702-af2a-34df80b46792"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:52.649512 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.649424 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-tls-assets" pod "084e8194-89a6-4702-af2a-34df80b46792" (UID: "084e8194-89a6-4702-af2a-34df80b46792"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:44:52.649572 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.649513 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-kube-api-access-dkjb7" pod "084e8194-89a6-4702-af2a-34df80b46792" (UID: "084e8194-89a6-4702-af2a-34df80b46792"). InnerVolumeSpecName "kube-api-access-dkjb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:44:52.649836 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.649815 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy" pod "084e8194-89a6-4702-af2a-34df80b46792" (UID: "084e8194-89a6-4702-af2a-34df80b46792"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:52.649980 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.649945 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-web" pod "084e8194-89a6-4702-af2a-34df80b46792" (UID: "084e8194-89a6-4702-af2a-34df80b46792"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:52.650199 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.650163 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-config-volume" pod "084e8194-89a6-4702-af2a-34df80b46792" (UID: "084e8194-89a6-4702-af2a-34df80b46792"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:52.650278 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.650224 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-config-out" pod "084e8194-89a6-4702-af2a-34df80b46792" (UID: "084e8194-89a6-4702-af2a-34df80b46792"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:44:52.650342 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.650289 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-metric" pod "084e8194-89a6-4702-af2a-34df80b46792" (UID: "084e8194-89a6-4702-af2a-34df80b46792"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:52.653630 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.653534 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-cluster-tls-config" pod "084e8194-89a6-4702-af2a-34df80b46792" (UID: "084e8194-89a6-4702-af2a-34df80b46792"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:52.661564 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.661541 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-web-config" pod "084e8194-89a6-4702-af2a-34df80b46792" (UID: "084e8194-89a6-4702-af2a-34df80b46792"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:52.702621 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.702532 6587 generic.go:356] "Generic (PLEG): container finished" podID="084e8194-89a6-4702-af2a-34df80b46792" containerID="aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c" exitCode=0 Oct 30 04:44:52.702621 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.702559 6587 generic.go:356] "Generic (PLEG): container finished" podID="084e8194-89a6-4702-af2a-34df80b46792" containerID="467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123" exitCode=0 Oct 30 04:44:52.702621 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.702612 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.702860 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.702611 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerDied","Data":"aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c"} Oct 30 04:44:52.702860 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.702695 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerDied","Data":"467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123"} Oct 30 04:44:52.702860 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.702740 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"084e8194-89a6-4702-af2a-34df80b46792","Type":"ContainerDied","Data":"af9d29a9ea9dc92cd9b476d44afe8091a22404a8f59fbef697d73a00b5fa1da0"} Oct 30 04:44:52.702860 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.702762 6587 scope.go:122] "RemoveContainer" containerID="7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a" Oct 30 04:44:52.713819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.713796 6587 scope.go:122] "RemoveContainer" containerID="aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c" Oct 30 04:44:52.724657 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.724528 6587 scope.go:122] "RemoveContainer" containerID="055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88" Oct 30 04:44:52.733413 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.733394 6587 scope.go:122] "RemoveContainer" containerID="467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123" Oct 30 04:44:52.740336 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.740319 6587 scope.go:122] "RemoveContainer" containerID="336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d" Oct 30 04:44:52.746145 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.746118 6587 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:52.746228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.746155 6587 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-main-db\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:52.746228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.746177 6587 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:52.746228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.746195 6587 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/084e8194-89a6-4702-af2a-34df80b46792-config-out\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:52.746228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.746209 6587 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-metrics-client-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:52.746228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.746223 6587 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:52.746400 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.746237 6587 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-cluster-tls-config\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:52.746400 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.746251 6587 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-main-tls\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:52.746400 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.746266 6587 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:52.746400 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.746279 6587 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-tls-assets\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:52.746400 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.746291 6587 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/084e8194-89a6-4702-af2a-34df80b46792-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:52.746400 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.746304 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkjb7\" (UniqueName: \"kubernetes.io/projected/084e8194-89a6-4702-af2a-34df80b46792-kube-api-access-dkjb7\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:52.746400 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.746317 6587 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/084e8194-89a6-4702-af2a-34df80b46792-web-config\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:52.752828 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.752801 6587 scope.go:122] "RemoveContainer" containerID="79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0" Oct 30 04:44:52.759621 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.759599 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 04:44:52.761250 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.761233 6587 scope.go:122] "RemoveContainer" containerID="19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22" Oct 30 04:44:52.762805 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.762788 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 04:44:52.768706 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.768685 6587 scope.go:122] "RemoveContainer" containerID="7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a" Oct 30 04:44:52.768991 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:52.768970 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a\": container with ID starting with 7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a not found: ID does not exist" containerID="7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a" Oct 30 04:44:52.769060 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.769002 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a"} err="failed to get container status \"7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a\": rpc error: code = NotFound desc = could not find container \"7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a\": container with ID starting with 7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a not found: ID does not exist" Oct 30 04:44:52.769060 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.769042 6587 scope.go:122] "RemoveContainer" containerID="aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c" Oct 30 04:44:52.769307 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:52.769291 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c\": container with ID starting with aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c not found: ID does not exist" containerID="aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c" Oct 30 04:44:52.769391 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.769312 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c"} err="failed to get container status \"aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c\": rpc error: code = NotFound desc = could not find container \"aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c\": container with ID starting with aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c not found: ID does not exist" Oct 30 04:44:52.769391 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.769325 6587 scope.go:122] "RemoveContainer" containerID="055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88" Oct 30 04:44:52.769624 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:52.769593 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88\": container with ID starting with 055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88 not found: ID does not exist" containerID="055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88" Oct 30 04:44:52.769679 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.769634 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88"} err="failed to get container status \"055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88\": rpc error: code = NotFound desc = could not find container \"055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88\": container with ID starting with 055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88 not found: ID does not exist" Oct 30 04:44:52.769679 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.769657 6587 scope.go:122] "RemoveContainer" containerID="467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123" Oct 30 04:44:52.770273 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:52.769954 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123\": container with ID starting with 467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123 not found: ID does not exist" containerID="467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123" Oct 30 04:44:52.770273 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.769994 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123"} err="failed to get container status \"467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123\": rpc error: code = NotFound desc = could not find container \"467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123\": container with ID starting with 467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123 not found: ID does not exist" Oct 30 04:44:52.770273 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.770016 6587 scope.go:122] "RemoveContainer" containerID="336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d" Oct 30 04:44:52.770582 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:52.770561 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d\": container with ID starting with 336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d not found: ID does not exist" containerID="336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d" Oct 30 04:44:52.770638 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.770586 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d"} err="failed to get container status \"336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d\": rpc error: code = NotFound desc = could not find container \"336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d\": container with ID starting with 336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d not found: ID does not exist" Oct 30 04:44:52.770638 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.770601 6587 scope.go:122] "RemoveContainer" containerID="79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0" Oct 30 04:44:52.770893 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:52.770876 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0\": container with ID starting with 79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0 not found: ID does not exist" containerID="79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0" Oct 30 04:44:52.770965 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.770902 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0"} err="failed to get container status \"79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0\": rpc error: code = NotFound desc = could not find container \"79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0\": container with ID starting with 79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0 not found: ID does not exist" Oct 30 04:44:52.770965 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.770922 6587 scope.go:122] "RemoveContainer" containerID="19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22" Oct 30 04:44:52.771284 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:52.771263 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22\": container with ID starting with 19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22 not found: ID does not exist" containerID="19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22" Oct 30 04:44:52.771397 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.771283 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22"} err="failed to get container status \"19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22\": rpc error: code = NotFound desc = could not find container \"19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22\": container with ID starting with 19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22 not found: ID does not exist" Oct 30 04:44:52.771397 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.771296 6587 scope.go:122] "RemoveContainer" containerID="7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a" Oct 30 04:44:52.771595 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.771577 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a"} err="failed to get container status \"7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a\": rpc error: code = NotFound desc = could not find container \"7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a\": container with ID starting with 7f32965308d07d5c93e25b3873a10ec7b33c3e576a336b9ab790ccc10c9cf75a not found: ID does not exist" Oct 30 04:44:52.771595 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.771595 6587 scope.go:122] "RemoveContainer" containerID="aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c" Oct 30 04:44:52.771961 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.771934 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c"} err="failed to get container status \"aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c\": rpc error: code = NotFound desc = could not find container \"aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c\": container with ID starting with aecdf7c1a0ec5f414a6bbc2f5d5880ef76316e611ccda047daf5892e980fef7c not found: ID does not exist" Oct 30 04:44:52.772049 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.771965 6587 scope.go:122] "RemoveContainer" containerID="055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88" Oct 30 04:44:52.772245 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.772224 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88"} err="failed to get container status \"055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88\": rpc error: code = NotFound desc = could not find container \"055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88\": container with ID starting with 055d10d21dc3a3a0755d5ea6790823f454a09e6109d5cd0544d0ef7a1d93ea88 not found: ID does not exist" Oct 30 04:44:52.772290 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.772245 6587 scope.go:122] "RemoveContainer" containerID="467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123" Oct 30 04:44:52.772511 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.772488 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123"} err="failed to get container status \"467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123\": rpc error: code = NotFound desc = could not find container \"467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123\": container with ID starting with 467949966e1fdbfe0e869cae777317f240a3703a0d101b1910bc44c60b435123 not found: ID does not exist" Oct 30 04:44:52.772563 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.772513 6587 scope.go:122] "RemoveContainer" containerID="336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d" Oct 30 04:44:52.772761 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.772743 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d"} err="failed to get container status \"336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d\": rpc error: code = NotFound desc = could not find container \"336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d\": container with ID starting with 336a72ca63d432a7691204c17b54d5051d59fb55010f2bc2a49958bec6fb969d not found: ID does not exist" Oct 30 04:44:52.772815 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.772762 6587 scope.go:122] "RemoveContainer" containerID="79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0" Oct 30 04:44:52.773011 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.772994 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0"} err="failed to get container status \"79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0\": rpc error: code = NotFound desc = could not find container \"79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0\": container with ID starting with 79f9805b226e42cd880971537361a9165968bdad96dff183167496ef8e4d5ad0 not found: ID does not exist" Oct 30 04:44:52.773064 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.773011 6587 scope.go:122] "RemoveContainer" containerID="19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22" Oct 30 04:44:52.773235 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.773218 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22"} err="failed to get container status \"19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22\": rpc error: code = NotFound desc = could not find container \"19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22\": container with ID starting with 19ab170106ee009015943f7af100bfa5f023a758f03f29c9806c15e174a13e22 not found: ID does not exist" Oct 30 04:44:52.789538 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789453 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 04:44:52.789834 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789814 6587 cpu_manager.go:414] "containerMap: removing container" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="alertmanager" Oct 30 04:44:52.789834 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789833 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="alertmanager" Oct 30 04:44:52.790007 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789854 6587 cpu_manager.go:414] "containerMap: removing container" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="config-reloader" Oct 30 04:44:52.790007 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789863 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="config-reloader" Oct 30 04:44:52.790007 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789892 6587 cpu_manager.go:414] "containerMap: removing container" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="kube-rbac-proxy-metric" Oct 30 04:44:52.790007 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789900 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="kube-rbac-proxy-metric" Oct 30 04:44:52.790007 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789913 6587 cpu_manager.go:414] "containerMap: removing container" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="kube-rbac-proxy" Oct 30 04:44:52.790007 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789920 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="kube-rbac-proxy" Oct 30 04:44:52.790007 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789932 6587 cpu_manager.go:414] "containerMap: removing container" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="kube-rbac-proxy-web" Oct 30 04:44:52.790007 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789938 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="kube-rbac-proxy-web" Oct 30 04:44:52.790007 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789950 6587 cpu_manager.go:414] "containerMap: removing container" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="init-config-reloader" Oct 30 04:44:52.790007 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789956 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="init-config-reloader" Oct 30 04:44:52.790007 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789968 6587 cpu_manager.go:414] "containerMap: removing container" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="prom-label-proxy" Oct 30 04:44:52.790007 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.789976 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="prom-label-proxy" Oct 30 04:44:52.790597 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.790048 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="kube-rbac-proxy" Oct 30 04:44:52.790597 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.790059 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="kube-rbac-proxy-web" Oct 30 04:44:52.790597 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.790067 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="init-config-reloader" Oct 30 04:44:52.790597 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.790076 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="prom-label-proxy" Oct 30 04:44:52.790597 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.790085 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="alertmanager" Oct 30 04:44:52.790597 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.790093 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="config-reloader" Oct 30 04:44:52.790597 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.790106 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="084e8194-89a6-4702-af2a-34df80b46792" containerName="kube-rbac-proxy-metric" Oct 30 04:44:52.793541 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.793518 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.809438 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.809410 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 04:44:52.809554 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.809473 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 04:44:52.809554 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.809485 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 04:44:52.809554 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.809497 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 04:44:52.809554 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.809482 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 04:44:52.809554 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.809499 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 04:44:52.809554 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.809544 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 04:44:52.809840 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.809606 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 04:44:52.809840 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.809764 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s48rw\"" Oct 30 04:44:52.823144 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.823119 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 04:44:52.823795 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.823778 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 04:44:52.846968 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.846940 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e3a61c90-c5ce-42ad-80ed-f94d548ef369-config-out\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.847074 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.846973 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.847074 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.846991 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e3a61c90-c5ce-42ad-80ed-f94d548ef369-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.847236 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.847103 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-web-config\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.847236 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.847189 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3a61c90-c5ce-42ad-80ed-f94d548ef369-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.847338 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.847236 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-config-volume\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.847338 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.847264 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.847338 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.847292 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.847516 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.847405 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e3a61c90-c5ce-42ad-80ed-f94d548ef369-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.847516 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.847436 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3a61c90-c5ce-42ad-80ed-f94d548ef369-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.847516 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.847487 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.847621 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.847519 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8dt\" (UniqueName: \"kubernetes.io/projected/e3a61c90-c5ce-42ad-80ed-f94d548ef369-kube-api-access-nj8dt\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.847621 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.847546 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948314 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948281 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e3a61c90-c5ce-42ad-80ed-f94d548ef369-config-out\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948314 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948314 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948533 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948330 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e3a61c90-c5ce-42ad-80ed-f94d548ef369-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948533 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948379 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-web-config\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948533 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948410 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3a61c90-c5ce-42ad-80ed-f94d548ef369-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948533 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948439 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-config-volume\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948533 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948462 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948533 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948485 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948533 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948517 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e3a61c90-c5ce-42ad-80ed-f94d548ef369-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948876 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948542 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3a61c90-c5ce-42ad-80ed-f94d548ef369-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948876 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948562 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948876 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948602 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8dt\" (UniqueName: \"kubernetes.io/projected/e3a61c90-c5ce-42ad-80ed-f94d548ef369-kube-api-access-nj8dt\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948876 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948628 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.948876 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948766 6587 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 04:44:52.948876 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.948810 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e3a61c90-c5ce-42ad-80ed-f94d548ef369-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.949770 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.949500 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3a61c90-c5ce-42ad-80ed-f94d548ef369-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.949770 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.949709 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3a61c90-c5ce-42ad-80ed-f94d548ef369-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.951617 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.951553 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e3a61c90-c5ce-42ad-80ed-f94d548ef369-config-out\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.951745 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.951615 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e3a61c90-c5ce-42ad-80ed-f94d548ef369-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.951745 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.951677 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.951865 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.951770 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.952016 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.951992 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-web-config\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.952133 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.952117 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.952311 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.952287 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.952898 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.952846 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.953201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.953185 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e3a61c90-c5ce-42ad-80ed-f94d548ef369-config-volume\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:52.967755 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:52.967737 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8dt\" (UniqueName: \"kubernetes.io/projected/e3a61c90-c5ce-42ad-80ed-f94d548ef369-kube-api-access-nj8dt\") pod \"alertmanager-main-0\" (UID: \"e3a61c90-c5ce-42ad-80ed-f94d548ef369\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:53.108972 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:53.108939 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 04:44:53.422386 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:53.422346 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 04:44:53.423154 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:53.423121 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a61c90_c5ce_42ad_80ed_f94d548ef369.slice/crio-280067e66488ea5737b27d72dd2284ec5e2fb3199fafa3b51a9e8951eb40c776 WatchSource:0}: Error finding container 280067e66488ea5737b27d72dd2284ec5e2fb3199fafa3b51a9e8951eb40c776: Status 404 returned error can't find the container with id 280067e66488ea5737b27d72dd2284ec5e2fb3199fafa3b51a9e8951eb40c776 Oct 30 04:44:53.577772 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:53.577743 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084e8194-89a6-4702-af2a-34df80b46792" path="/var/lib/kubelet/pods/084e8194-89a6-4702-af2a-34df80b46792/volumes" Oct 30 04:44:53.707885 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:53.707812 6587 generic.go:356] "Generic (PLEG): container finished" podID="e3a61c90-c5ce-42ad-80ed-f94d548ef369" containerID="a6a11fa96e97acbf46f6eaabc4f5054acff7c939146d40e4fb1198dc6c9da08a" exitCode=0 Oct 30 04:44:53.708268 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:53.707895 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e3a61c90-c5ce-42ad-80ed-f94d548ef369","Type":"ContainerDied","Data":"a6a11fa96e97acbf46f6eaabc4f5054acff7c939146d40e4fb1198dc6c9da08a"} Oct 30 04:44:53.708268 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:53.707966 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e3a61c90-c5ce-42ad-80ed-f94d548ef369","Type":"ContainerStarted","Data":"280067e66488ea5737b27d72dd2284ec5e2fb3199fafa3b51a9e8951eb40c776"} Oct 30 04:44:54.715964 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:54.715926 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e3a61c90-c5ce-42ad-80ed-f94d548ef369","Type":"ContainerStarted","Data":"7e15369b1a01f745ec440d7c0ace626ca80b6fc52862f446e0eba611a2f2ddee"} Oct 30 04:44:54.716393 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:54.716039 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e3a61c90-c5ce-42ad-80ed-f94d548ef369","Type":"ContainerStarted","Data":"5d5c43397a93beefeba021005f84e79e24ad905ee79ff72339edd6d8e62d10ce"} Oct 30 04:44:54.716393 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:54.716084 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e3a61c90-c5ce-42ad-80ed-f94d548ef369","Type":"ContainerStarted","Data":"39db67937ded94f55ae5d939b2dd2844e37d29bcca3ebf5e52b5c2e4ddd85973"} Oct 30 04:44:54.716393 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:54.716121 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e3a61c90-c5ce-42ad-80ed-f94d548ef369","Type":"ContainerStarted","Data":"c7b91537252270f4d536f0ff9699f72322ee6cba06161766cd9dadc153353906"} Oct 30 04:44:54.716393 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:54.716157 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e3a61c90-c5ce-42ad-80ed-f94d548ef369","Type":"ContainerStarted","Data":"273e132b52b409670c85be0bc1f99e6daeacc8588c0659bd7ee15fd6b90b93f5"} Oct 30 04:44:54.716393 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:54.716191 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e3a61c90-c5ce-42ad-80ed-f94d548ef369","Type":"ContainerStarted","Data":"073385b94e1bcf97bea915425584742a80678e66f314f472c8d4fcedf41c5e79"} Oct 30 04:44:54.811950 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:54.801772 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.787564186 podStartE2EDuration="2.801754859s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.190673ms" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:54.801089276 +0000 UTC m=+19.328856063" watchObservedRunningTime="2026-10-30 04:44:54.801754859 +0000 UTC m=+19.329521648" Oct 30 04:44:55.335840 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.335802 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5cd9d8f774-qwxql"] Oct 30 04:44:55.369185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.369152 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5cd9d8f774-qwxql"] Oct 30 04:44:55.369351 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.369296 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.388121 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.388092 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 04:44:55.388121 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.388097 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 04:44:55.388297 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.388120 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 04:44:55.388297 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.388098 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 04:44:55.388297 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.388161 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fh7s4\"" Oct 30 04:44:55.427970 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.427947 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 04:44:55.441803 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.441785 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 04:44:55.469178 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.469155 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-serving-certs-ca-bundle\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.469321 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.469185 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmwgm\" (UniqueName: \"kubernetes.io/projected/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-kube-api-access-jmwgm\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.469321 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.469221 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-secret-telemeter-client\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.469321 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.469241 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-telemeter-client-tls\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.469480 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.469415 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.469480 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.469451 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.469480 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.469475 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-metrics-client-ca\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.469609 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.469568 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-federate-client-tls\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.570085 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.570051 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmwgm\" (UniqueName: \"kubernetes.io/projected/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-kube-api-access-jmwgm\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.570244 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.570101 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-secret-telemeter-client\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.570244 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.570127 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-telemeter-client-tls\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.570244 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.570166 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.570244 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.570201 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.570244 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.570234 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-metrics-client-ca\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.570502 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.570275 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-federate-client-tls\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.570502 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.570329 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-serving-certs-ca-bundle\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.571107 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.571074 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-metrics-client-ca\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.571228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.571082 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-serving-certs-ca-bundle\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.571296 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.571274 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.572738 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.572712 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-secret-telemeter-client\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.572738 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.572732 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.572864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.572820 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-federate-client-tls\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.572960 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.572939 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-telemeter-client-tls\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.589386 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.589321 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmwgm\" (UniqueName: \"kubernetes.io/projected/dabdaf17-12d9-48bc-94e3-fac2c28cc3dc-kube-api-access-jmwgm\") pod \"telemeter-client-5cd9d8f774-qwxql\" (UID: \"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc\") " pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.720205 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.720175 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" Oct 30 04:44:55.956009 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.955929 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 04:44:55.956403 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.956217 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="prometheus" containerID="cri-o://3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148" gracePeriod=600 Oct 30 04:44:55.956403 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.956263 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="thanos-sidecar" containerID="cri-o://ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a" gracePeriod=600 Oct 30 04:44:55.956403 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.956286 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5" gracePeriod=600 Oct 30 04:44:55.956403 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.956301 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="config-reloader" containerID="cri-o://70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c" gracePeriod=600 Oct 30 04:44:55.956403 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.956248 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="kube-rbac-proxy" containerID="cri-o://2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65" gracePeriod=600 Oct 30 04:44:55.956807 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:55.956286 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="kube-rbac-proxy-web" containerID="cri-o://04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5" gracePeriod=600 Oct 30 04:44:55.996048 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:55.995857 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:44:56.054109 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.054083 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5cd9d8f774-qwxql"] Oct 30 04:44:56.054861 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:56.054834 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabdaf17_12d9_48bc_94e3_fac2c28cc3dc.slice/crio-d18493dca4cc40dc770bbab9dda8278dd0171fece6eb3e914e78dbf7d2e9ee7e WatchSource:0}: Error finding container d18493dca4cc40dc770bbab9dda8278dd0171fece6eb3e914e78dbf7d2e9ee7e: Status 404 returned error can't find the container with id d18493dca4cc40dc770bbab9dda8278dd0171fece6eb3e914e78dbf7d2e9ee7e Oct 30 04:44:56.063571 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.063547 6587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:44:56.194899 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.194878 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.276167 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276092 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-thanos-prometheus-http-client-file\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276167 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276133 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-0\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276167 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276154 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-metrics-client-certs\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276466 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276177 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-2\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276466 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276208 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-kube-api-access-l9w2d\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-kube-api-access-l9w2d\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276466 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276257 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-serving-certs-ca-bundle\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276466 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276299 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-tls\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276466 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276323 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-tls-assets\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-tls-assets\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276466 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276352 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276466 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276402 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-grpc-tls\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276466 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276428 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276476 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-web-config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-web-config\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276512 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-trusted-ca-bundle\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276562 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-1\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276562 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-2" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:56.276868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276604 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-config\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276639 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-kubelet-serving-ca-bundle\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276665 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-metrics-client-ca\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276691 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-db\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276724 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-config-out\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.276868 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.276759 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-kube-rbac-proxy\") pod \"fc295553-de72-4758-83ad-7123e5e29f03\" (UID: \"fc295553-de72-4758-83ad-7123e5e29f03\") " Oct 30 04:44:56.277379 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.277281 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-serving-certs-ca-bundle" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:56.277379 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.277332 6587 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.277969 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.277879 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-kubelet-serving-ca-bundle" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:56.277969 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.277904 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-0" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:56.279197 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.278566 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-metrics-client-ca" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:56.279197 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.278604 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-trusted-ca-bundle" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:56.279197 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.278907 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-1" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:44:56.279597 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.279555 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-thanos-prometheus-http-client-file" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:56.279794 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.279765 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-db" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:44:56.280327 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.280296 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-metrics-client-certs" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:56.281138 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.281112 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-kube-rbac-proxy" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:56.281594 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.281567 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-config" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:56.282421 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.282384 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-tls" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:56.282521 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.282498 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-kube-api-access-l9w2d" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "kube-api-access-l9w2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:44:56.282599 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.282511 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-kube-rbac-proxy-web" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:56.282678 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.282661 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-tls-assets" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:44:56.282744 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.282685 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-thanos-sidecar-tls" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:56.282807 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.282774 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-grpc-tls" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:56.283830 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.283809 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-config-out" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:44:56.290497 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.290476 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-web-config" pod "fc295553-de72-4758-83ad-7123e5e29f03" (UID: "fc295553-de72-4758-83ad-7123e5e29f03"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:44:56.377953 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.377910 6587 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-web-config\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.377953 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.377951 6587 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.377967 6587 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.377980 6587 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-config\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.377994 6587 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378005 6587 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-metrics-client-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378018 6587 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-db\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378032 6587 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc295553-de72-4758-83ad-7123e5e29f03-config-out\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378045 6587 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-kube-rbac-proxy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378058 6587 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378074 6587 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378088 6587 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-metrics-client-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378101 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9w2d\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-kube-api-access-l9w2d\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378114 6587 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc295553-de72-4758-83ad-7123e5e29f03-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378127 6587 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378141 6587 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc295553-de72-4758-83ad-7123e5e29f03-tls-assets\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378156 6587 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378174 6587 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-grpc-tls\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.378182 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.378187 6587 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc295553-de72-4758-83ad-7123e5e29f03-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:44:56.726946 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.726909 6587 generic.go:356] "Generic (PLEG): container finished" podID="fc295553-de72-4758-83ad-7123e5e29f03" containerID="1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5" exitCode=0 Oct 30 04:44:56.726946 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.726934 6587 generic.go:356] "Generic (PLEG): container finished" podID="fc295553-de72-4758-83ad-7123e5e29f03" containerID="2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65" exitCode=0 Oct 30 04:44:56.726946 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.726943 6587 generic.go:356] "Generic (PLEG): container finished" podID="fc295553-de72-4758-83ad-7123e5e29f03" containerID="04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5" exitCode=0 Oct 30 04:44:56.726946 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.726951 6587 generic.go:356] "Generic (PLEG): container finished" podID="fc295553-de72-4758-83ad-7123e5e29f03" containerID="ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a" exitCode=0 Oct 30 04:44:56.727576 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.726958 6587 generic.go:356] "Generic (PLEG): container finished" podID="fc295553-de72-4758-83ad-7123e5e29f03" containerID="70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c" exitCode=0 Oct 30 04:44:56.727576 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.726967 6587 generic.go:356] "Generic (PLEG): container finished" podID="fc295553-de72-4758-83ad-7123e5e29f03" containerID="3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148" exitCode=0 Oct 30 04:44:56.727576 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.726988 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerDied","Data":"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5"} Oct 30 04:44:56.727576 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.727056 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerDied","Data":"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65"} Oct 30 04:44:56.727576 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.726994 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.727576 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.727089 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerDied","Data":"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5"} Oct 30 04:44:56.727576 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.727121 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerDied","Data":"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a"} Oct 30 04:44:56.727576 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.727142 6587 scope.go:122] "RemoveContainer" containerID="1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5" Oct 30 04:44:56.727576 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.727154 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerDied","Data":"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c"} Oct 30 04:44:56.727576 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.727197 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerDied","Data":"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148"} Oct 30 04:44:56.727576 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.727230 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc295553-de72-4758-83ad-7123e5e29f03","Type":"ContainerDied","Data":"d0fb8002b53a36d7660e29b9f85de87d3922a1daab2c0190cf339b402a3bddf8"} Oct 30 04:44:56.728648 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.728627 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" event={"ID":"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc","Type":"ContainerStarted","Data":"d18493dca4cc40dc770bbab9dda8278dd0171fece6eb3e914e78dbf7d2e9ee7e"} Oct 30 04:44:56.737081 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.737063 6587 scope.go:122] "RemoveContainer" containerID="2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65" Oct 30 04:44:56.746198 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.746173 6587 scope.go:122] "RemoveContainer" containerID="04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5" Oct 30 04:44:56.753884 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.753727 6587 scope.go:122] "RemoveContainer" containerID="ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a" Oct 30 04:44:56.762326 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.762309 6587 scope.go:122] "RemoveContainer" containerID="70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c" Oct 30 04:44:56.770177 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.770163 6587 scope.go:122] "RemoveContainer" containerID="3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148" Oct 30 04:44:56.778456 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.778436 6587 scope.go:122] "RemoveContainer" containerID="f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7" Oct 30 04:44:56.783798 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.783778 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 04:44:56.786646 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.786628 6587 scope.go:122] "RemoveContainer" containerID="1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5" Oct 30 04:44:56.786929 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:56.786911 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5\": container with ID starting with 1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5 not found: ID does not exist" containerID="1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5" Oct 30 04:44:56.786987 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.786940 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5"} err="failed to get container status \"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5\": rpc error: code = NotFound desc = could not find container \"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5\": container with ID starting with 1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5 not found: ID does not exist" Oct 30 04:44:56.786987 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.786958 6587 scope.go:122] "RemoveContainer" containerID="2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65" Oct 30 04:44:56.787229 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:56.787209 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65\": container with ID starting with 2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65 not found: ID does not exist" containerID="2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65" Oct 30 04:44:56.787313 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.787237 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65"} err="failed to get container status \"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65\": rpc error: code = NotFound desc = could not find container \"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65\": container with ID starting with 2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65 not found: ID does not exist" Oct 30 04:44:56.787313 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.787261 6587 scope.go:122] "RemoveContainer" containerID="04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5" Oct 30 04:44:56.787586 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:56.787559 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5\": container with ID starting with 04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5 not found: ID does not exist" containerID="04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5" Oct 30 04:44:56.787693 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.787593 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5"} err="failed to get container status \"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5\": rpc error: code = NotFound desc = could not find container \"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5\": container with ID starting with 04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5 not found: ID does not exist" Oct 30 04:44:56.787693 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.787617 6587 scope.go:122] "RemoveContainer" containerID="ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a" Oct 30 04:44:56.787907 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:56.787877 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a\": container with ID starting with ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a not found: ID does not exist" containerID="ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a" Oct 30 04:44:56.787980 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.787907 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a"} err="failed to get container status \"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a\": rpc error: code = NotFound desc = could not find container \"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a\": container with ID starting with ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a not found: ID does not exist" Oct 30 04:44:56.787980 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.787929 6587 scope.go:122] "RemoveContainer" containerID="70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c" Oct 30 04:44:56.788276 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:56.788252 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c\": container with ID starting with 70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c not found: ID does not exist" containerID="70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c" Oct 30 04:44:56.788341 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.788283 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c"} err="failed to get container status \"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c\": rpc error: code = NotFound desc = could not find container \"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c\": container with ID starting with 70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c not found: ID does not exist" Oct 30 04:44:56.788341 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.788302 6587 scope.go:122] "RemoveContainer" containerID="3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148" Oct 30 04:44:56.788592 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:56.788576 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148\": container with ID starting with 3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148 not found: ID does not exist" containerID="3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148" Oct 30 04:44:56.788656 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.788595 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148"} err="failed to get container status \"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148\": rpc error: code = NotFound desc = could not find container \"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148\": container with ID starting with 3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148 not found: ID does not exist" Oct 30 04:44:56.788656 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.788609 6587 scope.go:122] "RemoveContainer" containerID="f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7" Oct 30 04:44:56.788895 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:44:56.788862 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7\": container with ID starting with f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7 not found: ID does not exist" containerID="f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7" Oct 30 04:44:56.788964 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.788900 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7"} err="failed to get container status \"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7\": rpc error: code = NotFound desc = could not find container \"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7\": container with ID starting with f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7 not found: ID does not exist" Oct 30 04:44:56.788964 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.788914 6587 scope.go:122] "RemoveContainer" containerID="1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5" Oct 30 04:44:56.789190 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.789164 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5"} err="failed to get container status \"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5\": rpc error: code = NotFound desc = could not find container \"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5\": container with ID starting with 1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5 not found: ID does not exist" Oct 30 04:44:56.789190 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.789189 6587 scope.go:122] "RemoveContainer" containerID="2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65" Oct 30 04:44:56.789488 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.789468 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65"} err="failed to get container status \"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65\": rpc error: code = NotFound desc = could not find container \"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65\": container with ID starting with 2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65 not found: ID does not exist" Oct 30 04:44:56.789539 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.789497 6587 scope.go:122] "RemoveContainer" containerID="04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5" Oct 30 04:44:56.789738 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.789714 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5"} err="failed to get container status \"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5\": rpc error: code = NotFound desc = could not find container \"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5\": container with ID starting with 04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5 not found: ID does not exist" Oct 30 04:44:56.789738 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.789737 6587 scope.go:122] "RemoveContainer" containerID="ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a" Oct 30 04:44:56.790015 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.789995 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a"} err="failed to get container status \"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a\": rpc error: code = NotFound desc = could not find container \"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a\": container with ID starting with ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a not found: ID does not exist" Oct 30 04:44:56.790015 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.790013 6587 scope.go:122] "RemoveContainer" containerID="70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c" Oct 30 04:44:56.790314 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.790292 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c"} err="failed to get container status \"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c\": rpc error: code = NotFound desc = could not find container \"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c\": container with ID starting with 70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c not found: ID does not exist" Oct 30 04:44:56.790395 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.790314 6587 scope.go:122] "RemoveContainer" containerID="3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148" Oct 30 04:44:56.790585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.790566 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148"} err="failed to get container status \"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148\": rpc error: code = NotFound desc = could not find container \"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148\": container with ID starting with 3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148 not found: ID does not exist" Oct 30 04:44:56.790632 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.790587 6587 scope.go:122] "RemoveContainer" containerID="f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7" Oct 30 04:44:56.790899 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.790813 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7"} err="failed to get container status \"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7\": rpc error: code = NotFound desc = could not find container \"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7\": container with ID starting with f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7 not found: ID does not exist" Oct 30 04:44:56.790998 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.790910 6587 scope.go:122] "RemoveContainer" containerID="1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5" Oct 30 04:44:56.791301 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.791260 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5"} err="failed to get container status \"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5\": rpc error: code = NotFound desc = could not find container \"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5\": container with ID starting with 1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5 not found: ID does not exist" Oct 30 04:44:56.791301 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.791287 6587 scope.go:122] "RemoveContainer" containerID="2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65" Oct 30 04:44:56.791583 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.791559 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65"} err="failed to get container status \"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65\": rpc error: code = NotFound desc = could not find container \"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65\": container with ID starting with 2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65 not found: ID does not exist" Oct 30 04:44:56.791583 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.791580 6587 scope.go:122] "RemoveContainer" containerID="04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5" Oct 30 04:44:56.791855 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.791823 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5"} err="failed to get container status \"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5\": rpc error: code = NotFound desc = could not find container \"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5\": container with ID starting with 04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5 not found: ID does not exist" Oct 30 04:44:56.791855 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.791845 6587 scope.go:122] "RemoveContainer" containerID="ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a" Oct 30 04:44:56.792189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.792163 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a"} err="failed to get container status \"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a\": rpc error: code = NotFound desc = could not find container \"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a\": container with ID starting with ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a not found: ID does not exist" Oct 30 04:44:56.792308 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.792190 6587 scope.go:122] "RemoveContainer" containerID="70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c" Oct 30 04:44:56.792505 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.792480 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c"} err="failed to get container status \"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c\": rpc error: code = NotFound desc = could not find container \"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c\": container with ID starting with 70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c not found: ID does not exist" Oct 30 04:44:56.792609 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.792506 6587 scope.go:122] "RemoveContainer" containerID="3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148" Oct 30 04:44:56.792807 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.792785 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148"} err="failed to get container status \"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148\": rpc error: code = NotFound desc = could not find container \"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148\": container with ID starting with 3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148 not found: ID does not exist" Oct 30 04:44:56.792807 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.792807 6587 scope.go:122] "RemoveContainer" containerID="f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7" Oct 30 04:44:56.793069 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.793051 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7"} err="failed to get container status \"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7\": rpc error: code = NotFound desc = could not find container \"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7\": container with ID starting with f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7 not found: ID does not exist" Oct 30 04:44:56.793069 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.793069 6587 scope.go:122] "RemoveContainer" containerID="1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5" Oct 30 04:44:56.793311 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.793293 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5"} err="failed to get container status \"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5\": rpc error: code = NotFound desc = could not find container \"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5\": container with ID starting with 1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5 not found: ID does not exist" Oct 30 04:44:56.793413 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.793311 6587 scope.go:122] "RemoveContainer" containerID="2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65" Oct 30 04:44:56.793620 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.793600 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65"} err="failed to get container status \"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65\": rpc error: code = NotFound desc = could not find container \"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65\": container with ID starting with 2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65 not found: ID does not exist" Oct 30 04:44:56.793698 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.793627 6587 scope.go:122] "RemoveContainer" containerID="04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5" Oct 30 04:44:56.793882 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.793860 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5"} err="failed to get container status \"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5\": rpc error: code = NotFound desc = could not find container \"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5\": container with ID starting with 04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5 not found: ID does not exist" Oct 30 04:44:56.794005 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.793883 6587 scope.go:122] "RemoveContainer" containerID="ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a" Oct 30 04:44:56.794630 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.794604 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a"} err="failed to get container status \"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a\": rpc error: code = NotFound desc = could not find container \"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a\": container with ID starting with ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a not found: ID does not exist" Oct 30 04:44:56.794630 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.794628 6587 scope.go:122] "RemoveContainer" containerID="70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c" Oct 30 04:44:56.794909 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.794878 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c"} err="failed to get container status \"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c\": rpc error: code = NotFound desc = could not find container \"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c\": container with ID starting with 70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c not found: ID does not exist" Oct 30 04:44:56.794958 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.794911 6587 scope.go:122] "RemoveContainer" containerID="3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148" Oct 30 04:44:56.795212 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.795181 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148"} err="failed to get container status \"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148\": rpc error: code = NotFound desc = could not find container \"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148\": container with ID starting with 3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148 not found: ID does not exist" Oct 30 04:44:56.795212 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.795208 6587 scope.go:122] "RemoveContainer" containerID="f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7" Oct 30 04:44:56.795533 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.795511 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7"} err="failed to get container status \"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7\": rpc error: code = NotFound desc = could not find container \"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7\": container with ID starting with f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7 not found: ID does not exist" Oct 30 04:44:56.795533 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.795532 6587 scope.go:122] "RemoveContainer" containerID="1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5" Oct 30 04:44:56.795773 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.795749 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5"} err="failed to get container status \"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5\": rpc error: code = NotFound desc = could not find container \"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5\": container with ID starting with 1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5 not found: ID does not exist" Oct 30 04:44:56.795773 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.795772 6587 scope.go:122] "RemoveContainer" containerID="2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65" Oct 30 04:44:56.796030 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.796002 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65"} err="failed to get container status \"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65\": rpc error: code = NotFound desc = could not find container \"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65\": container with ID starting with 2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65 not found: ID does not exist" Oct 30 04:44:56.796030 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.796029 6587 scope.go:122] "RemoveContainer" containerID="04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5" Oct 30 04:44:56.796377 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.796332 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5"} err="failed to get container status \"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5\": rpc error: code = NotFound desc = could not find container \"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5\": container with ID starting with 04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5 not found: ID does not exist" Oct 30 04:44:56.796377 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.796370 6587 scope.go:122] "RemoveContainer" containerID="ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a" Oct 30 04:44:56.796794 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.796744 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a"} err="failed to get container status \"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a\": rpc error: code = NotFound desc = could not find container \"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a\": container with ID starting with ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a not found: ID does not exist" Oct 30 04:44:56.796878 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.796793 6587 scope.go:122] "RemoveContainer" containerID="70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c" Oct 30 04:44:56.797082 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.797061 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c"} err="failed to get container status \"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c\": rpc error: code = NotFound desc = could not find container \"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c\": container with ID starting with 70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c not found: ID does not exist" Oct 30 04:44:56.797147 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.797081 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 04:44:56.797147 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.797084 6587 scope.go:122] "RemoveContainer" containerID="3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148" Oct 30 04:44:56.797426 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.797403 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148"} err="failed to get container status \"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148\": rpc error: code = NotFound desc = could not find container \"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148\": container with ID starting with 3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148 not found: ID does not exist" Oct 30 04:44:56.797426 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.797425 6587 scope.go:122] "RemoveContainer" containerID="f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7" Oct 30 04:44:56.797685 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.797668 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7"} err="failed to get container status \"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7\": rpc error: code = NotFound desc = could not find container \"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7\": container with ID starting with f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7 not found: ID does not exist" Oct 30 04:44:56.797737 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.797685 6587 scope.go:122] "RemoveContainer" containerID="1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5" Oct 30 04:44:56.797953 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.797932 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5"} err="failed to get container status \"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5\": rpc error: code = NotFound desc = could not find container \"1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5\": container with ID starting with 1823ae3fb412a0cda719cacc1bad650314f132dc160fa5ef7ec6c1606ce2cbe5 not found: ID does not exist" Oct 30 04:44:56.798038 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.797954 6587 scope.go:122] "RemoveContainer" containerID="2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65" Oct 30 04:44:56.798221 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.798201 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65"} err="failed to get container status \"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65\": rpc error: code = NotFound desc = could not find container \"2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65\": container with ID starting with 2c7a747ff9492364e57a5a5bbec0436b01a15d75e15bd72cff7b90ed3fe20b65 not found: ID does not exist" Oct 30 04:44:56.798277 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.798221 6587 scope.go:122] "RemoveContainer" containerID="04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5" Oct 30 04:44:56.798643 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.798546 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5"} err="failed to get container status \"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5\": rpc error: code = NotFound desc = could not find container \"04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5\": container with ID starting with 04948d455a75b02e71f57bb0fc530e886c86e85c03c23b79921c4fec63bb06d5 not found: ID does not exist" Oct 30 04:44:56.798643 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.798577 6587 scope.go:122] "RemoveContainer" containerID="ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a" Oct 30 04:44:56.798867 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.798835 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a"} err="failed to get container status \"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a\": rpc error: code = NotFound desc = could not find container \"ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a\": container with ID starting with ca35ae83d1813e717e86a07d2f96c3cebe12ae868aa65f869efc241e3821994a not found: ID does not exist" Oct 30 04:44:56.798867 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.798866 6587 scope.go:122] "RemoveContainer" containerID="70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c" Oct 30 04:44:56.799124 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.799105 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c"} err="failed to get container status \"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c\": rpc error: code = NotFound desc = could not find container \"70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c\": container with ID starting with 70ec03a0cadf3a0cdb9332894a665bf0b2cfebe87342483cf9f5f5cb7c36a77c not found: ID does not exist" Oct 30 04:44:56.799198 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.799130 6587 scope.go:122] "RemoveContainer" containerID="3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148" Oct 30 04:44:56.799400 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.799381 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148"} err="failed to get container status \"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148\": rpc error: code = NotFound desc = could not find container \"3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148\": container with ID starting with 3748b738b84de2f98b65ac66706b4b1e1b5ee49d764958cd570d4bf7bd46e148 not found: ID does not exist" Oct 30 04:44:56.799400 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.799400 6587 scope.go:122] "RemoveContainer" containerID="f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7" Oct 30 04:44:56.799642 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.799620 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7"} err="failed to get container status \"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7\": rpc error: code = NotFound desc = could not find container \"f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7\": container with ID starting with f91aa7b7359f5c6f7cdb39a760d3c3079c99e48271466c039ad5eab7b646aff7 not found: ID does not exist" Oct 30 04:44:56.814140 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814119 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 04:44:56.814466 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814443 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="kube-rbac-proxy" Oct 30 04:44:56.814466 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814463 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="kube-rbac-proxy" Oct 30 04:44:56.814585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814476 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="init-config-reloader" Oct 30 04:44:56.814585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814481 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="init-config-reloader" Oct 30 04:44:56.814585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814491 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="config-reloader" Oct 30 04:44:56.814585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814496 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="config-reloader" Oct 30 04:44:56.814585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814502 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="prometheus" Oct 30 04:44:56.814585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814507 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="prometheus" Oct 30 04:44:56.814585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814514 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="kube-rbac-proxy-web" Oct 30 04:44:56.814585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814518 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="kube-rbac-proxy-web" Oct 30 04:44:56.814585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814527 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="kube-rbac-proxy-thanos" Oct 30 04:44:56.814585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814535 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="kube-rbac-proxy-thanos" Oct 30 04:44:56.814585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814551 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="thanos-sidecar" Oct 30 04:44:56.814585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814555 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="thanos-sidecar" Oct 30 04:44:56.815121 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814598 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="init-config-reloader" Oct 30 04:44:56.815121 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814611 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="config-reloader" Oct 30 04:44:56.815121 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814620 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="prometheus" Oct 30 04:44:56.815121 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814627 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="kube-rbac-proxy-web" Oct 30 04:44:56.815121 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814632 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="kube-rbac-proxy-thanos" Oct 30 04:44:56.815121 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814639 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="thanos-sidecar" Oct 30 04:44:56.815121 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.814646 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc295553-de72-4758-83ad-7123e5e29f03" containerName="kube-rbac-proxy" Oct 30 04:44:56.818104 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.818088 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.835084 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835038 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 04:44:56.835084 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835045 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hsqv5\"" Oct 30 04:44:56.835192 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835125 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 04:44:56.835192 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835170 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 04:44:56.835301 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835287 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 04:44:56.835384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835341 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 04:44:56.835469 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835450 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 04:44:56.835518 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835504 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 04:44:56.835558 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835528 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 04:44:56.835607 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835594 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 04:44:56.835681 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835668 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 04:44:56.835725 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835450 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3fklq5stm461m\"" Oct 30 04:44:56.835756 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835722 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 04:44:56.835756 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.835730 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 04:44:56.849818 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.849796 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 04:44:56.849943 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.849924 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 04:44:56.850716 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.850691 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 04:44:56.882373 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.882328 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.882521 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.882392 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.882521 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.882461 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-config\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.882664 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.882556 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.882664 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.882592 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.882664 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.882624 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-web-config\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.882852 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.882673 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.882852 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.882737 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.882852 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.882774 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8aeae53-ab5d-4890-9d61-f6402edc264d-config-out\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.882852 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.882825 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8aeae53-ab5d-4890-9d61-f6402edc264d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.883042 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.882891 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.883042 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.882924 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.883042 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.882999 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.883042 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.883027 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpbc5\" (UniqueName: \"kubernetes.io/projected/f8aeae53-ab5d-4890-9d61-f6402edc264d-kube-api-access-hpbc5\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.883371 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.883059 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.883371 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.883101 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.883371 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.883159 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.883371 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.883242 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.883371 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.883284 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.883371 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.883325 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.984605 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.984568 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.984605 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.984604 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.984845 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.984621 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-web-config\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.984845 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.984638 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.984845 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.984753 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.984845 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.984803 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8aeae53-ab5d-4890-9d61-f6402edc264d-config-out\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.985047 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.984946 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8aeae53-ab5d-4890-9d61-f6402edc264d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.985047 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.984992 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.985836 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.986027 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.986314 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.986386 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.986435 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.986462 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpbc5\" (UniqueName: \"kubernetes.io/projected/f8aeae53-ab5d-4890-9d61-f6402edc264d-kube-api-access-hpbc5\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.986511 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.986608 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.986680 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.986718 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.986844 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.986910 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.986950 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.987057 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.987214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.987086 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-config\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.988486 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.988315 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8aeae53-ab5d-4890-9d61-f6402edc264d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.989593 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.988890 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.989593 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.989032 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.989593 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.989241 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.990307 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.990282 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-config\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.990940 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.990608 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.991088 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.991064 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.992111 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.992085 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.992742 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.992391 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.993048 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.993027 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8aeae53-ab5d-4890-9d61-f6402edc264d-config-out\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.993621 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.993467 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.993621 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.993480 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.994695 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.994658 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8aeae53-ab5d-4890-9d61-f6402edc264d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.994837 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.994814 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.995240 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.995065 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-web-config\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:56.997197 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:56.997174 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8aeae53-ab5d-4890-9d61-f6402edc264d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:57.008713 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:57.008693 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpbc5\" (UniqueName: \"kubernetes.io/projected/f8aeae53-ab5d-4890-9d61-f6402edc264d-kube-api-access-hpbc5\") pod \"prometheus-k8s-0\" (UID: \"f8aeae53-ab5d-4890-9d61-f6402edc264d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:57.137099 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:57.137024 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:44:57.461528 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:57.461491 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 04:44:57.462412 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:44:57.462387 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8aeae53_ab5d_4890_9d61_f6402edc264d.slice/crio-5540231808a3b88ba01c89373113d5252fea26dbce481cd689f7858bf0fb9dc1 WatchSource:0}: Error finding container 5540231808a3b88ba01c89373113d5252fea26dbce481cd689f7858bf0fb9dc1: Status 404 returned error can't find the container with id 5540231808a3b88ba01c89373113d5252fea26dbce481cd689f7858bf0fb9dc1 Oct 30 04:44:57.578336 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:57.578310 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc295553-de72-4758-83ad-7123e5e29f03" path="/var/lib/kubelet/pods/fc295553-de72-4758-83ad-7123e5e29f03/volumes" Oct 30 04:44:57.733094 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:57.733064 6587 generic.go:356] "Generic (PLEG): container finished" podID="f8aeae53-ab5d-4890-9d61-f6402edc264d" containerID="33d7995a3b4479fc07d59091fa7a388f246a9951126430f0cc514feb1a7dba00" exitCode=0 Oct 30 04:44:57.733468 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:57.733171 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8aeae53-ab5d-4890-9d61-f6402edc264d","Type":"ContainerDied","Data":"33d7995a3b4479fc07d59091fa7a388f246a9951126430f0cc514feb1a7dba00"} Oct 30 04:44:57.733468 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:57.733255 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8aeae53-ab5d-4890-9d61-f6402edc264d","Type":"ContainerStarted","Data":"5540231808a3b88ba01c89373113d5252fea26dbce481cd689f7858bf0fb9dc1"} Oct 30 04:44:57.735604 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:57.735584 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" event={"ID":"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc","Type":"ContainerStarted","Data":"afaccb964ce95f29a9f5e6796f70a33b7e6bcdb003d96b5088f3367bff16896b"} Oct 30 04:44:58.741101 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:58.741064 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" event={"ID":"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc","Type":"ContainerStarted","Data":"07b6dd40003e1ee77cd5280374571e5ff8127803fbd5f2b6ce687b2538801eca"} Oct 30 04:44:58.741560 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:58.741117 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" event={"ID":"dabdaf17-12d9-48bc-94e3-fac2c28cc3dc","Type":"ContainerStarted","Data":"48703cacd0462589995850f6652b1d5399e5a1274cc156c41a974b348817d919"} Oct 30 04:44:58.744113 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:58.744090 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8aeae53-ab5d-4890-9d61-f6402edc264d","Type":"ContainerStarted","Data":"b85e697e278f4b8f68937ad94f471d2000467c24a377e830afc00f6230351265"} Oct 30 04:44:58.744221 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:58.744140 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8aeae53-ab5d-4890-9d61-f6402edc264d","Type":"ContainerStarted","Data":"95f24af5d4bd145fab19847453c3315bd3adf2e35481a3ee95bca247bd2f7b3a"} Oct 30 04:44:58.744221 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:58.744165 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8aeae53-ab5d-4890-9d61-f6402edc264d","Type":"ContainerStarted","Data":"50a148a6b68177c3ff9b9c3b3d785c8596bb3f4d1d8379b5899d2f729b9563a7"} Oct 30 04:44:58.744221 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:58.744196 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8aeae53-ab5d-4890-9d61-f6402edc264d","Type":"ContainerStarted","Data":"2d7c90736cc0794b25ec31299a5b3edb7412d419202985f1dc5a346d37baa818"} Oct 30 04:44:58.744221 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:58.744218 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8aeae53-ab5d-4890-9d61-f6402edc264d","Type":"ContainerStarted","Data":"bc8c6347edf0d46cf7062fc8a128a564e77890ae53b433b831738a83f99786de"} Oct 30 04:44:58.744375 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:58.744239 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8aeae53-ab5d-4890-9d61-f6402edc264d","Type":"ContainerStarted","Data":"d7d6e02fc6ad8a8306fcbf863e31145ce3eb17788b892961230ab09969e2d70e"} Oct 30 04:44:58.787220 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:58.787170 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5cd9d8f774-qwxql" podStartSLOduration=2.194282176 podStartE2EDuration="3.787158749s" totalImagesPullingTime="1.592876573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:58.786092562 +0000 UTC m=+23.313859347" watchObservedRunningTime="2026-10-30 04:44:58.787158749 +0000 UTC m=+23.314925534" Oct 30 04:44:58.836821 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:44:58.836773 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8118336360000002 podStartE2EDuration="2.836761012s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.927376ms" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:58.836453836 +0000 UTC m=+23.364220623" watchObservedRunningTime="2026-10-30 04:44:58.836761012 +0000 UTC m=+23.364527822" Oct 30 04:45:02.137149 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:45:02.137115 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:45:06.150078 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:45:06.150049 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tk9mv" Oct 30 04:45:12.711773 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:45:12.711739 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bblhz" Oct 30 04:45:25.983189 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:45:25.983140 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:45:44.807379 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:45:44.807330 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:45:44.822879 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:45:44.822855 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:45:44.926542 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:45:44.926513 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 04:45:55.982862 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:45:55.982827 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:46:25.983418 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:46:25.983323 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:46:55.983376 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:46:55.983327 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:47:25.983197 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:47:25.983154 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:47:55.984089 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:47:55.984007 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:48:25.983813 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:48:25.983777 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:48:30.554725 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:30.554692 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-dnkvm"] Oct 30 04:48:30.557928 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:30.557909 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dnkvm" Oct 30 04:48:30.576196 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:30.576177 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dnkvm"] Oct 30 04:48:30.588750 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:30.588724 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 04:48:30.588841 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:30.588765 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lwsln\"" Oct 30 04:48:30.588841 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:30.588764 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 04:48:30.633829 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:30.633802 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcj7v\" (UniqueName: \"kubernetes.io/projected/d4654e29-2c2c-4f8e-a742-9d06385a74f5-kube-api-access-dcj7v\") pod \"cert-manager-597b95cb84-dnkvm\" (UID: \"d4654e29-2c2c-4f8e-a742-9d06385a74f5\") " pod="cert-manager/cert-manager-597b95cb84-dnkvm" Oct 30 04:48:30.633971 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:30.633868 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4654e29-2c2c-4f8e-a742-9d06385a74f5-bound-sa-token\") pod \"cert-manager-597b95cb84-dnkvm\" (UID: \"d4654e29-2c2c-4f8e-a742-9d06385a74f5\") " pod="cert-manager/cert-manager-597b95cb84-dnkvm" Oct 30 04:48:30.734683 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:30.734645 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4654e29-2c2c-4f8e-a742-9d06385a74f5-bound-sa-token\") pod \"cert-manager-597b95cb84-dnkvm\" (UID: \"d4654e29-2c2c-4f8e-a742-9d06385a74f5\") " pod="cert-manager/cert-manager-597b95cb84-dnkvm" Oct 30 04:48:30.734858 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:30.734710 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcj7v\" (UniqueName: \"kubernetes.io/projected/d4654e29-2c2c-4f8e-a742-9d06385a74f5-kube-api-access-dcj7v\") pod \"cert-manager-597b95cb84-dnkvm\" (UID: \"d4654e29-2c2c-4f8e-a742-9d06385a74f5\") " pod="cert-manager/cert-manager-597b95cb84-dnkvm" Oct 30 04:48:30.755379 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:30.755341 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4654e29-2c2c-4f8e-a742-9d06385a74f5-bound-sa-token\") pod \"cert-manager-597b95cb84-dnkvm\" (UID: \"d4654e29-2c2c-4f8e-a742-9d06385a74f5\") " pod="cert-manager/cert-manager-597b95cb84-dnkvm" Oct 30 04:48:30.755521 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:30.755434 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcj7v\" (UniqueName: \"kubernetes.io/projected/d4654e29-2c2c-4f8e-a742-9d06385a74f5-kube-api-access-dcj7v\") pod \"cert-manager-597b95cb84-dnkvm\" (UID: \"d4654e29-2c2c-4f8e-a742-9d06385a74f5\") " pod="cert-manager/cert-manager-597b95cb84-dnkvm" Oct 30 04:48:30.884663 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:30.884577 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dnkvm" Oct 30 04:48:31.185227 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:31.185161 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dnkvm"] Oct 30 04:48:31.186954 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:48:31.186927 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4654e29_2c2c_4f8e_a742_9d06385a74f5.slice/crio-5625ad049ddfaf2df44efa5f897fd9348df388d372988ed9d20aca038c7042e0 WatchSource:0}: Error finding container 5625ad049ddfaf2df44efa5f897fd9348df388d372988ed9d20aca038c7042e0: Status 404 returned error can't find the container with id 5625ad049ddfaf2df44efa5f897fd9348df388d372988ed9d20aca038c7042e0 Oct 30 04:48:31.474144 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:31.474059 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dnkvm" event={"ID":"d4654e29-2c2c-4f8e-a742-9d06385a74f5","Type":"ContainerStarted","Data":"5625ad049ddfaf2df44efa5f897fd9348df388d372988ed9d20aca038c7042e0"} Oct 30 04:48:34.487166 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:34.487127 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dnkvm" event={"ID":"d4654e29-2c2c-4f8e-a742-9d06385a74f5","Type":"ContainerStarted","Data":"d0d6795dff89e504ad8a67b7b10f1e0c1141681cd251dba11db180d3358b6715"} Oct 30 04:48:53.081253 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.081203 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-dnkvm" podStartSLOduration=20.278979868 podStartE2EDuration="23.081190124s" totalImagesPullingTime="2.802210256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:48:34.53651249 +0000 UTC m=+239.064279279" watchObservedRunningTime="2026-10-30 04:48:53.081190124 +0000 UTC m=+257.608956945" Oct 30 04:48:53.081978 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.081961 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6"] Oct 30 04:48:53.085242 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.085224 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.102767 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.102739 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wr5jk\"" Oct 30 04:48:53.102879 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.102782 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 04:48:53.102965 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.102949 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 04:48:53.116040 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.116010 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 04:48:53.116171 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.116080 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 04:48:53.117396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.117373 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6"] Oct 30 04:48:53.186468 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.186433 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ac919fe3-ac98-44cd-8d95-6df70a406fa4-manifests\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.186468 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.186472 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac919fe3-ac98-44cd-8d95-6df70a406fa4-tmp\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.186675 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.186549 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ac919fe3-ac98-44cd-8d95-6df70a406fa4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.186675 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.186621 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkfpj\" (UniqueName: \"kubernetes.io/projected/ac919fe3-ac98-44cd-8d95-6df70a406fa4-kube-api-access-lkfpj\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.186675 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.186647 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac919fe3-ac98-44cd-8d95-6df70a406fa4-webhook-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.186773 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.186691 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac919fe3-ac98-44cd-8d95-6df70a406fa4-metrics-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.208068 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.208038 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 04:48:53.287076 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.287037 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ac919fe3-ac98-44cd-8d95-6df70a406fa4-manifests\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.287076 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.287083 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac919fe3-ac98-44cd-8d95-6df70a406fa4-tmp\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.287313 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.287116 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ac919fe3-ac98-44cd-8d95-6df70a406fa4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.287313 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.287138 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkfpj\" (UniqueName: \"kubernetes.io/projected/ac919fe3-ac98-44cd-8d95-6df70a406fa4-kube-api-access-lkfpj\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.287313 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.287158 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac919fe3-ac98-44cd-8d95-6df70a406fa4-webhook-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.287500 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.287310 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac919fe3-ac98-44cd-8d95-6df70a406fa4-metrics-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.287500 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.287457 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ac919fe3-ac98-44cd-8d95-6df70a406fa4-manifests\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.289322 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.289287 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac919fe3-ac98-44cd-8d95-6df70a406fa4-tmp\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.289482 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.289467 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac919fe3-ac98-44cd-8d95-6df70a406fa4-webhook-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.289658 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.289639 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac919fe3-ac98-44cd-8d95-6df70a406fa4-metrics-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.289698 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.289649 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ac919fe3-ac98-44cd-8d95-6df70a406fa4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.307048 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.307026 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkfpj\" (UniqueName: \"kubernetes.io/projected/ac919fe3-ac98-44cd-8d95-6df70a406fa4-kube-api-access-lkfpj\") pod \"opendatahub-operator-controller-manager-8484dd48c9-2v6v6\" (UID: \"ac919fe3-ac98-44cd-8d95-6df70a406fa4\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.399770 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.399700 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:53.754871 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:53.754837 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6"] Oct 30 04:48:54.558284 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:54.558242 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" event={"ID":"ac919fe3-ac98-44cd-8d95-6df70a406fa4","Type":"ContainerStarted","Data":"c18cb5778afd9dd23c704a10cd6d00c7815651e4fbc04404f5b4e24e5e733792"} Oct 30 04:48:55.987277 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:48:55.987239 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:48:56.567449 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:56.567417 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" event={"ID":"ac919fe3-ac98-44cd-8d95-6df70a406fa4","Type":"ContainerStarted","Data":"a695c318a751d1e1450ca122f527d588f06f5876ec5c2575b193603eb8acc5ec"} Oct 30 04:48:57.572219 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:57.572180 6587 generic.go:356] "Generic (PLEG): container finished" podID="ac919fe3-ac98-44cd-8d95-6df70a406fa4" containerID="a695c318a751d1e1450ca122f527d588f06f5876ec5c2575b193603eb8acc5ec" exitCode=0 Oct 30 04:48:57.572706 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:57.572248 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" event={"ID":"ac919fe3-ac98-44cd-8d95-6df70a406fa4","Type":"ContainerDied","Data":"a695c318a751d1e1450ca122f527d588f06f5876ec5c2575b193603eb8acc5ec"} Oct 30 04:48:58.577608 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:58.577572 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" event={"ID":"ac919fe3-ac98-44cd-8d95-6df70a406fa4","Type":"ContainerStarted","Data":"e7b2ec6e759b6da299bba84489ab76ba98ed62467fdd5196d74a7641954ab953"} Oct 30 04:48:58.578066 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:58.577687 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:48:58.656481 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:48:58.656430 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" podStartSLOduration=2.2138083809999998 podStartE2EDuration="5.656414914s" totalImagesPullingTime="3.028198966s" totalInitContainerRuntime="414.407567ms" isStatefulPod=false podCreationTimestamp="2026-10-30 04:48:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:48:58.642517277 +0000 UTC m=+263.170284073" watchObservedRunningTime="2026-10-30 04:48:58.656414914 +0000 UTC m=+263.184181694" Oct 30 04:49:09.583261 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:09.583232 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" Oct 30 04:49:11.752202 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.752164 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk"] Oct 30 04:49:11.757851 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.757832 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:11.788344 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.788314 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:49:11.788344 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.788336 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qck5r\"" Oct 30 04:49:11.788572 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.788370 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 04:49:11.788572 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.788425 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 04:49:11.788572 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.788429 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 04:49:11.788572 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.788456 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 04:49:11.789192 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.789171 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk"] Oct 30 04:49:11.836047 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.836024 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8db881b9-9efa-4501-a82f-53731d2c3625-cert\") pod \"lws-controller-manager-7f587b55d5-6ztqk\" (UID: \"8db881b9-9efa-4501-a82f-53731d2c3625\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:11.836156 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.836071 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8db881b9-9efa-4501-a82f-53731d2c3625-manager-config\") pod \"lws-controller-manager-7f587b55d5-6ztqk\" (UID: \"8db881b9-9efa-4501-a82f-53731d2c3625\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:11.836156 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.836146 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bhrp\" (UniqueName: \"kubernetes.io/projected/8db881b9-9efa-4501-a82f-53731d2c3625-kube-api-access-6bhrp\") pod \"lws-controller-manager-7f587b55d5-6ztqk\" (UID: \"8db881b9-9efa-4501-a82f-53731d2c3625\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:11.836240 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.836179 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8db881b9-9efa-4501-a82f-53731d2c3625-metrics-cert\") pod \"lws-controller-manager-7f587b55d5-6ztqk\" (UID: \"8db881b9-9efa-4501-a82f-53731d2c3625\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:11.936832 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.936803 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8db881b9-9efa-4501-a82f-53731d2c3625-cert\") pod \"lws-controller-manager-7f587b55d5-6ztqk\" (UID: \"8db881b9-9efa-4501-a82f-53731d2c3625\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:11.937005 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.936856 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8db881b9-9efa-4501-a82f-53731d2c3625-manager-config\") pod \"lws-controller-manager-7f587b55d5-6ztqk\" (UID: \"8db881b9-9efa-4501-a82f-53731d2c3625\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:11.937005 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.936980 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bhrp\" (UniqueName: \"kubernetes.io/projected/8db881b9-9efa-4501-a82f-53731d2c3625-kube-api-access-6bhrp\") pod \"lws-controller-manager-7f587b55d5-6ztqk\" (UID: \"8db881b9-9efa-4501-a82f-53731d2c3625\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:11.937138 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.937013 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8db881b9-9efa-4501-a82f-53731d2c3625-metrics-cert\") pod \"lws-controller-manager-7f587b55d5-6ztqk\" (UID: \"8db881b9-9efa-4501-a82f-53731d2c3625\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:11.937623 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.937594 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8db881b9-9efa-4501-a82f-53731d2c3625-manager-config\") pod \"lws-controller-manager-7f587b55d5-6ztqk\" (UID: \"8db881b9-9efa-4501-a82f-53731d2c3625\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:11.939410 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.939389 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8db881b9-9efa-4501-a82f-53731d2c3625-cert\") pod \"lws-controller-manager-7f587b55d5-6ztqk\" (UID: \"8db881b9-9efa-4501-a82f-53731d2c3625\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:11.939496 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.939445 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8db881b9-9efa-4501-a82f-53731d2c3625-metrics-cert\") pod \"lws-controller-manager-7f587b55d5-6ztqk\" (UID: \"8db881b9-9efa-4501-a82f-53731d2c3625\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:11.956568 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:11.956547 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bhrp\" (UniqueName: \"kubernetes.io/projected/8db881b9-9efa-4501-a82f-53731d2c3625-kube-api-access-6bhrp\") pod \"lws-controller-manager-7f587b55d5-6ztqk\" (UID: \"8db881b9-9efa-4501-a82f-53731d2c3625\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:12.067606 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:12.067572 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:12.353751 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:12.353714 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk"] Oct 30 04:49:12.355398 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:49:12.355351 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db881b9_9efa_4501_a82f_53731d2c3625.slice/crio-e57925a8b264f16c97adaf75ce55198df2835846e0ec3a6733787e8d6e8ffa99 WatchSource:0}: Error finding container e57925a8b264f16c97adaf75ce55198df2835846e0ec3a6733787e8d6e8ffa99: Status 404 returned error can't find the container with id e57925a8b264f16c97adaf75ce55198df2835846e0ec3a6733787e8d6e8ffa99 Oct 30 04:49:12.623632 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:12.623555 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" event={"ID":"8db881b9-9efa-4501-a82f-53731d2c3625","Type":"ContainerStarted","Data":"e57925a8b264f16c97adaf75ce55198df2835846e0ec3a6733787e8d6e8ffa99"} Oct 30 04:49:15.635635 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.635595 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" event={"ID":"8db881b9-9efa-4501-a82f-53731d2c3625","Type":"ContainerStarted","Data":"b9f3bf139a731d442827a390f8872b3aa6416b7a643d6de12a1bce324622e0ea"} Oct 30 04:49:15.636209 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.635690 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:15.676653 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.676603 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" podStartSLOduration=2.445365518 podStartE2EDuration="4.676586989s" totalImagesPullingTime="2.231221471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:15.675381255 +0000 UTC m=+280.203148039" watchObservedRunningTime="2026-10-30 04:49:15.676586989 +0000 UTC m=+280.204353774" Oct 30 04:49:15.867769 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.867736 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54668d58ff-2xllz"] Oct 30 04:49:15.872121 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.872100 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" Oct 30 04:49:15.893449 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.889914 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 04:49:15.893449 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.889926 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 04:49:15.893449 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.890819 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54668d58ff-2xllz"] Oct 30 04:49:15.910448 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.910420 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hmrrz\"" Oct 30 04:49:15.910448 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.910420 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 04:49:15.910676 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.910515 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 04:49:15.970457 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.970426 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddc306dd-f974-4fd8-abb7-067320d7ca83-tmp\") pod \"kube-auth-proxy-54668d58ff-2xllz\" (UID: \"ddc306dd-f974-4fd8-abb7-067320d7ca83\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" Oct 30 04:49:15.970639 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.970465 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc306dd-f974-4fd8-abb7-067320d7ca83-tls-certs\") pod \"kube-auth-proxy-54668d58ff-2xllz\" (UID: \"ddc306dd-f974-4fd8-abb7-067320d7ca83\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" Oct 30 04:49:15.970639 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:15.970573 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcndt\" (UniqueName: \"kubernetes.io/projected/ddc306dd-f974-4fd8-abb7-067320d7ca83-kube-api-access-lcndt\") pod \"kube-auth-proxy-54668d58ff-2xllz\" (UID: \"ddc306dd-f974-4fd8-abb7-067320d7ca83\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" Oct 30 04:49:16.071103 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:16.071067 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddc306dd-f974-4fd8-abb7-067320d7ca83-tmp\") pod \"kube-auth-proxy-54668d58ff-2xllz\" (UID: \"ddc306dd-f974-4fd8-abb7-067320d7ca83\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" Oct 30 04:49:16.071289 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:16.071112 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc306dd-f974-4fd8-abb7-067320d7ca83-tls-certs\") pod \"kube-auth-proxy-54668d58ff-2xllz\" (UID: \"ddc306dd-f974-4fd8-abb7-067320d7ca83\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" Oct 30 04:49:16.071289 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:16.071146 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcndt\" (UniqueName: \"kubernetes.io/projected/ddc306dd-f974-4fd8-abb7-067320d7ca83-kube-api-access-lcndt\") pod \"kube-auth-proxy-54668d58ff-2xllz\" (UID: \"ddc306dd-f974-4fd8-abb7-067320d7ca83\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" Oct 30 04:49:16.073402 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:16.073379 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddc306dd-f974-4fd8-abb7-067320d7ca83-tmp\") pod \"kube-auth-proxy-54668d58ff-2xllz\" (UID: \"ddc306dd-f974-4fd8-abb7-067320d7ca83\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" Oct 30 04:49:16.073634 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:16.073611 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc306dd-f974-4fd8-abb7-067320d7ca83-tls-certs\") pod \"kube-auth-proxy-54668d58ff-2xllz\" (UID: \"ddc306dd-f974-4fd8-abb7-067320d7ca83\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" Oct 30 04:49:16.091209 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:16.091177 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcndt\" (UniqueName: \"kubernetes.io/projected/ddc306dd-f974-4fd8-abb7-067320d7ca83-kube-api-access-lcndt\") pod \"kube-auth-proxy-54668d58ff-2xllz\" (UID: \"ddc306dd-f974-4fd8-abb7-067320d7ca83\") " pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" Oct 30 04:49:16.185646 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:16.185294 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" Oct 30 04:49:16.476604 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:16.476579 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54668d58ff-2xllz"] Oct 30 04:49:16.477720 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:49:16.477697 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc306dd_f974_4fd8_abb7_067320d7ca83.slice/crio-af1f49b6395d56ca4c35d128f1313bf05a7c4d847b62501ab86917d62ec8cb27 WatchSource:0}: Error finding container af1f49b6395d56ca4c35d128f1313bf05a7c4d847b62501ab86917d62ec8cb27: Status 404 returned error can't find the container with id af1f49b6395d56ca4c35d128f1313bf05a7c4d847b62501ab86917d62ec8cb27 Oct 30 04:49:16.643156 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:16.643077 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" event={"ID":"ddc306dd-f974-4fd8-abb7-067320d7ca83","Type":"ContainerStarted","Data":"af1f49b6395d56ca4c35d128f1313bf05a7c4d847b62501ab86917d62ec8cb27"} Oct 30 04:49:19.656320 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:19.656284 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" event={"ID":"ddc306dd-f974-4fd8-abb7-067320d7ca83","Type":"ContainerStarted","Data":"88cd0a6abbd07717b2200664e5e5a11b6520041306020af504405142f1a2fb65"} Oct 30 04:49:19.695737 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:19.695693 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54668d58ff-2xllz" podStartSLOduration=2.3281149 podStartE2EDuration="4.695678501s" totalImagesPullingTime="2.367563601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:19.694963632 +0000 UTC m=+284.222730418" watchObservedRunningTime="2026-10-30 04:49:19.695678501 +0000 UTC m=+284.223445286" Oct 30 04:49:25.984684 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:49:25.984646 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:49:26.645291 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:26.645259 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-6ztqk" Oct 30 04:49:35.612079 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:35.612048 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 04:49:35.612602 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:49:35.612560 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 04:49:55.984278 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:49:55.984235 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:50:15.797336 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:15.797300 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-dp85z"] Oct 30 04:50:15.804409 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:15.804387 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" Oct 30 04:50:15.819982 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:15.819956 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-dp85z"] Oct 30 04:50:15.901735 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:15.901709 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 04:50:15.901735 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:15.901711 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 04:50:15.901735 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:15.901733 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-nrnjj\"" Oct 30 04:50:15.924700 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:15.924677 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68p2b\" (UniqueName: \"kubernetes.io/projected/5c2574b8-389f-46fa-bb0c-d144b95455c1-kube-api-access-68p2b\") pod \"odh-model-controller-f87fcf56c-dp85z\" (UID: \"5c2574b8-389f-46fa-bb0c-d144b95455c1\") " pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" Oct 30 04:50:15.924780 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:15.924724 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c2574b8-389f-46fa-bb0c-d144b95455c1-cert\") pod \"odh-model-controller-f87fcf56c-dp85z\" (UID: \"5c2574b8-389f-46fa-bb0c-d144b95455c1\") " pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" Oct 30 04:50:15.924815 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:15.924803 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2574b8-389f-46fa-bb0c-d144b95455c1-metrics-certs\") pod \"odh-model-controller-f87fcf56c-dp85z\" (UID: \"5c2574b8-389f-46fa-bb0c-d144b95455c1\") " pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" Oct 30 04:50:16.026113 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:16.026081 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c2574b8-389f-46fa-bb0c-d144b95455c1-cert\") pod \"odh-model-controller-f87fcf56c-dp85z\" (UID: \"5c2574b8-389f-46fa-bb0c-d144b95455c1\") " pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" Oct 30 04:50:16.026258 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:16.026128 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2574b8-389f-46fa-bb0c-d144b95455c1-metrics-certs\") pod \"odh-model-controller-f87fcf56c-dp85z\" (UID: \"5c2574b8-389f-46fa-bb0c-d144b95455c1\") " pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" Oct 30 04:50:16.026258 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:16.026190 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68p2b\" (UniqueName: \"kubernetes.io/projected/5c2574b8-389f-46fa-bb0c-d144b95455c1-kube-api-access-68p2b\") pod \"odh-model-controller-f87fcf56c-dp85z\" (UID: \"5c2574b8-389f-46fa-bb0c-d144b95455c1\") " pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" Oct 30 04:50:16.028647 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:16.028625 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2574b8-389f-46fa-bb0c-d144b95455c1-metrics-certs\") pod \"odh-model-controller-f87fcf56c-dp85z\" (UID: \"5c2574b8-389f-46fa-bb0c-d144b95455c1\") " pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" Oct 30 04:50:16.028790 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:16.028768 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c2574b8-389f-46fa-bb0c-d144b95455c1-cert\") pod \"odh-model-controller-f87fcf56c-dp85z\" (UID: \"5c2574b8-389f-46fa-bb0c-d144b95455c1\") " pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" Oct 30 04:50:16.048134 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:16.048114 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68p2b\" (UniqueName: \"kubernetes.io/projected/5c2574b8-389f-46fa-bb0c-d144b95455c1-kube-api-access-68p2b\") pod \"odh-model-controller-f87fcf56c-dp85z\" (UID: \"5c2574b8-389f-46fa-bb0c-d144b95455c1\") " pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" Oct 30 04:50:16.117516 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:16.117482 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" Oct 30 04:50:16.408275 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:16.408236 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-dp85z"] Oct 30 04:50:16.409453 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:50:16.409423 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2574b8_389f_46fa_bb0c_d144b95455c1.slice/crio-5763a0a7181ecbbd560dd3399645e60978bae8e99851db73d821479cba483635 WatchSource:0}: Error finding container 5763a0a7181ecbbd560dd3399645e60978bae8e99851db73d821479cba483635: Status 404 returned error can't find the container with id 5763a0a7181ecbbd560dd3399645e60978bae8e99851db73d821479cba483635 Oct 30 04:50:16.410788 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:16.410771 6587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:50:16.866877 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:16.866842 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" event={"ID":"5c2574b8-389f-46fa-bb0c-d144b95455c1","Type":"ContainerStarted","Data":"5763a0a7181ecbbd560dd3399645e60978bae8e99851db73d821479cba483635"} Oct 30 04:50:19.881875 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:19.881835 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" event={"ID":"5c2574b8-389f-46fa-bb0c-d144b95455c1","Type":"ContainerStarted","Data":"bc6fc1a943811c27fafe7e707578fb193840b51e1d66c69b810fce08c7d080a0"} Oct 30 04:50:19.882313 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:19.881919 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" Oct 30 04:50:19.924293 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:19.924242 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" podStartSLOduration=2.409126916 podStartE2EDuration="4.924227068s" totalImagesPullingTime="2.515100152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:50:19.923575906 +0000 UTC m=+344.451342692" watchObservedRunningTime="2026-10-30 04:50:19.924227068 +0000 UTC m=+344.451993854" Oct 30 04:50:25.983675 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:50:25.983633 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:50:30.888314 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:50:30.888282 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-dp85z" Oct 30 04:50:55.983273 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:50:55.983186 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:51:25.984292 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:51:25.984256 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:51:33.072154 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.072115 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz"] Oct 30 04:51:33.075603 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.075582 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:33.089797 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.089766 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz"] Oct 30 04:51:33.102544 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.102515 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 04:51:33.102681 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.102537 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 04:51:33.102681 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.102582 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4tst5\"" Oct 30 04:51:33.175087 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.175054 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnsjf\" (UniqueName: \"kubernetes.io/projected/9ac55eb9-a36f-4ff8-ab83-f227b9724868-kube-api-access-tnsjf\") pod \"kuadrant-operator-controller-manager-6b6fb56846-6nvmz\" (UID: \"9ac55eb9-a36f-4ff8-ab83-f227b9724868\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:33.175251 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.175093 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ac55eb9-a36f-4ff8-ab83-f227b9724868-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-6nvmz\" (UID: \"9ac55eb9-a36f-4ff8-ab83-f227b9724868\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:33.276391 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.276342 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnsjf\" (UniqueName: \"kubernetes.io/projected/9ac55eb9-a36f-4ff8-ab83-f227b9724868-kube-api-access-tnsjf\") pod \"kuadrant-operator-controller-manager-6b6fb56846-6nvmz\" (UID: \"9ac55eb9-a36f-4ff8-ab83-f227b9724868\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:33.276571 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.276411 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ac55eb9-a36f-4ff8-ab83-f227b9724868-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-6nvmz\" (UID: \"9ac55eb9-a36f-4ff8-ab83-f227b9724868\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:33.276786 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.276768 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ac55eb9-a36f-4ff8-ab83-f227b9724868-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-6nvmz\" (UID: \"9ac55eb9-a36f-4ff8-ab83-f227b9724868\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:33.303312 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.303282 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnsjf\" (UniqueName: \"kubernetes.io/projected/9ac55eb9-a36f-4ff8-ab83-f227b9724868-kube-api-access-tnsjf\") pod \"kuadrant-operator-controller-manager-6b6fb56846-6nvmz\" (UID: \"9ac55eb9-a36f-4ff8-ab83-f227b9724868\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:33.388188 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.388097 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:33.705377 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:33.705333 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz"] Oct 30 04:51:33.705538 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:51:33.705510 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac55eb9_a36f_4ff8_ab83_f227b9724868.slice/crio-016d2e019785bcba32d0473b3be9b8b1d16d3520cc3117498cf5f26725de6afb WatchSource:0}: Error finding container 016d2e019785bcba32d0473b3be9b8b1d16d3520cc3117498cf5f26725de6afb: Status 404 returned error can't find the container with id 016d2e019785bcba32d0473b3be9b8b1d16d3520cc3117498cf5f26725de6afb Oct 30 04:51:34.169279 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:34.169243 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" event={"ID":"9ac55eb9-a36f-4ff8-ab83-f227b9724868","Type":"ContainerStarted","Data":"016d2e019785bcba32d0473b3be9b8b1d16d3520cc3117498cf5f26725de6afb"} Oct 30 04:51:38.188745 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:38.188713 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" event={"ID":"9ac55eb9-a36f-4ff8-ab83-f227b9724868","Type":"ContainerStarted","Data":"f1ef6e2c5fdda65b10082b89bee038a6cb518a75d228e8dc0d39f552845e2d8e"} Oct 30 04:51:38.189143 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:38.188785 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:38.259952 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:38.259904 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" podStartSLOduration=0.892768157 podStartE2EDuration="5.259890071s" totalImagesPullingTime="4.367121914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:38.245157416 +0000 UTC m=+422.772924202" watchObservedRunningTime="2026-10-30 04:51:38.259890071 +0000 UTC m=+422.787656857" Oct 30 04:51:49.195723 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:49.195691 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:52.735140 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.735107 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz"] Oct 30 04:51:52.735630 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.735312 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" podUID="9ac55eb9-a36f-4ff8-ab83-f227b9724868" containerName="manager" containerID="cri-o://f1ef6e2c5fdda65b10082b89bee038a6cb518a75d228e8dc0d39f552845e2d8e" gracePeriod=2 Oct 30 04:51:52.736166 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.736147 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz"] Oct 30 04:51:52.748498 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.748464 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7"] Oct 30 04:51:52.748980 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.748957 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9ac55eb9-a36f-4ff8-ab83-f227b9724868" containerName="manager" Oct 30 04:51:52.749076 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.748982 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac55eb9-a36f-4ff8-ab83-f227b9724868" containerName="manager" Oct 30 04:51:52.749076 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.749065 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac55eb9-a36f-4ff8-ab83-f227b9724868" containerName="manager" Oct 30 04:51:52.750132 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:51:52.750103 6587 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-6nvmz\" is forbidden: User \"system:node:ip-10-0-134-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-105.ec2.internal' and this object" podUID="9ac55eb9-a36f-4ff8-ab83-f227b9724868" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:52.752015 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.751996 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" Oct 30 04:51:52.766898 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:51:52.766859 6587 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-6nvmz\" is forbidden: User \"system:node:ip-10-0-134-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-105.ec2.internal' and this object" podUID="9ac55eb9-a36f-4ff8-ab83-f227b9724868" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:52.782064 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.782035 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7"] Oct 30 04:51:52.852144 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.852105 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz7j2\" (UniqueName: \"kubernetes.io/projected/33e72376-c2c5-4422-830c-e754c73a89c6-kube-api-access-fz7j2\") pod \"kuadrant-operator-controller-manager-6b6fb56846-c2zb7\" (UID: \"33e72376-c2c5-4422-830c-e754c73a89c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" Oct 30 04:51:52.852264 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.852240 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33e72376-c2c5-4422-830c-e754c73a89c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-c2zb7\" (UID: \"33e72376-c2c5-4422-830c-e754c73a89c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" Oct 30 04:51:52.953064 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.953029 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33e72376-c2c5-4422-830c-e754c73a89c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-c2zb7\" (UID: \"33e72376-c2c5-4422-830c-e754c73a89c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" Oct 30 04:51:52.953221 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.953179 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz7j2\" (UniqueName: \"kubernetes.io/projected/33e72376-c2c5-4422-830c-e754c73a89c6-kube-api-access-fz7j2\") pod \"kuadrant-operator-controller-manager-6b6fb56846-c2zb7\" (UID: \"33e72376-c2c5-4422-830c-e754c73a89c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" Oct 30 04:51:52.953525 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.953504 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33e72376-c2c5-4422-830c-e754c73a89c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-c2zb7\" (UID: \"33e72376-c2c5-4422-830c-e754c73a89c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" Oct 30 04:51:52.977652 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.977631 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:52.980782 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:52.980761 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz7j2\" (UniqueName: \"kubernetes.io/projected/33e72376-c2c5-4422-830c-e754c73a89c6-kube-api-access-fz7j2\") pod \"kuadrant-operator-controller-manager-6b6fb56846-c2zb7\" (UID: \"33e72376-c2c5-4422-830c-e754c73a89c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" Oct 30 04:51:52.993064 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:51:52.993013 6587 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-6nvmz\" is forbidden: User \"system:node:ip-10-0-134-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-105.ec2.internal' and this object" podUID="9ac55eb9-a36f-4ff8-ab83-f227b9724868" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:53.122324 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.122292 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" Oct 30 04:51:53.154475 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.154445 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac55eb9-a36f-4ff8-ab83-f227b9724868-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ac55eb9-a36f-4ff8-ab83-f227b9724868-extensions-socket-volume\") pod \"9ac55eb9-a36f-4ff8-ab83-f227b9724868\" (UID: \"9ac55eb9-a36f-4ff8-ab83-f227b9724868\") " Oct 30 04:51:53.154629 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.154503 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac55eb9-a36f-4ff8-ab83-f227b9724868-kube-api-access-tnsjf\" (UniqueName: \"kubernetes.io/projected/9ac55eb9-a36f-4ff8-ab83-f227b9724868-kube-api-access-tnsjf\") pod \"9ac55eb9-a36f-4ff8-ab83-f227b9724868\" (UID: \"9ac55eb9-a36f-4ff8-ab83-f227b9724868\") " Oct 30 04:51:53.154950 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.154923 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac55eb9-a36f-4ff8-ab83-f227b9724868-extensions-socket-volume" pod "9ac55eb9-a36f-4ff8-ab83-f227b9724868" (UID: "9ac55eb9-a36f-4ff8-ab83-f227b9724868"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:51:53.156573 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.156553 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac55eb9-a36f-4ff8-ab83-f227b9724868-kube-api-access-tnsjf" pod "9ac55eb9-a36f-4ff8-ab83-f227b9724868" (UID: "9ac55eb9-a36f-4ff8-ab83-f227b9724868"). InnerVolumeSpecName "kube-api-access-tnsjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:51:53.252112 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.252019 6587 generic.go:356] "Generic (PLEG): container finished" podID="9ac55eb9-a36f-4ff8-ab83-f227b9724868" containerID="f1ef6e2c5fdda65b10082b89bee038a6cb518a75d228e8dc0d39f552845e2d8e" exitCode=0 Oct 30 04:51:53.252112 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.252063 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:53.252112 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.252076 6587 scope.go:122] "RemoveContainer" containerID="f1ef6e2c5fdda65b10082b89bee038a6cb518a75d228e8dc0d39f552845e2d8e" Oct 30 04:51:53.255468 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.255450 6587 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ac55eb9-a36f-4ff8-ab83-f227b9724868-extensions-socket-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:51:53.255547 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.255472 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnsjf\" (UniqueName: \"kubernetes.io/projected/9ac55eb9-a36f-4ff8-ab83-f227b9724868-kube-api-access-tnsjf\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:51:53.260520 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.260504 6587 scope.go:122] "RemoveContainer" containerID="f1ef6e2c5fdda65b10082b89bee038a6cb518a75d228e8dc0d39f552845e2d8e" Oct 30 04:51:53.260804 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:51:53.260778 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ef6e2c5fdda65b10082b89bee038a6cb518a75d228e8dc0d39f552845e2d8e\": container with ID starting with f1ef6e2c5fdda65b10082b89bee038a6cb518a75d228e8dc0d39f552845e2d8e not found: ID does not exist" containerID="f1ef6e2c5fdda65b10082b89bee038a6cb518a75d228e8dc0d39f552845e2d8e" Oct 30 04:51:53.260885 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.260817 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ef6e2c5fdda65b10082b89bee038a6cb518a75d228e8dc0d39f552845e2d8e"} err="failed to get container status \"f1ef6e2c5fdda65b10082b89bee038a6cb518a75d228e8dc0d39f552845e2d8e\": rpc error: code = NotFound desc = could not find container \"f1ef6e2c5fdda65b10082b89bee038a6cb518a75d228e8dc0d39f552845e2d8e\": container with ID starting with f1ef6e2c5fdda65b10082b89bee038a6cb518a75d228e8dc0d39f552845e2d8e not found: ID does not exist" Oct 30 04:51:53.266984 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:51:53.266957 6587 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-6nvmz\" is forbidden: User \"system:node:ip-10-0-134-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-105.ec2.internal' and this object" podUID="9ac55eb9-a36f-4ff8-ab83-f227b9724868" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:53.281656 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:51:53.281629 6587 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-6nvmz\" is forbidden: User \"system:node:ip-10-0-134-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-105.ec2.internal' and this object" podUID="9ac55eb9-a36f-4ff8-ab83-f227b9724868" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6nvmz" Oct 30 04:51:53.440857 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.440829 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7"] Oct 30 04:51:53.441499 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:51:53.441477 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e72376_c2c5_4422_830c_e754c73a89c6.slice/crio-38386e74ece3948d1818cd54985bea12d5135869d190811c50f9fa29232c7243 WatchSource:0}: Error finding container 38386e74ece3948d1818cd54985bea12d5135869d190811c50f9fa29232c7243: Status 404 returned error can't find the container with id 38386e74ece3948d1818cd54985bea12d5135869d190811c50f9fa29232c7243 Oct 30 04:51:53.578837 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:53.578808 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac55eb9-a36f-4ff8-ab83-f227b9724868" path="/var/lib/kubelet/pods/9ac55eb9-a36f-4ff8-ab83-f227b9724868/volumes" Oct 30 04:51:54.259109 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:54.259068 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" event={"ID":"33e72376-c2c5-4422-830c-e754c73a89c6","Type":"ContainerStarted","Data":"b62533498dd5de2172519ed203fe683feffa1b4a17e0a72fe69c99ff003af43b"} Oct 30 04:51:54.259649 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:54.259126 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" event={"ID":"33e72376-c2c5-4422-830c-e754c73a89c6","Type":"ContainerStarted","Data":"38386e74ece3948d1818cd54985bea12d5135869d190811c50f9fa29232c7243"} Oct 30 04:51:54.259649 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:54.259208 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" Oct 30 04:51:54.335000 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:51:54.334928 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" podStartSLOduration=2.334906243 podStartE2EDuration="2.334906243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:54.320235263 +0000 UTC m=+438.848002050" watchObservedRunningTime="2026-10-30 04:51:54.334906243 +0000 UTC m=+438.862673031" Oct 30 04:51:55.985168 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:51:55.985127 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:52:05.266633 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:05.266598 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" Oct 30 04:52:07.820698 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:07.820666 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7"] Oct 30 04:52:07.821190 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:07.820836 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" podUID="33e72376-c2c5-4422-830c-e754c73a89c6" containerName="manager" containerID="cri-o://b62533498dd5de2172519ed203fe683feffa1b4a17e0a72fe69c99ff003af43b" gracePeriod=10 Oct 30 04:52:08.071522 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.071467 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" Oct 30 04:52:08.082005 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.081979 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e72376-c2c5-4422-830c-e754c73a89c6-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33e72376-c2c5-4422-830c-e754c73a89c6-extensions-socket-volume\") pod \"33e72376-c2c5-4422-830c-e754c73a89c6\" (UID: \"33e72376-c2c5-4422-830c-e754c73a89c6\") " Oct 30 04:52:08.082131 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.082017 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e72376-c2c5-4422-830c-e754c73a89c6-kube-api-access-fz7j2\" (UniqueName: \"kubernetes.io/projected/33e72376-c2c5-4422-830c-e754c73a89c6-kube-api-access-fz7j2\") pod \"33e72376-c2c5-4422-830c-e754c73a89c6\" (UID: \"33e72376-c2c5-4422-830c-e754c73a89c6\") " Oct 30 04:52:08.082328 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.082306 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e72376-c2c5-4422-830c-e754c73a89c6-extensions-socket-volume" pod "33e72376-c2c5-4422-830c-e754c73a89c6" (UID: "33e72376-c2c5-4422-830c-e754c73a89c6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:52:08.084050 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.084018 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e72376-c2c5-4422-830c-e754c73a89c6-kube-api-access-fz7j2" pod "33e72376-c2c5-4422-830c-e754c73a89c6" (UID: "33e72376-c2c5-4422-830c-e754c73a89c6"). InnerVolumeSpecName "kube-api-access-fz7j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:52:08.103202 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.103178 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95"] Oct 30 04:52:08.103672 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.103648 6587 cpu_manager.go:414] "containerMap: removing container" podUID="33e72376-c2c5-4422-830c-e754c73a89c6" containerName="manager" Oct 30 04:52:08.103672 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.103668 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e72376-c2c5-4422-830c-e754c73a89c6" containerName="manager" Oct 30 04:52:08.103798 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.103763 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e72376-c2c5-4422-830c-e754c73a89c6" containerName="manager" Oct 30 04:52:08.106699 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.106680 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" Oct 30 04:52:08.154116 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.154080 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95"] Oct 30 04:52:08.183180 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.183139 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d296c368-73b0-4326-87bc-cd3508c843b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-zdd95\" (UID: \"d296c368-73b0-4326-87bc-cd3508c843b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" Oct 30 04:52:08.183331 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.183220 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8gr6\" (UniqueName: \"kubernetes.io/projected/d296c368-73b0-4326-87bc-cd3508c843b9-kube-api-access-j8gr6\") pod \"kuadrant-operator-controller-manager-57bb9bfff-zdd95\" (UID: \"d296c368-73b0-4326-87bc-cd3508c843b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" Oct 30 04:52:08.183331 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.183311 6587 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33e72376-c2c5-4422-830c-e754c73a89c6-extensions-socket-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:52:08.183331 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.183327 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz7j2\" (UniqueName: \"kubernetes.io/projected/33e72376-c2c5-4422-830c-e754c73a89c6-kube-api-access-fz7j2\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:52:08.284012 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.283967 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8gr6\" (UniqueName: \"kubernetes.io/projected/d296c368-73b0-4326-87bc-cd3508c843b9-kube-api-access-j8gr6\") pod \"kuadrant-operator-controller-manager-57bb9bfff-zdd95\" (UID: \"d296c368-73b0-4326-87bc-cd3508c843b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" Oct 30 04:52:08.284188 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.284044 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d296c368-73b0-4326-87bc-cd3508c843b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-zdd95\" (UID: \"d296c368-73b0-4326-87bc-cd3508c843b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" Oct 30 04:52:08.284398 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.284343 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d296c368-73b0-4326-87bc-cd3508c843b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-zdd95\" (UID: \"d296c368-73b0-4326-87bc-cd3508c843b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" Oct 30 04:52:08.311706 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.311676 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8gr6\" (UniqueName: \"kubernetes.io/projected/d296c368-73b0-4326-87bc-cd3508c843b9-kube-api-access-j8gr6\") pod \"kuadrant-operator-controller-manager-57bb9bfff-zdd95\" (UID: \"d296c368-73b0-4326-87bc-cd3508c843b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" Oct 30 04:52:08.315475 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.315449 6587 generic.go:356] "Generic (PLEG): container finished" podID="33e72376-c2c5-4422-830c-e754c73a89c6" containerID="b62533498dd5de2172519ed203fe683feffa1b4a17e0a72fe69c99ff003af43b" exitCode=0 Oct 30 04:52:08.315632 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.315495 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" Oct 30 04:52:08.315632 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.315496 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" event={"ID":"33e72376-c2c5-4422-830c-e754c73a89c6","Type":"ContainerDied","Data":"b62533498dd5de2172519ed203fe683feffa1b4a17e0a72fe69c99ff003af43b"} Oct 30 04:52:08.315704 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.315638 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7" event={"ID":"33e72376-c2c5-4422-830c-e754c73a89c6","Type":"ContainerDied","Data":"38386e74ece3948d1818cd54985bea12d5135869d190811c50f9fa29232c7243"} Oct 30 04:52:08.315704 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.315682 6587 scope.go:122] "RemoveContainer" containerID="b62533498dd5de2172519ed203fe683feffa1b4a17e0a72fe69c99ff003af43b" Oct 30 04:52:08.329855 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.329835 6587 scope.go:122] "RemoveContainer" containerID="b62533498dd5de2172519ed203fe683feffa1b4a17e0a72fe69c99ff003af43b" Oct 30 04:52:08.330117 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:52:08.330097 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b62533498dd5de2172519ed203fe683feffa1b4a17e0a72fe69c99ff003af43b\": container with ID starting with b62533498dd5de2172519ed203fe683feffa1b4a17e0a72fe69c99ff003af43b not found: ID does not exist" containerID="b62533498dd5de2172519ed203fe683feffa1b4a17e0a72fe69c99ff003af43b" Oct 30 04:52:08.330172 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.330125 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b62533498dd5de2172519ed203fe683feffa1b4a17e0a72fe69c99ff003af43b"} err="failed to get container status \"b62533498dd5de2172519ed203fe683feffa1b4a17e0a72fe69c99ff003af43b\": rpc error: code = NotFound desc = could not find container \"b62533498dd5de2172519ed203fe683feffa1b4a17e0a72fe69c99ff003af43b\": container with ID starting with b62533498dd5de2172519ed203fe683feffa1b4a17e0a72fe69c99ff003af43b not found: ID does not exist" Oct 30 04:52:08.372943 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.372919 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7"] Oct 30 04:52:08.375735 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.375712 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-c2zb7"] Oct 30 04:52:08.419122 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.419089 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" Oct 30 04:52:08.723294 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:08.723272 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95"] Oct 30 04:52:08.726073 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:52:08.726044 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd296c368_73b0_4326_87bc_cd3508c843b9.slice/crio-7034af9dbdb6729d5032ef93502d2bd5b8ad981c76eb61742d859877755384d9 WatchSource:0}: Error finding container 7034af9dbdb6729d5032ef93502d2bd5b8ad981c76eb61742d859877755384d9: Status 404 returned error can't find the container with id 7034af9dbdb6729d5032ef93502d2bd5b8ad981c76eb61742d859877755384d9 Oct 30 04:52:09.322435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:09.322397 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" event={"ID":"d296c368-73b0-4326-87bc-cd3508c843b9","Type":"ContainerStarted","Data":"bf05d17d075e966e7983d83256849e46458bc2a55dff08bfc8a5e7faaee7fc81"} Oct 30 04:52:09.322845 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:09.322457 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" event={"ID":"d296c368-73b0-4326-87bc-cd3508c843b9","Type":"ContainerStarted","Data":"7034af9dbdb6729d5032ef93502d2bd5b8ad981c76eb61742d859877755384d9"} Oct 30 04:52:09.322845 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:09.322485 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" Oct 30 04:52:09.397317 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:09.397260 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" podStartSLOduration=1.39724681 podStartE2EDuration="1.39724681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:09.382470659 +0000 UTC m=+453.910237445" watchObservedRunningTime="2026-10-30 04:52:09.39724681 +0000 UTC m=+453.925013595" Oct 30 04:52:09.578985 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:09.578895 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e72376-c2c5-4422-830c-e754c73a89c6" path="/var/lib/kubelet/pods/33e72376-c2c5-4422-830c-e754c73a89c6/volumes" Oct 30 04:52:20.329420 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:20.329319 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" Oct 30 04:52:25.983710 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:52:25.983671 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:52:41.428221 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:41.428184 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gcfv7"] Oct 30 04:52:41.433190 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:41.433170 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-gcfv7" Oct 30 04:52:41.450941 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:41.450914 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gcfv7"] Oct 30 04:52:41.462924 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:41.462889 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5g8hn\"" Oct 30 04:52:41.574587 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:41.574546 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxlz5\" (UniqueName: \"kubernetes.io/projected/9e33e80f-693a-46da-9541-914bac98da32-kube-api-access-fxlz5\") pod \"authorino-57f7f8659f-gcfv7\" (UID: \"9e33e80f-693a-46da-9541-914bac98da32\") " pod="kuadrant-system/authorino-57f7f8659f-gcfv7" Oct 30 04:52:41.675843 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:41.675807 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxlz5\" (UniqueName: \"kubernetes.io/projected/9e33e80f-693a-46da-9541-914bac98da32-kube-api-access-fxlz5\") pod \"authorino-57f7f8659f-gcfv7\" (UID: \"9e33e80f-693a-46da-9541-914bac98da32\") " pod="kuadrant-system/authorino-57f7f8659f-gcfv7" Oct 30 04:52:41.695656 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:41.695592 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxlz5\" (UniqueName: \"kubernetes.io/projected/9e33e80f-693a-46da-9541-914bac98da32-kube-api-access-fxlz5\") pod \"authorino-57f7f8659f-gcfv7\" (UID: \"9e33e80f-693a-46da-9541-914bac98da32\") " pod="kuadrant-system/authorino-57f7f8659f-gcfv7" Oct 30 04:52:41.746423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:41.746394 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-gcfv7" Oct 30 04:52:42.032818 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:42.032787 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gcfv7"] Oct 30 04:52:42.033833 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:52:42.033807 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e33e80f_693a_46da_9541_914bac98da32.slice/crio-3e34c263d214763e7907b3bea71243a3c216cfa80698e3386ffe69e5b7abba43 WatchSource:0}: Error finding container 3e34c263d214763e7907b3bea71243a3c216cfa80698e3386ffe69e5b7abba43: Status 404 returned error can't find the container with id 3e34c263d214763e7907b3bea71243a3c216cfa80698e3386ffe69e5b7abba43 Oct 30 04:52:42.240918 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:42.240888 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-f6ckp"] Oct 30 04:52:42.245395 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:42.245375 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-f6ckp" Oct 30 04:52:42.259599 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:42.259575 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-f6ckp"] Oct 30 04:52:42.380653 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:42.380625 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv67x\" (UniqueName: \"kubernetes.io/projected/92019dbf-d10b-45c8-b8c0-ced24cb52f90-kube-api-access-kv67x\") pod \"authorino-56c6c9ddd5-f6ckp\" (UID: \"92019dbf-d10b-45c8-b8c0-ced24cb52f90\") " pod="kuadrant-system/authorino-56c6c9ddd5-f6ckp" Oct 30 04:52:42.470155 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:42.470127 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-gcfv7" event={"ID":"9e33e80f-693a-46da-9541-914bac98da32","Type":"ContainerStarted","Data":"3e34c263d214763e7907b3bea71243a3c216cfa80698e3386ffe69e5b7abba43"} Oct 30 04:52:42.481617 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:42.481589 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv67x\" (UniqueName: \"kubernetes.io/projected/92019dbf-d10b-45c8-b8c0-ced24cb52f90-kube-api-access-kv67x\") pod \"authorino-56c6c9ddd5-f6ckp\" (UID: \"92019dbf-d10b-45c8-b8c0-ced24cb52f90\") " pod="kuadrant-system/authorino-56c6c9ddd5-f6ckp" Oct 30 04:52:42.501986 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:42.501966 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv67x\" (UniqueName: \"kubernetes.io/projected/92019dbf-d10b-45c8-b8c0-ced24cb52f90-kube-api-access-kv67x\") pod \"authorino-56c6c9ddd5-f6ckp\" (UID: \"92019dbf-d10b-45c8-b8c0-ced24cb52f90\") " pod="kuadrant-system/authorino-56c6c9ddd5-f6ckp" Oct 30 04:52:42.556822 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:42.556796 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-f6ckp" Oct 30 04:52:42.851978 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:42.851950 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-f6ckp"] Oct 30 04:52:42.853038 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:52:42.853008 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92019dbf_d10b_45c8_b8c0_ced24cb52f90.slice/crio-73e990da382ab88b3e15545b8773faf6d690b7c9296ce1df9c8470d3379a962f WatchSource:0}: Error finding container 73e990da382ab88b3e15545b8773faf6d690b7c9296ce1df9c8470d3379a962f: Status 404 returned error can't find the container with id 73e990da382ab88b3e15545b8773faf6d690b7c9296ce1df9c8470d3379a962f Oct 30 04:52:43.476121 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:43.476084 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-f6ckp" event={"ID":"92019dbf-d10b-45c8-b8c0-ced24cb52f90","Type":"ContainerStarted","Data":"73e990da382ab88b3e15545b8773faf6d690b7c9296ce1df9c8470d3379a962f"} Oct 30 04:52:44.481951 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:44.481915 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-gcfv7" event={"ID":"9e33e80f-693a-46da-9541-914bac98da32","Type":"ContainerStarted","Data":"67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81"} Oct 30 04:52:44.483377 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:44.483329 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-f6ckp" event={"ID":"92019dbf-d10b-45c8-b8c0-ced24cb52f90","Type":"ContainerStarted","Data":"bd63f9c3337430ffb21ff5a6d19a91c0547101ed87cd8a47a4a95c1923fde76b"} Oct 30 04:52:44.521460 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:44.521415 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-gcfv7" podStartSLOduration=2.040025316 podStartE2EDuration="3.521401652s" totalImagesPullingTime="1.481376336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:44.520111069 +0000 UTC m=+489.047877855" watchObservedRunningTime="2026-10-30 04:52:44.521401652 +0000 UTC m=+489.049168439" Oct 30 04:52:44.558610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:44.558565 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-f6ckp" podStartSLOduration=1.413323166 podStartE2EDuration="2.558551851s" totalImagesPullingTime="1.145228685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:44.557351672 +0000 UTC m=+489.085118459" watchObservedRunningTime="2026-10-30 04:52:44.558551851 +0000 UTC m=+489.086318665" Oct 30 04:52:44.577624 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:44.577585 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gcfv7"] Oct 30 04:52:46.491554 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:46.491497 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-gcfv7" podUID="9e33e80f-693a-46da-9541-914bac98da32" containerName="authorino" containerID="cri-o://67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81" gracePeriod=30 Oct 30 04:52:46.529389 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:52:46.529340 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e33e80f_693a_46da_9541_914bac98da32.slice/crio-67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81.scope\": RecentStats: unable to find data in memory cache]" Oct 30 04:52:46.742963 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:46.742898 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-gcfv7" Oct 30 04:52:46.923147 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:46.923110 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e33e80f-693a-46da-9541-914bac98da32-kube-api-access-fxlz5\" (UniqueName: \"kubernetes.io/projected/9e33e80f-693a-46da-9541-914bac98da32-kube-api-access-fxlz5\") pod \"9e33e80f-693a-46da-9541-914bac98da32\" (UID: \"9e33e80f-693a-46da-9541-914bac98da32\") " Oct 30 04:52:46.925318 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:46.925287 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e33e80f-693a-46da-9541-914bac98da32-kube-api-access-fxlz5" pod "9e33e80f-693a-46da-9541-914bac98da32" (UID: "9e33e80f-693a-46da-9541-914bac98da32"). InnerVolumeSpecName "kube-api-access-fxlz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:52:47.024619 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:47.024542 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxlz5\" (UniqueName: \"kubernetes.io/projected/9e33e80f-693a-46da-9541-914bac98da32-kube-api-access-fxlz5\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:52:47.496328 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:47.496294 6587 generic.go:356] "Generic (PLEG): container finished" podID="9e33e80f-693a-46da-9541-914bac98da32" containerID="67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81" exitCode=0 Oct 30 04:52:47.496820 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:47.496336 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-gcfv7" Oct 30 04:52:47.496820 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:47.496397 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-gcfv7" event={"ID":"9e33e80f-693a-46da-9541-914bac98da32","Type":"ContainerDied","Data":"67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81"} Oct 30 04:52:47.496820 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:47.496447 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-gcfv7" event={"ID":"9e33e80f-693a-46da-9541-914bac98da32","Type":"ContainerDied","Data":"3e34c263d214763e7907b3bea71243a3c216cfa80698e3386ffe69e5b7abba43"} Oct 30 04:52:47.496820 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:47.496474 6587 scope.go:122] "RemoveContainer" containerID="67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81" Oct 30 04:52:47.505580 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:47.505561 6587 scope.go:122] "RemoveContainer" containerID="67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81" Oct 30 04:52:47.505831 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:52:47.505812 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81\": container with ID starting with 67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81 not found: ID does not exist" containerID="67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81" Oct 30 04:52:47.505876 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:47.505841 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81"} err="failed to get container status \"67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81\": rpc error: code = NotFound desc = could not find container \"67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81\": container with ID starting with 67b81a9832f471601e853c4b7767629e591d100d93a914aeb834b7cee0b00f81 not found: ID does not exist" Oct 30 04:52:47.548086 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:47.548056 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gcfv7"] Oct 30 04:52:47.550168 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:47.550143 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gcfv7"] Oct 30 04:52:47.578517 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:52:47.578483 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e33e80f-693a-46da-9541-914bac98da32" path="/var/lib/kubelet/pods/9e33e80f-693a-46da-9541-914bac98da32/volumes" Oct 30 04:52:55.984801 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:52:55.984763 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:53:14.488463 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.488428 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-c9fdg"] Oct 30 04:53:14.488920 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.488773 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9e33e80f-693a-46da-9541-914bac98da32" containerName="authorino" Oct 30 04:53:14.488920 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.488785 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e33e80f-693a-46da-9541-914bac98da32" containerName="authorino" Oct 30 04:53:14.488920 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.488855 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e33e80f-693a-46da-9541-914bac98da32" containerName="authorino" Oct 30 04:53:14.494012 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.493993 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-c9fdg" Oct 30 04:53:14.508802 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.508778 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-c9fdg"] Oct 30 04:53:14.552067 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.552037 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l7xh\" (UniqueName: \"kubernetes.io/projected/6feb2e0b-4b56-4063-9fc8-8f8ebe767d11-kube-api-access-4l7xh\") pod \"authorino-6f6944bcdd-c9fdg\" (UID: \"6feb2e0b-4b56-4063-9fc8-8f8ebe767d11\") " pod="kuadrant-system/authorino-6f6944bcdd-c9fdg" Oct 30 04:53:14.653011 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.652981 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l7xh\" (UniqueName: \"kubernetes.io/projected/6feb2e0b-4b56-4063-9fc8-8f8ebe767d11-kube-api-access-4l7xh\") pod \"authorino-6f6944bcdd-c9fdg\" (UID: \"6feb2e0b-4b56-4063-9fc8-8f8ebe767d11\") " pod="kuadrant-system/authorino-6f6944bcdd-c9fdg" Oct 30 04:53:14.675101 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.675073 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-c9fdg"] Oct 30 04:53:14.675299 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:53:14.675279 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4l7xh], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-c9fdg" podUID="6feb2e0b-4b56-4063-9fc8-8f8ebe767d11" Oct 30 04:53:14.679252 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.679227 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l7xh\" (UniqueName: \"kubernetes.io/projected/6feb2e0b-4b56-4063-9fc8-8f8ebe767d11-kube-api-access-4l7xh\") pod \"authorino-6f6944bcdd-c9fdg\" (UID: \"6feb2e0b-4b56-4063-9fc8-8f8ebe767d11\") " pod="kuadrant-system/authorino-6f6944bcdd-c9fdg" Oct 30 04:53:14.696150 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.696122 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cp6cp"] Oct 30 04:53:14.700077 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.700059 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-cp6cp" Oct 30 04:53:14.717530 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.717506 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cp6cp"] Oct 30 04:53:14.717630 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.717509 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 04:53:14.753861 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.753805 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnr8n\" (UniqueName: \"kubernetes.io/projected/97dd191a-d58d-48f3-947a-a642660b8ab3-kube-api-access-cnr8n\") pod \"authorino-6889cbd8c4-cp6cp\" (UID: \"97dd191a-d58d-48f3-947a-a642660b8ab3\") " pod="kuadrant-system/authorino-6889cbd8c4-cp6cp" Oct 30 04:53:14.753861 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.753835 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97dd191a-d58d-48f3-947a-a642660b8ab3-tls-cert\") pod \"authorino-6889cbd8c4-cp6cp\" (UID: \"97dd191a-d58d-48f3-947a-a642660b8ab3\") " pod="kuadrant-system/authorino-6889cbd8c4-cp6cp" Oct 30 04:53:14.835433 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.835405 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cp6cp"] Oct 30 04:53:14.835630 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:53:14.835609 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cnr8n tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-cp6cp" podUID="97dd191a-d58d-48f3-947a-a642660b8ab3" Oct 30 04:53:14.854880 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.854851 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnr8n\" (UniqueName: \"kubernetes.io/projected/97dd191a-d58d-48f3-947a-a642660b8ab3-kube-api-access-cnr8n\") pod \"authorino-6889cbd8c4-cp6cp\" (UID: \"97dd191a-d58d-48f3-947a-a642660b8ab3\") " pod="kuadrant-system/authorino-6889cbd8c4-cp6cp" Oct 30 04:53:14.854989 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.854888 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97dd191a-d58d-48f3-947a-a642660b8ab3-tls-cert\") pod \"authorino-6889cbd8c4-cp6cp\" (UID: \"97dd191a-d58d-48f3-947a-a642660b8ab3\") " pod="kuadrant-system/authorino-6889cbd8c4-cp6cp" Oct 30 04:53:14.857282 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.857262 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97dd191a-d58d-48f3-947a-a642660b8ab3-tls-cert\") pod \"authorino-6889cbd8c4-cp6cp\" (UID: \"97dd191a-d58d-48f3-947a-a642660b8ab3\") " pod="kuadrant-system/authorino-6889cbd8c4-cp6cp" Oct 30 04:53:14.861588 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.861566 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b8b5dc7fb-bk2pm"] Oct 30 04:53:14.865409 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.865388 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" Oct 30 04:53:14.877438 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.877415 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnr8n\" (UniqueName: \"kubernetes.io/projected/97dd191a-d58d-48f3-947a-a642660b8ab3-kube-api-access-cnr8n\") pod \"authorino-6889cbd8c4-cp6cp\" (UID: \"97dd191a-d58d-48f3-947a-a642660b8ab3\") " pod="kuadrant-system/authorino-6889cbd8c4-cp6cp" Oct 30 04:53:14.881955 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.881929 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b8b5dc7fb-bk2pm"] Oct 30 04:53:14.955847 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.955819 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/939a88f4-2a1c-4113-b431-8690e1cb4b63-tls-cert\") pod \"authorino-6b8b5dc7fb-bk2pm\" (UID: \"939a88f4-2a1c-4113-b431-8690e1cb4b63\") " pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" Oct 30 04:53:14.956012 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:14.955901 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9jdf\" (UniqueName: \"kubernetes.io/projected/939a88f4-2a1c-4113-b431-8690e1cb4b63-kube-api-access-d9jdf\") pod \"authorino-6b8b5dc7fb-bk2pm\" (UID: \"939a88f4-2a1c-4113-b431-8690e1cb4b63\") " pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" Oct 30 04:53:15.057027 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.056994 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9jdf\" (UniqueName: \"kubernetes.io/projected/939a88f4-2a1c-4113-b431-8690e1cb4b63-kube-api-access-d9jdf\") pod \"authorino-6b8b5dc7fb-bk2pm\" (UID: \"939a88f4-2a1c-4113-b431-8690e1cb4b63\") " pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" Oct 30 04:53:15.057170 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.057050 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/939a88f4-2a1c-4113-b431-8690e1cb4b63-tls-cert\") pod \"authorino-6b8b5dc7fb-bk2pm\" (UID: \"939a88f4-2a1c-4113-b431-8690e1cb4b63\") " pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" Oct 30 04:53:15.059420 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.059394 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/939a88f4-2a1c-4113-b431-8690e1cb4b63-tls-cert\") pod \"authorino-6b8b5dc7fb-bk2pm\" (UID: \"939a88f4-2a1c-4113-b431-8690e1cb4b63\") " pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" Oct 30 04:53:15.076879 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.076855 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9jdf\" (UniqueName: \"kubernetes.io/projected/939a88f4-2a1c-4113-b431-8690e1cb4b63-kube-api-access-d9jdf\") pod \"authorino-6b8b5dc7fb-bk2pm\" (UID: \"939a88f4-2a1c-4113-b431-8690e1cb4b63\") " pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" Oct 30 04:53:15.177936 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.177903 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" Oct 30 04:53:15.466292 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.466263 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b8b5dc7fb-bk2pm"] Oct 30 04:53:15.466585 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:53:15.466563 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod939a88f4_2a1c_4113_b431_8690e1cb4b63.slice/crio-84ca9a6a2988a59dff75de46d15450d950945cd2fbcd029fb598f3fac0f9c4ea WatchSource:0}: Error finding container 84ca9a6a2988a59dff75de46d15450d950945cd2fbcd029fb598f3fac0f9c4ea: Status 404 returned error can't find the container with id 84ca9a6a2988a59dff75de46d15450d950945cd2fbcd029fb598f3fac0f9c4ea Oct 30 04:53:15.607324 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.607298 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-cp6cp" Oct 30 04:53:15.607324 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.607302 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" event={"ID":"939a88f4-2a1c-4113-b431-8690e1cb4b63","Type":"ContainerStarted","Data":"84ca9a6a2988a59dff75de46d15450d950945cd2fbcd029fb598f3fac0f9c4ea"} Oct 30 04:53:15.607801 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.607482 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-c9fdg" Oct 30 04:53:15.616054 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.616035 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-cp6cp" Oct 30 04:53:15.618432 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.618417 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-c9fdg" Oct 30 04:53:15.764199 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.764109 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97dd191a-d58d-48f3-947a-a642660b8ab3-kube-api-access-cnr8n\" (UniqueName: \"kubernetes.io/projected/97dd191a-d58d-48f3-947a-a642660b8ab3-kube-api-access-cnr8n\") pod \"97dd191a-d58d-48f3-947a-a642660b8ab3\" (UID: \"97dd191a-d58d-48f3-947a-a642660b8ab3\") " Oct 30 04:53:15.764199 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.764154 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97dd191a-d58d-48f3-947a-a642660b8ab3-tls-cert\" (UniqueName: \"kubernetes.io/secret/97dd191a-d58d-48f3-947a-a642660b8ab3-tls-cert\") pod \"97dd191a-d58d-48f3-947a-a642660b8ab3\" (UID: \"97dd191a-d58d-48f3-947a-a642660b8ab3\") " Oct 30 04:53:15.764540 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.764246 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6feb2e0b-4b56-4063-9fc8-8f8ebe767d11-kube-api-access-4l7xh\" (UniqueName: \"kubernetes.io/projected/6feb2e0b-4b56-4063-9fc8-8f8ebe767d11-kube-api-access-4l7xh\") pod \"6feb2e0b-4b56-4063-9fc8-8f8ebe767d11\" (UID: \"6feb2e0b-4b56-4063-9fc8-8f8ebe767d11\") " Oct 30 04:53:15.766346 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.766309 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dd191a-d58d-48f3-947a-a642660b8ab3-kube-api-access-cnr8n" pod "97dd191a-d58d-48f3-947a-a642660b8ab3" (UID: "97dd191a-d58d-48f3-947a-a642660b8ab3"). InnerVolumeSpecName "kube-api-access-cnr8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:53:15.766346 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.766328 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6feb2e0b-4b56-4063-9fc8-8f8ebe767d11-kube-api-access-4l7xh" pod "6feb2e0b-4b56-4063-9fc8-8f8ebe767d11" (UID: "6feb2e0b-4b56-4063-9fc8-8f8ebe767d11"). InnerVolumeSpecName "kube-api-access-4l7xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:53:15.766346 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.766319 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97dd191a-d58d-48f3-947a-a642660b8ab3-tls-cert" pod "97dd191a-d58d-48f3-947a-a642660b8ab3" (UID: "97dd191a-d58d-48f3-947a-a642660b8ab3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:53:15.864935 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.864903 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnr8n\" (UniqueName: \"kubernetes.io/projected/97dd191a-d58d-48f3-947a-a642660b8ab3-kube-api-access-cnr8n\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:53:15.864935 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.864928 6587 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97dd191a-d58d-48f3-947a-a642660b8ab3-tls-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:53:15.864935 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:15.864938 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l7xh\" (UniqueName: \"kubernetes.io/projected/6feb2e0b-4b56-4063-9fc8-8f8ebe767d11-kube-api-access-4l7xh\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:53:16.613149 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:16.613114 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" event={"ID":"939a88f4-2a1c-4113-b431-8690e1cb4b63","Type":"ContainerStarted","Data":"8d0bb06aa67f6d4b49f0ad2e640fa70ae6edf59f405aabab52a8e5a9db4cdaf1"} Oct 30 04:53:16.613643 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:16.613219 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-cp6cp" Oct 30 04:53:16.613643 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:16.613339 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-c9fdg" Oct 30 04:53:16.652976 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:16.652930 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" podStartSLOduration=2.045760654 podStartE2EDuration="2.652916206s" totalImagesPullingTime="607.155552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:53:16.65197569 +0000 UTC m=+521.179742474" watchObservedRunningTime="2026-10-30 04:53:16.652916206 +0000 UTC m=+521.180682992" Oct 30 04:53:16.676033 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:16.676008 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-f6ckp"] Oct 30 04:53:16.676237 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:16.676187 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-f6ckp" podUID="92019dbf-d10b-45c8-b8c0-ced24cb52f90" containerName="authorino" containerID="cri-o://bd63f9c3337430ffb21ff5a6d19a91c0547101ed87cd8a47a4a95c1923fde76b" gracePeriod=30 Oct 30 04:53:16.710502 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:16.710467 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-c9fdg"] Oct 30 04:53:16.714199 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:16.714170 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-c9fdg"] Oct 30 04:53:16.774476 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:16.774444 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cp6cp"] Oct 30 04:53:16.777575 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:16.777550 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cp6cp"] Oct 30 04:53:16.923901 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:16.923876 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-f6ckp" Oct 30 04:53:17.075400 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.075346 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92019dbf-d10b-45c8-b8c0-ced24cb52f90-kube-api-access-kv67x\" (UniqueName: \"kubernetes.io/projected/92019dbf-d10b-45c8-b8c0-ced24cb52f90-kube-api-access-kv67x\") pod \"92019dbf-d10b-45c8-b8c0-ced24cb52f90\" (UID: \"92019dbf-d10b-45c8-b8c0-ced24cb52f90\") " Oct 30 04:53:17.077473 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.077439 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92019dbf-d10b-45c8-b8c0-ced24cb52f90-kube-api-access-kv67x" pod "92019dbf-d10b-45c8-b8c0-ced24cb52f90" (UID: "92019dbf-d10b-45c8-b8c0-ced24cb52f90"). InnerVolumeSpecName "kube-api-access-kv67x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:53:17.176213 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.176139 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv67x\" (UniqueName: \"kubernetes.io/projected/92019dbf-d10b-45c8-b8c0-ced24cb52f90-kube-api-access-kv67x\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:53:17.578694 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.578657 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6feb2e0b-4b56-4063-9fc8-8f8ebe767d11" path="/var/lib/kubelet/pods/6feb2e0b-4b56-4063-9fc8-8f8ebe767d11/volumes" Oct 30 04:53:17.578934 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.578922 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97dd191a-d58d-48f3-947a-a642660b8ab3" path="/var/lib/kubelet/pods/97dd191a-d58d-48f3-947a-a642660b8ab3/volumes" Oct 30 04:53:17.618757 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.618725 6587 generic.go:356] "Generic (PLEG): container finished" podID="92019dbf-d10b-45c8-b8c0-ced24cb52f90" containerID="bd63f9c3337430ffb21ff5a6d19a91c0547101ed87cd8a47a4a95c1923fde76b" exitCode=0 Oct 30 04:53:17.619196 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.618792 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-f6ckp" Oct 30 04:53:17.619196 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.618810 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-f6ckp" event={"ID":"92019dbf-d10b-45c8-b8c0-ced24cb52f90","Type":"ContainerDied","Data":"bd63f9c3337430ffb21ff5a6d19a91c0547101ed87cd8a47a4a95c1923fde76b"} Oct 30 04:53:17.619196 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.618855 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-f6ckp" event={"ID":"92019dbf-d10b-45c8-b8c0-ced24cb52f90","Type":"ContainerDied","Data":"73e990da382ab88b3e15545b8773faf6d690b7c9296ce1df9c8470d3379a962f"} Oct 30 04:53:17.619196 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.618873 6587 scope.go:122] "RemoveContainer" containerID="bd63f9c3337430ffb21ff5a6d19a91c0547101ed87cd8a47a4a95c1923fde76b" Oct 30 04:53:17.628390 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.628351 6587 scope.go:122] "RemoveContainer" containerID="bd63f9c3337430ffb21ff5a6d19a91c0547101ed87cd8a47a4a95c1923fde76b" Oct 30 04:53:17.628660 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:53:17.628641 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd63f9c3337430ffb21ff5a6d19a91c0547101ed87cd8a47a4a95c1923fde76b\": container with ID starting with bd63f9c3337430ffb21ff5a6d19a91c0547101ed87cd8a47a4a95c1923fde76b not found: ID does not exist" containerID="bd63f9c3337430ffb21ff5a6d19a91c0547101ed87cd8a47a4a95c1923fde76b" Oct 30 04:53:17.628712 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.628669 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd63f9c3337430ffb21ff5a6d19a91c0547101ed87cd8a47a4a95c1923fde76b"} err="failed to get container status \"bd63f9c3337430ffb21ff5a6d19a91c0547101ed87cd8a47a4a95c1923fde76b\": rpc error: code = NotFound desc = could not find container \"bd63f9c3337430ffb21ff5a6d19a91c0547101ed87cd8a47a4a95c1923fde76b\": container with ID starting with bd63f9c3337430ffb21ff5a6d19a91c0547101ed87cd8a47a4a95c1923fde76b not found: ID does not exist" Oct 30 04:53:17.674295 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.674263 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-f6ckp"] Oct 30 04:53:17.675410 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:17.675388 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-f6ckp"] Oct 30 04:53:19.578847 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:19.578813 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92019dbf-d10b-45c8-b8c0-ced24cb52f90" path="/var/lib/kubelet/pods/92019dbf-d10b-45c8-b8c0-ced24cb52f90/volumes" Oct 30 04:53:25.984071 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:53:25.983969 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:53:30.859834 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:30.859800 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx"] Oct 30 04:53:30.860223 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:30.860120 6587 cpu_manager.go:414] "containerMap: removing container" podUID="92019dbf-d10b-45c8-b8c0-ced24cb52f90" containerName="authorino" Oct 30 04:53:30.860223 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:30.860130 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92019dbf-d10b-45c8-b8c0-ced24cb52f90" containerName="authorino" Oct 30 04:53:30.860223 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:30.860210 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="92019dbf-d10b-45c8-b8c0-ced24cb52f90" containerName="authorino" Oct 30 04:53:30.863233 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:30.863216 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:30.883331 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:30.883302 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx"] Oct 30 04:53:30.883503 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:30.883472 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 04:53:30.883582 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:30.883523 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 04:53:30.895175 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:30.895149 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 04:53:30.895319 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:30.895244 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-fq7pm\"" Oct 30 04:53:31.004608 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.004575 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stxll\" (UniqueName: \"kubernetes.io/projected/d02fa1f2-298d-4a23-a302-6a59e1cbd188-kube-api-access-stxll\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-tw5cx\" (UID: \"d02fa1f2-298d-4a23-a302-6a59e1cbd188\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:31.004608 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.004616 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d02fa1f2-298d-4a23-a302-6a59e1cbd188-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-tw5cx\" (UID: \"d02fa1f2-298d-4a23-a302-6a59e1cbd188\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:31.004834 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.004633 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d02fa1f2-298d-4a23-a302-6a59e1cbd188-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-tw5cx\" (UID: \"d02fa1f2-298d-4a23-a302-6a59e1cbd188\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:31.004834 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.004756 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d02fa1f2-298d-4a23-a302-6a59e1cbd188-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-tw5cx\" (UID: \"d02fa1f2-298d-4a23-a302-6a59e1cbd188\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:31.105323 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.105293 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stxll\" (UniqueName: \"kubernetes.io/projected/d02fa1f2-298d-4a23-a302-6a59e1cbd188-kube-api-access-stxll\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-tw5cx\" (UID: \"d02fa1f2-298d-4a23-a302-6a59e1cbd188\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:31.105505 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.105330 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d02fa1f2-298d-4a23-a302-6a59e1cbd188-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-tw5cx\" (UID: \"d02fa1f2-298d-4a23-a302-6a59e1cbd188\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:31.105505 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.105348 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d02fa1f2-298d-4a23-a302-6a59e1cbd188-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-tw5cx\" (UID: \"d02fa1f2-298d-4a23-a302-6a59e1cbd188\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:31.105505 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.105385 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d02fa1f2-298d-4a23-a302-6a59e1cbd188-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-tw5cx\" (UID: \"d02fa1f2-298d-4a23-a302-6a59e1cbd188\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:31.106032 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.106001 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d02fa1f2-298d-4a23-a302-6a59e1cbd188-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-tw5cx\" (UID: \"d02fa1f2-298d-4a23-a302-6a59e1cbd188\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:31.106198 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.106177 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d02fa1f2-298d-4a23-a302-6a59e1cbd188-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-tw5cx\" (UID: \"d02fa1f2-298d-4a23-a302-6a59e1cbd188\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:31.107802 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.107784 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d02fa1f2-298d-4a23-a302-6a59e1cbd188-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-tw5cx\" (UID: \"d02fa1f2-298d-4a23-a302-6a59e1cbd188\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:31.125301 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.125245 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stxll\" (UniqueName: \"kubernetes.io/projected/d02fa1f2-298d-4a23-a302-6a59e1cbd188-kube-api-access-stxll\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-tw5cx\" (UID: \"d02fa1f2-298d-4a23-a302-6a59e1cbd188\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:31.175546 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.175511 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:31.464712 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.464635 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx"] Oct 30 04:53:31.676988 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:31.676958 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" event={"ID":"d02fa1f2-298d-4a23-a302-6a59e1cbd188","Type":"ContainerStarted","Data":"c807af5ee406cc2ed0d11fd92c0302a074dc60acf63e8e6d1d5d1a2d56b9c8b6"} Oct 30 04:53:42.734546 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:42.734501 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" event={"ID":"d02fa1f2-298d-4a23-a302-6a59e1cbd188","Type":"ContainerStarted","Data":"1465550b0b57d7d261339e26ead135dd46ec08e6e5b3d877ce284be7b28859ab"} Oct 30 04:53:42.734949 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:42.734596 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:42.776508 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:42.776459 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" podStartSLOduration=1.644785278 podStartE2EDuration="12.776446926s" totalImagesPullingTime="11.131661648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:53:42.77572013 +0000 UTC m=+547.303486931" watchObservedRunningTime="2026-10-30 04:53:42.776446926 +0000 UTC m=+547.304213708" Oct 30 04:53:53.744032 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:53:53.743966 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" Oct 30 04:53:55.983642 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:53:55.983597 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:54:25.984081 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:54:25.984039 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:54:35.640329 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:54:35.640295 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 04:54:35.641507 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:54:35.641485 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 04:54:55.983755 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:54:55.983717 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:55:02.590996 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.590966 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb"] Oct 30 04:55:02.594341 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.594321 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.613979 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.613959 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 04:55:02.624278 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.624256 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 04:55:02.624410 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.624313 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m5wsj\"" Oct 30 04:55:02.624410 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.624340 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 04:55:02.626055 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.626036 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb"] Oct 30 04:55:02.706499 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.706471 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjgxh\" (UniqueName: \"kubernetes.io/projected/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-kube-api-access-fjgxh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.706668 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.706505 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.706717 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.706676 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.706753 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.706730 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.706852 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.706836 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.706894 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.706863 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.807374 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.807321 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.807536 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.807386 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.807536 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.807464 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgxh\" (UniqueName: \"kubernetes.io/projected/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-kube-api-access-fjgxh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.807536 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.807491 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.807662 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.807554 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.807662 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.807590 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.807883 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.807859 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.807955 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.807907 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.807955 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.807945 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.809920 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.809890 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.810120 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.810102 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.827018 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.826990 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjgxh\" (UniqueName: \"kubernetes.io/projected/cd43eb0e-0dca-47cd-9fda-bb4a0dbce266-kube-api-access-fjgxh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-cnqvb\" (UID: \"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:02.907521 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:02.907435 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:03.197122 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:03.197087 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb"] Oct 30 04:55:03.197648 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:55:03.197623 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd43eb0e_0dca_47cd_9fda_bb4a0dbce266.slice/crio-0245961469c4c59ca08ffd78f5cb21b1b3cf805d5ff90df1f0902f9674f2e59b WatchSource:0}: Error finding container 0245961469c4c59ca08ffd78f5cb21b1b3cf805d5ff90df1f0902f9674f2e59b: Status 404 returned error can't find the container with id 0245961469c4c59ca08ffd78f5cb21b1b3cf805d5ff90df1f0902f9674f2e59b Oct 30 04:55:04.071196 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:04.071161 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" event={"ID":"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266","Type":"ContainerStarted","Data":"0245961469c4c59ca08ffd78f5cb21b1b3cf805d5ff90df1f0902f9674f2e59b"} Oct 30 04:55:05.076594 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:05.076556 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" event={"ID":"cd43eb0e-0dca-47cd-9fda-bb4a0dbce266","Type":"ContainerStarted","Data":"e7b3b1b94374087b901813caef6a902a983688b1424dcfd490a0210c6784aa10"} Oct 30 04:55:05.118693 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:05.118642 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" podStartSLOduration=1.793698951 podStartE2EDuration="3.118627291s" totalImagesPullingTime="1.32492834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:05.117469718 +0000 UTC m=+629.645236503" watchObservedRunningTime="2026-10-30 04:55:05.118627291 +0000 UTC m=+629.646394076" Oct 30 04:55:05.908028 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:05.907983 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:06.354072 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.354034 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n"] Oct 30 04:55:06.358266 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.358240 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.376831 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.376798 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 04:55:06.390317 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.390291 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n"] Oct 30 04:55:06.544494 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.544463 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da31dd7c-961b-46e8-9f28-1e7e6780fa10-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.544664 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.544504 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da31dd7c-961b-46e8-9f28-1e7e6780fa10-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.544664 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.544650 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj9fk\" (UniqueName: \"kubernetes.io/projected/da31dd7c-961b-46e8-9f28-1e7e6780fa10-kube-api-access-qj9fk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.544776 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.544687 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da31dd7c-961b-46e8-9f28-1e7e6780fa10-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.544776 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.544710 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da31dd7c-961b-46e8-9f28-1e7e6780fa10-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.544883 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.544818 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da31dd7c-961b-46e8-9f28-1e7e6780fa10-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.646074 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.645998 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da31dd7c-961b-46e8-9f28-1e7e6780fa10-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.646074 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.646034 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da31dd7c-961b-46e8-9f28-1e7e6780fa10-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.646307 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.646095 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj9fk\" (UniqueName: \"kubernetes.io/projected/da31dd7c-961b-46e8-9f28-1e7e6780fa10-kube-api-access-qj9fk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.646307 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.646118 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da31dd7c-961b-46e8-9f28-1e7e6780fa10-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.646307 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.646132 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da31dd7c-961b-46e8-9f28-1e7e6780fa10-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.646500 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.646341 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da31dd7c-961b-46e8-9f28-1e7e6780fa10-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.646595 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.646572 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da31dd7c-961b-46e8-9f28-1e7e6780fa10-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.646819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.646787 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da31dd7c-961b-46e8-9f28-1e7e6780fa10-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.646919 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.646794 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da31dd7c-961b-46e8-9f28-1e7e6780fa10-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.648657 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.648627 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da31dd7c-961b-46e8-9f28-1e7e6780fa10-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.648796 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.648773 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da31dd7c-961b-46e8-9f28-1e7e6780fa10-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.666715 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.666682 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj9fk\" (UniqueName: \"kubernetes.io/projected/da31dd7c-961b-46e8-9f28-1e7e6780fa10-kube-api-access-qj9fk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n\" (UID: \"da31dd7c-961b-46e8-9f28-1e7e6780fa10\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.674395 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.674369 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:06.988371 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:06.988328 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n"] Oct 30 04:55:06.989406 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:55:06.989381 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda31dd7c_961b_46e8_9f28_1e7e6780fa10.slice/crio-8c5683ae7b2e762925feac38a4e82b4b6e65673a17ec373e3470f9e9f970aee7 WatchSource:0}: Error finding container 8c5683ae7b2e762925feac38a4e82b4b6e65673a17ec373e3470f9e9f970aee7: Status 404 returned error can't find the container with id 8c5683ae7b2e762925feac38a4e82b4b6e65673a17ec373e3470f9e9f970aee7 Oct 30 04:55:07.087619 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:07.087584 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" event={"ID":"da31dd7c-961b-46e8-9f28-1e7e6780fa10","Type":"ContainerStarted","Data":"8c5683ae7b2e762925feac38a4e82b4b6e65673a17ec373e3470f9e9f970aee7"} Oct 30 04:55:08.094552 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:08.094512 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" event={"ID":"da31dd7c-961b-46e8-9f28-1e7e6780fa10","Type":"ContainerStarted","Data":"526c8bd7f0475b16473c9a15db0dae29b1d18f073de1371747c4b9bb70bbeebd"} Oct 30 04:55:08.675340 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:08.675305 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:08.840305 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:08.840241 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" podStartSLOduration=2.667125268 podStartE2EDuration="2.840220784s" totalImagesPullingTime="173.095516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:08.138670542 +0000 UTC m=+632.666437338" watchObservedRunningTime="2026-10-30 04:55:08.840220784 +0000 UTC m=+633.367987572" Oct 30 04:55:08.842123 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:08.842087 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59"] Oct 30 04:55:08.848087 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:08.848025 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:08.864391 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:08.864345 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 04:55:08.869627 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:08.869602 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59"] Oct 30 04:55:08.970599 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:08.970516 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:08.970599 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:08.970565 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:08.970839 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:08.970724 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:08.970839 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:08.970773 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:08.970948 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:08.970849 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w5rv\" (UniqueName: \"kubernetes.io/projected/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-kube-api-access-4w5rv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:08.970948 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:08.970923 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.071917 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.071878 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.071917 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.071917 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.072162 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.072047 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.072162 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.072098 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.072162 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.072137 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w5rv\" (UniqueName: \"kubernetes.io/projected/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-kube-api-access-4w5rv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.072315 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.072209 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.072315 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.072295 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.072458 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.072388 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.072566 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.072536 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.074349 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.074329 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.074617 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.074602 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.100511 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.100483 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w5rv\" (UniqueName: \"kubernetes.io/projected/e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b-kube-api-access-4w5rv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59\" (UID: \"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.173375 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.173327 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:09.463604 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:09.463561 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59"] Oct 30 04:55:09.464306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:55:09.464278 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1c66fe9_17ee_4ae2_a1d4_3788a1a4937b.slice/crio-7fd4c9c34e64402c87e05b4e18292bc5445f646c2e50b1267ac88f208fd5ca75 WatchSource:0}: Error finding container 7fd4c9c34e64402c87e05b4e18292bc5445f646c2e50b1267ac88f208fd5ca75: Status 404 returned error can't find the container with id 7fd4c9c34e64402c87e05b4e18292bc5445f646c2e50b1267ac88f208fd5ca75 Oct 30 04:55:10.105869 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:10.105827 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" event={"ID":"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b","Type":"ContainerStarted","Data":"0ad7f429571048431bcb4351eceac10a98b2dad8f8cc142b5ca2d924a3743797"} Oct 30 04:55:10.106317 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:10.105884 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" event={"ID":"e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b","Type":"ContainerStarted","Data":"7fd4c9c34e64402c87e05b4e18292bc5445f646c2e50b1267ac88f208fd5ca75"} Oct 30 04:55:10.155119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:10.155058 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podStartSLOduration=1.953019252 podStartE2EDuration="2.155040325s" totalImagesPullingTime="202.021073ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:10.154161969 +0000 UTC m=+634.681928756" watchObservedRunningTime="2026-10-30 04:55:10.155040325 +0000 UTC m=+634.682807111" Oct 30 04:55:10.174399 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:10.174302 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:12.887031 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:12.886989 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8"] Oct 30 04:55:12.893384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:12.893337 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:12.908513 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:12.908490 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:12.909010 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:12.908983 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 04:55:12.924749 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:12.924720 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:12.924895 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:12.924785 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8"] Oct 30 04:55:13.014764 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.014720 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.014764 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.014759 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7l6p\" (UniqueName: \"kubernetes.io/projected/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-kube-api-access-c7l6p\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.014996 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.014849 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.014996 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.014874 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.015078 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.015003 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.015078 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.015061 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.116184 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.116146 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.116423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.116196 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7l6p\" (UniqueName: \"kubernetes.io/projected/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-kube-api-access-c7l6p\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.116423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.116260 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.116423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.116289 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.116423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.116346 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.116423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.116419 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.116959 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.116932 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.117203 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.117101 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.117338 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.117275 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.118968 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.118925 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.119729 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.119699 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.136501 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.136477 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" Oct 30 04:55:13.137271 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.137222 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7l6p\" (UniqueName: \"kubernetes.io/projected/c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7-kube-api-access-c7l6p\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dmbs8\" (UID: \"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.207104 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.207070 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:13.515432 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:13.515398 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8"] Oct 30 04:55:13.516764 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:55:13.516732 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04c6cd0_60c3_4a63_8115_d2af2c1cd1a7.slice/crio-6b078dbedf6b592562cd1fa1af201ce8a8dad0e6b2952d5ea7a123db87b4b804 WatchSource:0}: Error finding container 6b078dbedf6b592562cd1fa1af201ce8a8dad0e6b2952d5ea7a123db87b4b804: Status 404 returned error can't find the container with id 6b078dbedf6b592562cd1fa1af201ce8a8dad0e6b2952d5ea7a123db87b4b804 Oct 30 04:55:14.128441 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:14.128402 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" event={"ID":"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7","Type":"ContainerStarted","Data":"99c186853d86e4fa6a374a13660111bf8494c50ac0357c913d6f78620f16051e"} Oct 30 04:55:14.128860 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:14.128470 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" event={"ID":"c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7","Type":"ContainerStarted","Data":"6b078dbedf6b592562cd1fa1af201ce8a8dad0e6b2952d5ea7a123db87b4b804"} Oct 30 04:55:14.172418 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:14.172352 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" podStartSLOduration=1.9419606360000001 podStartE2EDuration="2.172337562s" totalImagesPullingTime="230.376926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:14.171321539 +0000 UTC m=+638.699088328" watchObservedRunningTime="2026-10-30 04:55:14.172337562 +0000 UTC m=+638.700104348" Oct 30 04:55:14.207542 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:14.207493 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:16.675376 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:16.675267 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:16.692111 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:16.692082 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:17.162665 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:17.162637 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 04:55:19.174171 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:19.174140 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:19.191254 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:19.191232 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:20.170288 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:20.170259 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:55:23.207976 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:23.207893 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:23.220433 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:23.220411 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:24.188680 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:24.188650 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" Oct 30 04:55:25.134709 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:25.134676 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59945dd659-9f5g7"] Oct 30 04:55:25.138223 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:25.138202 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59945dd659-9f5g7" Oct 30 04:55:25.155210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:25.155184 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59945dd659-9f5g7"] Oct 30 04:55:25.231765 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:25.231729 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8adbcc0-729e-4774-af4f-43d362118031-tls-cert\") pod \"authorino-59945dd659-9f5g7\" (UID: \"a8adbcc0-729e-4774-af4f-43d362118031\") " pod="kuadrant-system/authorino-59945dd659-9f5g7" Oct 30 04:55:25.231936 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:25.231892 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmckg\" (UniqueName: \"kubernetes.io/projected/a8adbcc0-729e-4774-af4f-43d362118031-kube-api-access-qmckg\") pod \"authorino-59945dd659-9f5g7\" (UID: \"a8adbcc0-729e-4774-af4f-43d362118031\") " pod="kuadrant-system/authorino-59945dd659-9f5g7" Oct 30 04:55:25.333259 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:25.333218 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmckg\" (UniqueName: \"kubernetes.io/projected/a8adbcc0-729e-4774-af4f-43d362118031-kube-api-access-qmckg\") pod \"authorino-59945dd659-9f5g7\" (UID: \"a8adbcc0-729e-4774-af4f-43d362118031\") " pod="kuadrant-system/authorino-59945dd659-9f5g7" Oct 30 04:55:25.333497 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:25.333289 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8adbcc0-729e-4774-af4f-43d362118031-tls-cert\") pod \"authorino-59945dd659-9f5g7\" (UID: \"a8adbcc0-729e-4774-af4f-43d362118031\") " pod="kuadrant-system/authorino-59945dd659-9f5g7" Oct 30 04:55:25.335711 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:25.335688 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8adbcc0-729e-4774-af4f-43d362118031-tls-cert\") pod \"authorino-59945dd659-9f5g7\" (UID: \"a8adbcc0-729e-4774-af4f-43d362118031\") " pod="kuadrant-system/authorino-59945dd659-9f5g7" Oct 30 04:55:25.362161 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:25.362131 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmckg\" (UniqueName: \"kubernetes.io/projected/a8adbcc0-729e-4774-af4f-43d362118031-kube-api-access-qmckg\") pod \"authorino-59945dd659-9f5g7\" (UID: \"a8adbcc0-729e-4774-af4f-43d362118031\") " pod="kuadrant-system/authorino-59945dd659-9f5g7" Oct 30 04:55:25.450306 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:25.450235 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59945dd659-9f5g7" Oct 30 04:55:25.740166 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:25.740138 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59945dd659-9f5g7"] Oct 30 04:55:25.740975 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:55:25.740935 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8adbcc0_729e_4774_af4f_43d362118031.slice/crio-55a9f818ebe6c176b1784b1693c13ed3b6c49b1b068d7c96c5e3d43f9d5841e9 WatchSource:0}: Error finding container 55a9f818ebe6c176b1784b1693c13ed3b6c49b1b068d7c96c5e3d43f9d5841e9: Status 404 returned error can't find the container with id 55a9f818ebe6c176b1784b1693c13ed3b6c49b1b068d7c96c5e3d43f9d5841e9 Oct 30 04:55:25.742429 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:25.742411 6587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:55:25.983987 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:55:25.983953 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:55:26.187982 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:26.187947 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59945dd659-9f5g7" event={"ID":"a8adbcc0-729e-4774-af4f-43d362118031","Type":"ContainerStarted","Data":"55a9f818ebe6c176b1784b1693c13ed3b6c49b1b068d7c96c5e3d43f9d5841e9"} Oct 30 04:55:27.196232 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.196196 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59945dd659-9f5g7" event={"ID":"a8adbcc0-729e-4774-af4f-43d362118031","Type":"ContainerStarted","Data":"98e2ce6041c458da06b4728d839241ecc7ad30666ff4e1767d8928eded5a8230"} Oct 30 04:55:27.269378 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.267985 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-59945dd659-9f5g7" podStartSLOduration=1.763168829 podStartE2EDuration="2.26796536s" totalImagesPullingTime="504.796531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:27.237903689 +0000 UTC m=+651.765670475" watchObservedRunningTime="2026-10-30 04:55:27.26796536 +0000 UTC m=+651.795732147" Oct 30 04:55:27.269378 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.268719 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b8b5dc7fb-bk2pm"] Oct 30 04:55:27.269378 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.269007 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" podUID="939a88f4-2a1c-4113-b431-8690e1cb4b63" containerName="authorino" containerID="cri-o://8d0bb06aa67f6d4b49f0ad2e640fa70ae6edf59f405aabab52a8e5a9db4cdaf1" gracePeriod=30 Oct 30 04:55:27.541012 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.540988 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" Oct 30 04:55:27.657525 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.657484 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/939a88f4-2a1c-4113-b431-8690e1cb4b63-tls-cert\" (UniqueName: \"kubernetes.io/secret/939a88f4-2a1c-4113-b431-8690e1cb4b63-tls-cert\") pod \"939a88f4-2a1c-4113-b431-8690e1cb4b63\" (UID: \"939a88f4-2a1c-4113-b431-8690e1cb4b63\") " Oct 30 04:55:27.657712 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.657590 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939a88f4-2a1c-4113-b431-8690e1cb4b63-kube-api-access-d9jdf\" (UniqueName: \"kubernetes.io/projected/939a88f4-2a1c-4113-b431-8690e1cb4b63-kube-api-access-d9jdf\") pod \"939a88f4-2a1c-4113-b431-8690e1cb4b63\" (UID: \"939a88f4-2a1c-4113-b431-8690e1cb4b63\") " Oct 30 04:55:27.660201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.660169 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939a88f4-2a1c-4113-b431-8690e1cb4b63-kube-api-access-d9jdf" pod "939a88f4-2a1c-4113-b431-8690e1cb4b63" (UID: "939a88f4-2a1c-4113-b431-8690e1cb4b63"). InnerVolumeSpecName "kube-api-access-d9jdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:55:27.669525 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.669496 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/939a88f4-2a1c-4113-b431-8690e1cb4b63-tls-cert" pod "939a88f4-2a1c-4113-b431-8690e1cb4b63" (UID: "939a88f4-2a1c-4113-b431-8690e1cb4b63"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:55:27.758562 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.758483 6587 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/939a88f4-2a1c-4113-b431-8690e1cb4b63-tls-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:55:27.758562 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.758520 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9jdf\" (UniqueName: \"kubernetes.io/projected/939a88f4-2a1c-4113-b431-8690e1cb4b63-kube-api-access-d9jdf\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:55:27.806564 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.806531 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6"] Oct 30 04:55:27.806915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.806901 6587 cpu_manager.go:414] "containerMap: removing container" podUID="939a88f4-2a1c-4113-b431-8690e1cb4b63" containerName="authorino" Oct 30 04:55:27.806959 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.806916 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="939a88f4-2a1c-4113-b431-8690e1cb4b63" containerName="authorino" Oct 30 04:55:27.806996 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.806987 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="939a88f4-2a1c-4113-b431-8690e1cb4b63" containerName="authorino" Oct 30 04:55:27.810085 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.810068 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" Oct 30 04:55:27.826696 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.826663 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-mxln4\"" Oct 30 04:55:27.828321 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.828296 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6"] Oct 30 04:55:27.829308 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.829289 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 04:55:27.829754 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.829734 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 04:55:27.873885 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.873849 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqctd\" (UniqueName: \"kubernetes.io/projected/66839335-97a5-4be3-a7e5-a10f18caac27-kube-api-access-rqctd\") pod \"ai-gateway-controller-5d9cd99f45-lwzf6\" (UID: \"66839335-97a5-4be3-a7e5-a10f18caac27\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" Oct 30 04:55:27.874086 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.873901 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66839335-97a5-4be3-a7e5-a10f18caac27-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-lwzf6\" (UID: \"66839335-97a5-4be3-a7e5-a10f18caac27\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" Oct 30 04:55:27.974628 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.974592 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqctd\" (UniqueName: \"kubernetes.io/projected/66839335-97a5-4be3-a7e5-a10f18caac27-kube-api-access-rqctd\") pod \"ai-gateway-controller-5d9cd99f45-lwzf6\" (UID: \"66839335-97a5-4be3-a7e5-a10f18caac27\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" Oct 30 04:55:27.974832 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.974639 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66839335-97a5-4be3-a7e5-a10f18caac27-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-lwzf6\" (UID: \"66839335-97a5-4be3-a7e5-a10f18caac27\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" Oct 30 04:55:27.977349 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.977319 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66839335-97a5-4be3-a7e5-a10f18caac27-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-lwzf6\" (UID: \"66839335-97a5-4be3-a7e5-a10f18caac27\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" Oct 30 04:55:27.996537 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:27.996505 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqctd\" (UniqueName: \"kubernetes.io/projected/66839335-97a5-4be3-a7e5-a10f18caac27-kube-api-access-rqctd\") pod \"ai-gateway-controller-5d9cd99f45-lwzf6\" (UID: \"66839335-97a5-4be3-a7e5-a10f18caac27\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" Oct 30 04:55:28.122790 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:28.122751 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" Oct 30 04:55:28.203002 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:28.202973 6587 generic.go:356] "Generic (PLEG): container finished" podID="939a88f4-2a1c-4113-b431-8690e1cb4b63" containerID="8d0bb06aa67f6d4b49f0ad2e640fa70ae6edf59f405aabab52a8e5a9db4cdaf1" exitCode=0 Oct 30 04:55:28.203540 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:28.203523 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" Oct 30 04:55:28.204982 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:28.204521 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" event={"ID":"939a88f4-2a1c-4113-b431-8690e1cb4b63","Type":"ContainerDied","Data":"8d0bb06aa67f6d4b49f0ad2e640fa70ae6edf59f405aabab52a8e5a9db4cdaf1"} Oct 30 04:55:28.204982 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:28.204568 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b8b5dc7fb-bk2pm" event={"ID":"939a88f4-2a1c-4113-b431-8690e1cb4b63","Type":"ContainerDied","Data":"84ca9a6a2988a59dff75de46d15450d950945cd2fbcd029fb598f3fac0f9c4ea"} Oct 30 04:55:28.204982 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:28.204707 6587 scope.go:122] "RemoveContainer" containerID="8d0bb06aa67f6d4b49f0ad2e640fa70ae6edf59f405aabab52a8e5a9db4cdaf1" Oct 30 04:55:28.215717 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:28.215698 6587 scope.go:122] "RemoveContainer" containerID="8d0bb06aa67f6d4b49f0ad2e640fa70ae6edf59f405aabab52a8e5a9db4cdaf1" Oct 30 04:55:28.216016 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:55:28.215996 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0bb06aa67f6d4b49f0ad2e640fa70ae6edf59f405aabab52a8e5a9db4cdaf1\": container with ID starting with 8d0bb06aa67f6d4b49f0ad2e640fa70ae6edf59f405aabab52a8e5a9db4cdaf1 not found: ID does not exist" containerID="8d0bb06aa67f6d4b49f0ad2e640fa70ae6edf59f405aabab52a8e5a9db4cdaf1" Oct 30 04:55:28.216111 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:28.216024 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0bb06aa67f6d4b49f0ad2e640fa70ae6edf59f405aabab52a8e5a9db4cdaf1"} err="failed to get container status \"8d0bb06aa67f6d4b49f0ad2e640fa70ae6edf59f405aabab52a8e5a9db4cdaf1\": rpc error: code = NotFound desc = could not find container \"8d0bb06aa67f6d4b49f0ad2e640fa70ae6edf59f405aabab52a8e5a9db4cdaf1\": container with ID starting with 8d0bb06aa67f6d4b49f0ad2e640fa70ae6edf59f405aabab52a8e5a9db4cdaf1 not found: ID does not exist" Oct 30 04:55:28.258040 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:28.258000 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b8b5dc7fb-bk2pm"] Oct 30 04:55:28.263231 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:28.263201 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6b8b5dc7fb-bk2pm"] Oct 30 04:55:28.426060 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:28.426033 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6"] Oct 30 04:55:28.428306 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:55:28.428273 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66839335_97a5_4be3_a7e5_a10f18caac27.slice/crio-33c08782f8ad73ad2140fcee38bab53c5fa025dd9fc762f51ec492f8c17b375c WatchSource:0}: Error finding container 33c08782f8ad73ad2140fcee38bab53c5fa025dd9fc762f51ec492f8c17b375c: Status 404 returned error can't find the container with id 33c08782f8ad73ad2140fcee38bab53c5fa025dd9fc762f51ec492f8c17b375c Oct 30 04:55:29.221442 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:29.221269 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" event={"ID":"66839335-97a5-4be3-a7e5-a10f18caac27","Type":"ContainerStarted","Data":"33c08782f8ad73ad2140fcee38bab53c5fa025dd9fc762f51ec492f8c17b375c"} Oct 30 04:55:29.581325 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:29.580970 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="939a88f4-2a1c-4113-b431-8690e1cb4b63" path="/var/lib/kubelet/pods/939a88f4-2a1c-4113-b431-8690e1cb4b63/volumes" Oct 30 04:55:32.237598 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:32.237567 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" event={"ID":"66839335-97a5-4be3-a7e5-a10f18caac27","Type":"ContainerStarted","Data":"5de1449b8491de185ebac06c08312e15571e5f50cd48e158613958e74fbbed5e"} Oct 30 04:55:32.237991 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:32.237686 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" Oct 30 04:55:32.278471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:32.278429 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" podStartSLOduration=2.531185002 podStartE2EDuration="5.278416333s" totalImagesPullingTime="2.747231331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:32.276993529 +0000 UTC m=+656.804760316" watchObservedRunningTime="2026-10-30 04:55:32.278416333 +0000 UTC m=+656.806183112" Oct 30 04:55:33.523486 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.523454 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-vqsmz"] Oct 30 04:55:33.527468 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.527449 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-vqsmz" Oct 30 04:55:33.554923 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.554897 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 04:55:33.556967 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.556942 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-vqsmz"] Oct 30 04:55:33.557919 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.557900 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fzxn4\"" Oct 30 04:55:33.571272 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.571246 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d"] Oct 30 04:55:33.574723 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.574697 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d" Oct 30 04:55:33.594678 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.594655 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d"] Oct 30 04:55:33.633225 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.633195 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlszp\" (UniqueName: \"kubernetes.io/projected/a8739f61-fe1e-4c58-b535-4815d55e05f1-kube-api-access-nlszp\") pod \"payload-processing-5f78687d5b-vqsmz\" (UID: \"a8739f61-fe1e-4c58-b535-4815d55e05f1\") " pod="openshift-ingress/payload-processing-5f78687d5b-vqsmz" Oct 30 04:55:33.633395 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.633312 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8739f61-fe1e-4c58-b535-4815d55e05f1-plugins-config-volume\") pod \"payload-processing-5f78687d5b-vqsmz\" (UID: \"a8739f61-fe1e-4c58-b535-4815d55e05f1\") " pod="openshift-ingress/payload-processing-5f78687d5b-vqsmz" Oct 30 04:55:33.734507 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.734469 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt59t\" (UniqueName: \"kubernetes.io/projected/d2ed078a-da76-4628-a77f-5b56163461d4-kube-api-access-lt59t\") pod \"payload-pre-processing-7b98bb678c-p6s8d\" (UID: \"d2ed078a-da76-4628-a77f-5b56163461d4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d" Oct 30 04:55:33.734703 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.734554 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlszp\" (UniqueName: \"kubernetes.io/projected/a8739f61-fe1e-4c58-b535-4815d55e05f1-kube-api-access-nlszp\") pod \"payload-processing-5f78687d5b-vqsmz\" (UID: \"a8739f61-fe1e-4c58-b535-4815d55e05f1\") " pod="openshift-ingress/payload-processing-5f78687d5b-vqsmz" Oct 30 04:55:33.734703 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.734601 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ed078a-da76-4628-a77f-5b56163461d4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-p6s8d\" (UID: \"d2ed078a-da76-4628-a77f-5b56163461d4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d" Oct 30 04:55:33.734851 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.734832 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8739f61-fe1e-4c58-b535-4815d55e05f1-plugins-config-volume\") pod \"payload-processing-5f78687d5b-vqsmz\" (UID: \"a8739f61-fe1e-4c58-b535-4815d55e05f1\") " pod="openshift-ingress/payload-processing-5f78687d5b-vqsmz" Oct 30 04:55:33.735647 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.735620 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8739f61-fe1e-4c58-b535-4815d55e05f1-plugins-config-volume\") pod \"payload-processing-5f78687d5b-vqsmz\" (UID: \"a8739f61-fe1e-4c58-b535-4815d55e05f1\") " pod="openshift-ingress/payload-processing-5f78687d5b-vqsmz" Oct 30 04:55:33.754833 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.754810 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlszp\" (UniqueName: \"kubernetes.io/projected/a8739f61-fe1e-4c58-b535-4815d55e05f1-kube-api-access-nlszp\") pod \"payload-processing-5f78687d5b-vqsmz\" (UID: \"a8739f61-fe1e-4c58-b535-4815d55e05f1\") " pod="openshift-ingress/payload-processing-5f78687d5b-vqsmz" Oct 30 04:55:33.835346 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.835309 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ed078a-da76-4628-a77f-5b56163461d4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-p6s8d\" (UID: \"d2ed078a-da76-4628-a77f-5b56163461d4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d" Oct 30 04:55:33.835543 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.835469 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt59t\" (UniqueName: \"kubernetes.io/projected/d2ed078a-da76-4628-a77f-5b56163461d4-kube-api-access-lt59t\") pod \"payload-pre-processing-7b98bb678c-p6s8d\" (UID: \"d2ed078a-da76-4628-a77f-5b56163461d4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d" Oct 30 04:55:33.836027 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.836004 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ed078a-da76-4628-a77f-5b56163461d4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-p6s8d\" (UID: \"d2ed078a-da76-4628-a77f-5b56163461d4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d" Oct 30 04:55:33.848244 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.848215 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-vqsmz" Oct 30 04:55:33.855683 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.855659 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt59t\" (UniqueName: \"kubernetes.io/projected/d2ed078a-da76-4628-a77f-5b56163461d4-kube-api-access-lt59t\") pod \"payload-pre-processing-7b98bb678c-p6s8d\" (UID: \"d2ed078a-da76-4628-a77f-5b56163461d4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d" Oct 30 04:55:33.888705 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:33.888666 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d" Oct 30 04:55:34.139823 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:34.139796 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-vqsmz"] Oct 30 04:55:34.140828 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:55:34.140804 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8739f61_fe1e_4c58_b535_4815d55e05f1.slice/crio-7a12193bff7368b0f2dea618ebaabd7e2b4df0f344e953e7f437d16eba1c8882 WatchSource:0}: Error finding container 7a12193bff7368b0f2dea618ebaabd7e2b4df0f344e953e7f437d16eba1c8882: Status 404 returned error can't find the container with id 7a12193bff7368b0f2dea618ebaabd7e2b4df0f344e953e7f437d16eba1c8882 Oct 30 04:55:34.207030 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:34.207004 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d"] Oct 30 04:55:34.207847 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:55:34.207816 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ed078a_da76_4628_a77f_5b56163461d4.slice/crio-bc84908a522458f80b27e2fa4b9b5b5ded3e26f10f01ee62168b5bd34126fa51 WatchSource:0}: Error finding container bc84908a522458f80b27e2fa4b9b5b5ded3e26f10f01ee62168b5bd34126fa51: Status 404 returned error can't find the container with id bc84908a522458f80b27e2fa4b9b5b5ded3e26f10f01ee62168b5bd34126fa51 Oct 30 04:55:34.247510 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:34.247477 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-vqsmz" event={"ID":"a8739f61-fe1e-4c58-b535-4815d55e05f1","Type":"ContainerStarted","Data":"7a12193bff7368b0f2dea618ebaabd7e2b4df0f344e953e7f437d16eba1c8882"} Oct 30 04:55:34.248616 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:34.248591 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d" event={"ID":"d2ed078a-da76-4628-a77f-5b56163461d4","Type":"ContainerStarted","Data":"bc84908a522458f80b27e2fa4b9b5b5ded3e26f10f01ee62168b5bd34126fa51"} Oct 30 04:55:37.270504 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:37.270458 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d" event={"ID":"d2ed078a-da76-4628-a77f-5b56163461d4","Type":"ContainerStarted","Data":"cbd3ae8aa97b1d09a63a307f32b45e7503b6273e57b31003e044b3c0a96594aa"} Oct 30 04:55:37.270926 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:37.270578 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d" Oct 30 04:55:37.272172 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:37.272138 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-vqsmz" event={"ID":"a8739f61-fe1e-4c58-b535-4815d55e05f1","Type":"ContainerStarted","Data":"f4f1d32926840a18cfd46319b136a509c3aa41baff7119450baf7da41092feec"} Oct 30 04:55:37.272302 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:37.272196 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-vqsmz" Oct 30 04:55:37.311108 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:37.311059 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d" podStartSLOduration=2.131896215 podStartE2EDuration="4.311045939s" totalImagesPullingTime="2.179149724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:37.310171167 +0000 UTC m=+661.837937955" watchObservedRunningTime="2026-10-30 04:55:37.311045939 +0000 UTC m=+661.838812724" Oct 30 04:55:37.348729 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:37.348679 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-vqsmz" podStartSLOduration=2.135126319 podStartE2EDuration="4.348665749s" totalImagesPullingTime="2.21353943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:37.348272479 +0000 UTC m=+661.876039265" watchObservedRunningTime="2026-10-30 04:55:37.348665749 +0000 UTC m=+661.876432536" Oct 30 04:55:40.846658 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:40.846623 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5"] Oct 30 04:55:40.852784 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:40.852762 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:40.868322 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:40.868299 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 04:55:40.881441 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:40.881415 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dr8cs\"" Oct 30 04:55:40.883301 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:40.883277 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5"] Oct 30 04:55:41.002825 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.002784 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.002825 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.002827 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdmkk\" (UniqueName: \"kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-kube-api-access-bdmkk\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.003083 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.002855 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f5e053-b8c8-4250-b07d-233a445dc717-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.003083 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.003014 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.003083 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.003078 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f5e053-b8c8-4250-b07d-233a445dc717-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.003212 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.003174 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.003267 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.003214 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.003319 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.003284 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.003404 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.003343 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.104070 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.103975 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.104070 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.104024 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f5e053-b8c8-4250-b07d-233a445dc717-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.104070 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.104057 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.104404 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.104090 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.104404 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.104114 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.104404 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.104268 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.104404 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.104344 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.104611 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.104408 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdmkk\" (UniqueName: \"kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-kube-api-access-bdmkk\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.104611 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.104447 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f5e053-b8c8-4250-b07d-233a445dc717-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.104611 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.104496 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.104611 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.104546 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.104611 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.104575 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.104870 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.104762 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.105109 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.105079 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f5e053-b8c8-4250-b07d-233a445dc717-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.106524 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.106497 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.106635 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.106622 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f5e053-b8c8-4250-b07d-233a445dc717-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.124173 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.124153 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdmkk\" (UniqueName: \"kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-kube-api-access-bdmkk\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.124284 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.124230 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-m84l5\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.166208 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.166179 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:41.470548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:41.470520 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5"] Oct 30 04:55:41.470717 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:55:41.470696 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f5e053_b8c8_4250_b07d_233a445dc717.slice/crio-313935acdafdc000f408a4e62bf132be485b6559621129509d39337dd22be346 WatchSource:0}: Error finding container 313935acdafdc000f408a4e62bf132be485b6559621129509d39337dd22be346: Status 404 returned error can't find the container with id 313935acdafdc000f408a4e62bf132be485b6559621129509d39337dd22be346 Oct 30 04:55:42.303891 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:42.303848 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" event={"ID":"19f5e053-b8c8-4250-b07d-233a445dc717","Type":"ContainerStarted","Data":"313935acdafdc000f408a4e62bf132be485b6559621129509d39337dd22be346"} Oct 30 04:55:43.244501 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:43.244468 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" Oct 30 04:55:43.521998 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:43.521919 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-77449b8677-8jpjr"] Oct 30 04:55:43.525845 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:43.525818 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-77449b8677-8jpjr" Oct 30 04:55:43.544656 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:43.544626 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-77449b8677-8jpjr"] Oct 30 04:55:43.629013 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:43.628974 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7d449b9-933c-4431-bbf5-5376c47eb048-metrics-certs\") pod \"ai-gateway-controller-77449b8677-8jpjr\" (UID: \"e7d449b9-933c-4431-bbf5-5376c47eb048\") " pod="opendatahub/ai-gateway-controller-77449b8677-8jpjr" Oct 30 04:55:43.629173 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:43.629115 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4wls\" (UniqueName: \"kubernetes.io/projected/e7d449b9-933c-4431-bbf5-5376c47eb048-kube-api-access-v4wls\") pod \"ai-gateway-controller-77449b8677-8jpjr\" (UID: \"e7d449b9-933c-4431-bbf5-5376c47eb048\") " pod="opendatahub/ai-gateway-controller-77449b8677-8jpjr" Oct 30 04:55:43.729845 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:43.729812 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4wls\" (UniqueName: \"kubernetes.io/projected/e7d449b9-933c-4431-bbf5-5376c47eb048-kube-api-access-v4wls\") pod \"ai-gateway-controller-77449b8677-8jpjr\" (UID: \"e7d449b9-933c-4431-bbf5-5376c47eb048\") " pod="opendatahub/ai-gateway-controller-77449b8677-8jpjr" Oct 30 04:55:43.730022 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:43.729938 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7d449b9-933c-4431-bbf5-5376c47eb048-metrics-certs\") pod \"ai-gateway-controller-77449b8677-8jpjr\" (UID: \"e7d449b9-933c-4431-bbf5-5376c47eb048\") " pod="opendatahub/ai-gateway-controller-77449b8677-8jpjr" Oct 30 04:55:43.732773 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:43.732743 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7d449b9-933c-4431-bbf5-5376c47eb048-metrics-certs\") pod \"ai-gateway-controller-77449b8677-8jpjr\" (UID: \"e7d449b9-933c-4431-bbf5-5376c47eb048\") " pod="opendatahub/ai-gateway-controller-77449b8677-8jpjr" Oct 30 04:55:43.750126 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:43.750099 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4wls\" (UniqueName: \"kubernetes.io/projected/e7d449b9-933c-4431-bbf5-5376c47eb048-kube-api-access-v4wls\") pod \"ai-gateway-controller-77449b8677-8jpjr\" (UID: \"e7d449b9-933c-4431-bbf5-5376c47eb048\") " pod="opendatahub/ai-gateway-controller-77449b8677-8jpjr" Oct 30 04:55:43.842144 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:43.842107 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-77449b8677-8jpjr" Oct 30 04:55:44.150126 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:44.150102 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-77449b8677-8jpjr"] Oct 30 04:55:44.314626 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:44.314589 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-77449b8677-8jpjr" event={"ID":"e7d449b9-933c-4431-bbf5-5376c47eb048","Type":"ContainerStarted","Data":"18afb10637eaa9a033faa176c6a69147cdffca6bc82d6c21109f2d4011724450"} Oct 30 04:55:44.848597 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:44.848571 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:55:44.848887 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:44.848634 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:55:44.848887 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:44.848663 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:55:45.344265 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:45.344213 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" event={"ID":"19f5e053-b8c8-4250-b07d-233a445dc717","Type":"ContainerStarted","Data":"b43fbc9187a2b82bf1da4e57fe3852a5d1a49de62e728a898dde9152c14376ae"} Oct 30 04:55:45.346873 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:45.346845 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-77449b8677-8jpjr" event={"ID":"e7d449b9-933c-4431-bbf5-5376c47eb048","Type":"ContainerStarted","Data":"fbd63b115a192707c5e77dfdfb35cd950aa4c0e4c4ddbb658deac633aa5faba6"} Oct 30 04:55:45.347035 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:45.346982 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-77449b8677-8jpjr" Oct 30 04:55:45.401277 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:45.401228 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" podStartSLOduration=2.026125414 podStartE2EDuration="5.40121361s" totalImagesPullingTime="3.375088196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:45.399591114 +0000 UTC m=+669.927357896" watchObservedRunningTime="2026-10-30 04:55:45.40121361 +0000 UTC m=+669.928980396" Oct 30 04:55:45.449057 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:45.449000 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-77449b8677-8jpjr" podStartSLOduration=2.124640665 podStartE2EDuration="2.448986343s" totalImagesPullingTime="324.345678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:45.446672626 +0000 UTC m=+669.974439414" watchObservedRunningTime="2026-10-30 04:55:45.448986343 +0000 UTC m=+669.976753186" Oct 30 04:55:46.167346 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:46.167316 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:46.168889 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:46.168856 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 04:55:46.169036 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:46.168919 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" podUID="19f5e053-b8c8-4250-b07d-233a445dc717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 04:55:47.167042 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:47.167006 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 04:55:47.167240 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:47.167073 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" podUID="19f5e053-b8c8-4250-b07d-233a445dc717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 04:55:48.166920 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:48.166887 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 04:55:48.167310 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:48.166939 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" podUID="19f5e053-b8c8-4250-b07d-233a445dc717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 04:55:48.280438 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:48.280410 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d" Oct 30 04:55:48.280633 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:48.280488 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-vqsmz" Oct 30 04:55:49.167184 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:49.167148 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 04:55:49.167687 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:49.167201 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" podUID="19f5e053-b8c8-4250-b07d-233a445dc717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 04:55:50.166655 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:50.166620 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 04:55:50.166856 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:50.166681 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" podUID="19f5e053-b8c8-4250-b07d-233a445dc717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 04:55:51.167005 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:51.166967 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 04:55:51.167415 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:51.167035 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" podUID="19f5e053-b8c8-4250-b07d-233a445dc717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 04:55:52.167016 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:52.166982 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 04:55:52.167392 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:52.167044 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" podUID="19f5e053-b8c8-4250-b07d-233a445dc717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 04:55:53.170722 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:53.170695 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:53.171101 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:53.170928 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:53.171629 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:53.171611 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 04:55:55.909018 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:55.908985 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt"] Oct 30 04:55:55.912561 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:55.912543 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:55.942960 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:55.942932 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt"] Oct 30 04:55:55.979606 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:55.979576 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:55.979773 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:55.979621 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:55.979773 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:55.979691 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:55.979860 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:55.979826 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:55.979905 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:55.979861 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:55.979905 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:55.979885 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:55.980021 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:55.980002 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:55.980064 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:55.980036 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:55.980117 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:55.980063 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghf24\" (UniqueName: \"kubernetes.io/projected/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-kube-api-access-ghf24\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:55.984169 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:55:55.984140 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:55:56.081046 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081012 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.081210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081094 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.081210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081128 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.081210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081166 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.081210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081204 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.081465 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081237 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.081465 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081264 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghf24\" (UniqueName: \"kubernetes.io/projected/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-kube-api-access-ghf24\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.081465 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081389 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.081465 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081449 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.081680 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081491 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.081680 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081568 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.081761 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081733 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.081884 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081866 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.081937 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.081921 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.083321 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.083305 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.083696 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.083680 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.100967 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.100945 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.101130 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.101111 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghf24\" (UniqueName: \"kubernetes.io/projected/ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd-kube-api-access-ghf24\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-52stt\" (UID: \"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.226430 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.226320 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:56.353456 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.353425 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-77449b8677-8jpjr" Oct 30 04:55:56.417506 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.417471 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6"] Oct 30 04:55:56.417778 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.417751 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" podUID="66839335-97a5-4be3-a7e5-a10f18caac27" containerName="manager" containerID="cri-o://5de1449b8491de185ebac06c08312e15571e5f50cd48e158613958e74fbbed5e" gracePeriod=30 Oct 30 04:55:56.529088 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.529063 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt"] Oct 30 04:55:56.529980 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:55:56.529958 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8662ab_ab3e_4271_b8a9_0a0ee4ff6bdd.slice/crio-39de4386635b26669e2c6fb78fcb22d6ccfc32dcf260ae0b8d907f3f44a2ebf5 WatchSource:0}: Error finding container 39de4386635b26669e2c6fb78fcb22d6ccfc32dcf260ae0b8d907f3f44a2ebf5: Status 404 returned error can't find the container with id 39de4386635b26669e2c6fb78fcb22d6ccfc32dcf260ae0b8d907f3f44a2ebf5 Oct 30 04:55:56.533061 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.533028 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:55:56.533157 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.533108 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:55:56.533157 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.533144 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:55:56.660879 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.660850 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" Oct 30 04:55:56.790567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.790494 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66839335-97a5-4be3-a7e5-a10f18caac27-kube-api-access-rqctd\" (UniqueName: \"kubernetes.io/projected/66839335-97a5-4be3-a7e5-a10f18caac27-kube-api-access-rqctd\") pod \"66839335-97a5-4be3-a7e5-a10f18caac27\" (UID: \"66839335-97a5-4be3-a7e5-a10f18caac27\") " Oct 30 04:55:56.790567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.790561 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66839335-97a5-4be3-a7e5-a10f18caac27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66839335-97a5-4be3-a7e5-a10f18caac27-metrics-certs\") pod \"66839335-97a5-4be3-a7e5-a10f18caac27\" (UID: \"66839335-97a5-4be3-a7e5-a10f18caac27\") " Oct 30 04:55:56.792729 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.792700 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66839335-97a5-4be3-a7e5-a10f18caac27-kube-api-access-rqctd" pod "66839335-97a5-4be3-a7e5-a10f18caac27" (UID: "66839335-97a5-4be3-a7e5-a10f18caac27"). InnerVolumeSpecName "kube-api-access-rqctd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:55:56.793111 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.793084 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66839335-97a5-4be3-a7e5-a10f18caac27-metrics-certs" pod "66839335-97a5-4be3-a7e5-a10f18caac27" (UID: "66839335-97a5-4be3-a7e5-a10f18caac27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:55:56.891701 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.891662 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqctd\" (UniqueName: \"kubernetes.io/projected/66839335-97a5-4be3-a7e5-a10f18caac27-kube-api-access-rqctd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:55:56.891701 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:56.891692 6587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66839335-97a5-4be3-a7e5-a10f18caac27-metrics-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:55:57.442878 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:57.442790 6587 generic.go:356] "Generic (PLEG): container finished" podID="66839335-97a5-4be3-a7e5-a10f18caac27" containerID="5de1449b8491de185ebac06c08312e15571e5f50cd48e158613958e74fbbed5e" exitCode=0 Oct 30 04:55:57.442878 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:57.442840 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" Oct 30 04:55:57.443427 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:57.442872 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" event={"ID":"66839335-97a5-4be3-a7e5-a10f18caac27","Type":"ContainerDied","Data":"5de1449b8491de185ebac06c08312e15571e5f50cd48e158613958e74fbbed5e"} Oct 30 04:55:57.443427 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:57.442946 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6" event={"ID":"66839335-97a5-4be3-a7e5-a10f18caac27","Type":"ContainerDied","Data":"33c08782f8ad73ad2140fcee38bab53c5fa025dd9fc762f51ec492f8c17b375c"} Oct 30 04:55:57.443427 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:57.442980 6587 scope.go:122] "RemoveContainer" containerID="5de1449b8491de185ebac06c08312e15571e5f50cd48e158613958e74fbbed5e" Oct 30 04:55:57.445091 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:57.445070 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" event={"ID":"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd","Type":"ContainerStarted","Data":"18da5c186d19ab670efab820b36b889374911bd7c31dcf35b31217acdd27d759"} Oct 30 04:55:57.445205 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:57.445108 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" event={"ID":"ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd","Type":"ContainerStarted","Data":"39de4386635b26669e2c6fb78fcb22d6ccfc32dcf260ae0b8d907f3f44a2ebf5"} Oct 30 04:55:57.452191 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:57.452174 6587 scope.go:122] "RemoveContainer" containerID="5de1449b8491de185ebac06c08312e15571e5f50cd48e158613958e74fbbed5e" Oct 30 04:55:57.452469 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:55:57.452451 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de1449b8491de185ebac06c08312e15571e5f50cd48e158613958e74fbbed5e\": container with ID starting with 5de1449b8491de185ebac06c08312e15571e5f50cd48e158613958e74fbbed5e not found: ID does not exist" containerID="5de1449b8491de185ebac06c08312e15571e5f50cd48e158613958e74fbbed5e" Oct 30 04:55:57.452545 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:57.452477 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de1449b8491de185ebac06c08312e15571e5f50cd48e158613958e74fbbed5e"} err="failed to get container status \"5de1449b8491de185ebac06c08312e15571e5f50cd48e158613958e74fbbed5e\": rpc error: code = NotFound desc = could not find container \"5de1449b8491de185ebac06c08312e15571e5f50cd48e158613958e74fbbed5e\": container with ID starting with 5de1449b8491de185ebac06c08312e15571e5f50cd48e158613958e74fbbed5e not found: ID does not exist" Oct 30 04:55:57.495330 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:57.495304 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6"] Oct 30 04:55:57.498732 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:57.498709 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-lwzf6"] Oct 30 04:55:57.568660 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:57.568598 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" podStartSLOduration=2.568581192 podStartE2EDuration="2.568581192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:57.567591932 +0000 UTC m=+682.095358717" watchObservedRunningTime="2026-10-30 04:55:57.568581192 +0000 UTC m=+682.096347979" Oct 30 04:55:57.579623 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:57.579595 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66839335-97a5-4be3-a7e5-a10f18caac27" path="/var/lib/kubelet/pods/66839335-97a5-4be3-a7e5-a10f18caac27/volumes" Oct 30 04:55:58.240384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:58.232699 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:55:58.240384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:58.236987 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 04:55:58.240384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:58.237045 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" podUID="ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 04:55:59.234754 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:59.234718 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 04:55:59.235337 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:55:59.235303 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" podUID="ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 04:56:00.236822 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:00.236782 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 04:56:00.237490 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:00.237455 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" podUID="ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 04:56:01.240735 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:01.240693 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 04:56:01.241157 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:01.240753 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" podUID="ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 04:56:02.234384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:02.233738 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 04:56:02.234384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:02.233799 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" podUID="ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 04:56:03.227522 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:03.227487 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 04:56:03.227945 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:03.227540 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" podUID="ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 04:56:04.848152 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:04.848119 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:56:04.848755 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:04.848408 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:56:04.849474 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:04.849448 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" Oct 30 04:56:25.984672 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:56:25.984637 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:56:35.992049 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:35.991974 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j"] Oct 30 04:56:35.992498 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:35.992382 6587 cpu_manager.go:414] "containerMap: removing container" podUID="66839335-97a5-4be3-a7e5-a10f18caac27" containerName="manager" Oct 30 04:56:35.992498 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:35.992397 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66839335-97a5-4be3-a7e5-a10f18caac27" containerName="manager" Oct 30 04:56:35.992568 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:35.992507 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="66839335-97a5-4be3-a7e5-a10f18caac27" containerName="manager" Oct 30 04:56:35.997260 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:35.997234 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.023166 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.023131 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j"] Oct 30 04:56:36.050918 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.050886 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-83e83e-openshift-default-dockercfg-zgpn5\"" Oct 30 04:56:36.080496 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.080450 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.080640 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.080524 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.080640 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.080575 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df4067a-ac1c-496d-83d1-470dd606212d-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.080717 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.080702 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rd84\" (UniqueName: \"kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-kube-api-access-7rd84\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.080752 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.080741 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.080784 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.080771 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.080824 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.080811 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df4067a-ac1c-496d-83d1-470dd606212d-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.080863 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.080838 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.080896 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.080871 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.181538 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.181496 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.181716 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.181560 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df4067a-ac1c-496d-83d1-470dd606212d-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.181716 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.181593 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.181716 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.181629 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.181716 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.181688 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.181956 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.181753 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.181956 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.181776 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df4067a-ac1c-496d-83d1-470dd606212d-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.181956 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.181824 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rd84\" (UniqueName: \"kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-kube-api-access-7rd84\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.181956 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.181867 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.182609 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.182569 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.182857 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.182834 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.183473 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.183450 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.183547 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.183510 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df4067a-ac1c-496d-83d1-470dd606212d-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.183783 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.183763 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.184631 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.184608 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.186179 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.186156 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df4067a-ac1c-496d-83d1-470dd606212d-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.202426 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.202405 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.202885 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.202869 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rd84\" (UniqueName: \"kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-kube-api-access-7rd84\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.317455 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.317353 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:36.395532 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.393431 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c"] Oct 30 04:56:36.398850 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.398826 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.416655 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.416614 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-cff4ac-openshift-default-dockercfg-xcpvh\"" Oct 30 04:56:36.419426 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.419402 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c"] Oct 30 04:56:36.484795 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.484758 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.484957 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.484807 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.484957 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.484836 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.484957 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.484942 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.485080 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.485002 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.485080 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.485060 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.485190 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.485176 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.485245 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.485221 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkmsk\" (UniqueName: \"kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-kube-api-access-wkmsk\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.485312 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.485296 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a110c9-725e-4466-b182-1f608f0d0ba0-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586062 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.585982 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586062 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.586037 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586062 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.586059 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkmsk\" (UniqueName: \"kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-kube-api-access-wkmsk\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586332 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.586079 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a110c9-725e-4466-b182-1f608f0d0ba0-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586332 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.586109 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586332 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.586165 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586332 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.586200 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586332 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.586251 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586332 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.586274 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586658 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.586451 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.586704 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586779 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.586734 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586837 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.586811 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.586958 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.586930 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a110c9-725e-4466-b182-1f608f0d0ba0-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.588560 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.588538 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.588678 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.588661 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.606224 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.606193 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.606353 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.606299 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkmsk\" (UniqueName: \"kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-kube-api-access-wkmsk\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.628839 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.628813 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j"] Oct 30 04:56:36.629600 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:36.629576 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df4067a_ac1c_496d_83d1_470dd606212d.slice/crio-3eac9c59af28aeddaa59374fe0c9dc35d864c6d8f585e1fc3ec4fc550c62cffe WatchSource:0}: Error finding container 3eac9c59af28aeddaa59374fe0c9dc35d864c6d8f585e1fc3ec4fc550c62cffe: Status 404 returned error can't find the container with id 3eac9c59af28aeddaa59374fe0c9dc35d864c6d8f585e1fc3ec4fc550c62cffe Oct 30 04:56:36.631879 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.631850 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:36.631950 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.631909 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:36.631987 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.631954 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:36.678455 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.678427 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78"] Oct 30 04:56:36.685006 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.684981 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.704656 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.703575 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d55037-openshift-default-dockercfg-ppf9l\"" Oct 30 04:56:36.706166 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.706138 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78"] Oct 30 04:56:36.714026 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.714003 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:36.788291 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.788259 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.788444 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.788297 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.788444 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.788342 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.788444 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.788382 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff505e8-bee4-4f87-93e7-113497e13649-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.788444 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.788412 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff505e8-bee4-4f87-93e7-113497e13649-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.788444 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.788432 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.788685 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.788462 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.788685 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.788487 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.788685 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.788511 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zt4r\" (UniqueName: \"kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-kube-api-access-7zt4r\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.889460 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.889377 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.889460 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.889422 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff505e8-bee4-4f87-93e7-113497e13649-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.889460 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.889449 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff505e8-bee4-4f87-93e7-113497e13649-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.889790 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.889471 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.889790 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.889511 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.889790 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.889535 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.889790 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.889565 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zt4r\" (UniqueName: \"kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-kube-api-access-7zt4r\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.889790 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.889620 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.889790 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.889646 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.890132 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.889784 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.890132 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.889976 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.890132 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.890031 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.890288 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.890225 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.890288 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.890229 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff505e8-bee4-4f87-93e7-113497e13649-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.892116 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.892089 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.892263 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.892240 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff505e8-bee4-4f87-93e7-113497e13649-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.909212 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.909185 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:36.909313 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:36.909244 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zt4r\" (UniqueName: \"kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-kube-api-access-7zt4r\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:37.013895 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.013835 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:37.028755 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.028553 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c"] Oct 30 04:56:37.029993 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:37.029961 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a110c9_725e_4466_b182_1f608f0d0ba0.slice/crio-3aacee0c1a3108f94b8b0d9df9d0139718de73f548eb2fb94e1642c0680d6315 WatchSource:0}: Error finding container 3aacee0c1a3108f94b8b0d9df9d0139718de73f548eb2fb94e1642c0680d6315: Status 404 returned error can't find the container with id 3aacee0c1a3108f94b8b0d9df9d0139718de73f548eb2fb94e1642c0680d6315 Oct 30 04:56:37.032606 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.032572 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:37.032698 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.032668 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:37.032740 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.032715 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:37.325888 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.325858 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78"] Oct 30 04:56:37.326797 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:37.326755 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff505e8_bee4_4f87_93e7_113497e13649.slice/crio-c6412295bf5d5425f6bf2be7f54dd1623be0c1f8b06eda094a4e572ea994e302 WatchSource:0}: Error finding container c6412295bf5d5425f6bf2be7f54dd1623be0c1f8b06eda094a4e572ea994e302: Status 404 returned error can't find the container with id c6412295bf5d5425f6bf2be7f54dd1623be0c1f8b06eda094a4e572ea994e302 Oct 30 04:56:37.329252 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.329221 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:37.329341 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.329289 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:37.329341 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.329320 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:56:37.628834 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.628739 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" event={"ID":"a3a110c9-725e-4466-b182-1f608f0d0ba0","Type":"ContainerStarted","Data":"3d037236381e06aefd12320ec64df2c9a5de69c2703e82cf9f9d1de25efc776f"} Oct 30 04:56:37.628834 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.628805 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" event={"ID":"a3a110c9-725e-4466-b182-1f608f0d0ba0","Type":"ContainerStarted","Data":"3aacee0c1a3108f94b8b0d9df9d0139718de73f548eb2fb94e1642c0680d6315"} Oct 30 04:56:37.630656 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.630626 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" event={"ID":"6ff505e8-bee4-4f87-93e7-113497e13649","Type":"ContainerStarted","Data":"93a935b2fdbd1e32c26a7cf0c6d6b6e26ff57dfe72f43e7aba3fcfa9474d928f"} Oct 30 04:56:37.630790 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.630692 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" event={"ID":"6ff505e8-bee4-4f87-93e7-113497e13649","Type":"ContainerStarted","Data":"c6412295bf5d5425f6bf2be7f54dd1623be0c1f8b06eda094a4e572ea994e302"} Oct 30 04:56:37.632411 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.632383 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" event={"ID":"6df4067a-ac1c-496d-83d1-470dd606212d","Type":"ContainerStarted","Data":"3b6ecc9a07c19efada4e684d89ea5f4d5e2f37d9dbd7e4e2804f4542c7b55b20"} Oct 30 04:56:37.632514 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.632433 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" event={"ID":"6df4067a-ac1c-496d-83d1-470dd606212d","Type":"ContainerStarted","Data":"3eac9c59af28aeddaa59374fe0c9dc35d864c6d8f585e1fc3ec4fc550c62cffe"} Oct 30 04:56:37.672239 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.672186 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" podStartSLOduration=1.672173526 podStartE2EDuration="1.672173526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:37.671677619 +0000 UTC m=+722.199444405" watchObservedRunningTime="2026-10-30 04:56:37.672173526 +0000 UTC m=+722.199940311" Oct 30 04:56:37.713285 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.713239 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" podStartSLOduration=2.713222137 podStartE2EDuration="2.713222137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:37.712120676 +0000 UTC m=+722.239887463" watchObservedRunningTime="2026-10-30 04:56:37.713222137 +0000 UTC m=+722.240988926" Oct 30 04:56:37.714874 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.714853 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:37.754236 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:37.754193 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" podStartSLOduration=1.754176065 podStartE2EDuration="1.754176065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:37.753259441 +0000 UTC m=+722.281026234" watchObservedRunningTime="2026-10-30 04:56:37.754176065 +0000 UTC m=+722.281942851" Oct 30 04:56:38.014669 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:38.014518 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:38.045018 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:38.044989 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:38.318505 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:38.318465 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:38.324611 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:38.324583 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:38.637171 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:38.637069 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:38.637450 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:38.637427 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:38.638086 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:38.638068 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:56:38.638185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:38.638119 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:56:38.719798 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:38.719769 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:39.643159 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:39.643105 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:39.644002 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:39.643983 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:56:41.418895 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.418861 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72"] Oct 30 04:56:41.422908 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.422889 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" Oct 30 04:56:41.463380 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.462938 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-397179\"" Oct 30 04:56:41.463380 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.463013 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-397179-dockercfg-4kz5g\"" Oct 30 04:56:41.472081 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.468750 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72"] Oct 30 04:56:41.476649 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.475959 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4"] Oct 30 04:56:41.480368 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.480336 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" Oct 30 04:56:41.504670 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.504641 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4"] Oct 30 04:56:41.542071 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.542045 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461cc202-2197-4c68-b5a7-734fb8e27072-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72\" (UID: \"461cc202-2197-4c68-b5a7-734fb8e27072\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" Oct 30 04:56:41.542229 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.542082 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4lxq\" (UniqueName: \"kubernetes.io/projected/461cc202-2197-4c68-b5a7-734fb8e27072-kube-api-access-j4lxq\") pod \"payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72\" (UID: \"461cc202-2197-4c68-b5a7-734fb8e27072\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" Oct 30 04:56:41.642630 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.642596 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461cc202-2197-4c68-b5a7-734fb8e27072-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72\" (UID: \"461cc202-2197-4c68-b5a7-734fb8e27072\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" Oct 30 04:56:41.642824 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.642652 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lxq\" (UniqueName: \"kubernetes.io/projected/461cc202-2197-4c68-b5a7-734fb8e27072-kube-api-access-j4lxq\") pod \"payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72\" (UID: \"461cc202-2197-4c68-b5a7-734fb8e27072\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" Oct 30 04:56:41.642824 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.642735 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6dm2\" (UniqueName: \"kubernetes.io/projected/34ed4f94-1796-44b1-9de3-60065c919d4a-kube-api-access-c6dm2\") pod \"payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4\" (UID: \"34ed4f94-1796-44b1-9de3-60065c919d4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" Oct 30 04:56:41.642824 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.642775 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ed4f94-1796-44b1-9de3-60065c919d4a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4\" (UID: \"34ed4f94-1796-44b1-9de3-60065c919d4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" Oct 30 04:56:41.643305 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.643285 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461cc202-2197-4c68-b5a7-734fb8e27072-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72\" (UID: \"461cc202-2197-4c68-b5a7-734fb8e27072\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" Oct 30 04:56:41.665433 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.665407 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lxq\" (UniqueName: \"kubernetes.io/projected/461cc202-2197-4c68-b5a7-734fb8e27072-kube-api-access-j4lxq\") pod \"payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72\" (UID: \"461cc202-2197-4c68-b5a7-734fb8e27072\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" Oct 30 04:56:41.736239 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.736164 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" Oct 30 04:56:41.743997 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.743759 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6dm2\" (UniqueName: \"kubernetes.io/projected/34ed4f94-1796-44b1-9de3-60065c919d4a-kube-api-access-c6dm2\") pod \"payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4\" (UID: \"34ed4f94-1796-44b1-9de3-60065c919d4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" Oct 30 04:56:41.743997 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.743816 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ed4f94-1796-44b1-9de3-60065c919d4a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4\" (UID: \"34ed4f94-1796-44b1-9de3-60065c919d4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" Oct 30 04:56:41.746711 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.746685 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ed4f94-1796-44b1-9de3-60065c919d4a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4\" (UID: \"34ed4f94-1796-44b1-9de3-60065c919d4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" Oct 30 04:56:41.765141 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.765116 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6dm2\" (UniqueName: \"kubernetes.io/projected/34ed4f94-1796-44b1-9de3-60065c919d4a-kube-api-access-c6dm2\") pod \"payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4\" (UID: \"34ed4f94-1796-44b1-9de3-60065c919d4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" Oct 30 04:56:41.797598 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:41.797570 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" Oct 30 04:56:42.039244 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:42.039218 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72"] Oct 30 04:56:42.040732 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:42.040698 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461cc202_2197_4c68_b5a7_734fb8e27072.slice/crio-93195b9f484da9f66b32d5daa12119089cc479c8d45021607edbea6d1a14b9fc WatchSource:0}: Error finding container 93195b9f484da9f66b32d5daa12119089cc479c8d45021607edbea6d1a14b9fc: Status 404 returned error can't find the container with id 93195b9f484da9f66b32d5daa12119089cc479c8d45021607edbea6d1a14b9fc Oct 30 04:56:42.105671 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:42.105640 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ed4f94_1796_44b1_9de3_60065c919d4a.slice/crio-e1b36643785a3faea00c6ebe38976f0f661a6c161b69ddae56b5951b9a997b59 WatchSource:0}: Error finding container e1b36643785a3faea00c6ebe38976f0f661a6c161b69ddae56b5951b9a997b59: Status 404 returned error can't find the container with id e1b36643785a3faea00c6ebe38976f0f661a6c161b69ddae56b5951b9a997b59 Oct 30 04:56:42.105964 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:42.105944 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4"] Oct 30 04:56:42.659377 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:42.659337 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" event={"ID":"34ed4f94-1796-44b1-9de3-60065c919d4a","Type":"ContainerStarted","Data":"50603cc7d711a8eea11b0e1bad9f9eb929fe6e45d78bce9804f6cd44dfd912ac"} Oct 30 04:56:42.659840 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:42.659399 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" event={"ID":"34ed4f94-1796-44b1-9de3-60065c919d4a","Type":"ContainerStarted","Data":"e1b36643785a3faea00c6ebe38976f0f661a6c161b69ddae56b5951b9a997b59"} Oct 30 04:56:42.659840 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:42.659429 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" Oct 30 04:56:42.660766 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:42.660741 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" event={"ID":"461cc202-2197-4c68-b5a7-734fb8e27072","Type":"ContainerStarted","Data":"1f9a94e78ff8d0e786adf74d5d0cae559b687bfd6b3acc8c444a7688b9b8dcbb"} Oct 30 04:56:42.660848 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:42.660785 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" event={"ID":"461cc202-2197-4c68-b5a7-734fb8e27072","Type":"ContainerStarted","Data":"93195b9f484da9f66b32d5daa12119089cc479c8d45021607edbea6d1a14b9fc"} Oct 30 04:56:42.660848 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:42.660817 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" Oct 30 04:56:42.700686 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:42.700642 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" podStartSLOduration=1.700628432 podStartE2EDuration="1.700628432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:42.699072294 +0000 UTC m=+727.226839080" watchObservedRunningTime="2026-10-30 04:56:42.700628432 +0000 UTC m=+727.228395218" Oct 30 04:56:42.738428 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:42.738349 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" podStartSLOduration=1.738332884 podStartE2EDuration="1.738332884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:42.737496754 +0000 UTC m=+727.265263540" watchObservedRunningTime="2026-10-30 04:56:42.738332884 +0000 UTC m=+727.266099671" Oct 30 04:56:43.342518 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.342482 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj"] Oct 30 04:56:43.346283 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.346264 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" Oct 30 04:56:43.369348 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.369325 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-889ca9\"" Oct 30 04:56:43.397599 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.397570 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj"] Oct 30 04:56:43.399739 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.399719 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk"] Oct 30 04:56:43.403449 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.403430 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" Oct 30 04:56:43.424815 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.424559 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-889ca9-dockercfg-fc6nn\"" Oct 30 04:56:43.426661 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.426640 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk"] Oct 30 04:56:43.463489 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.463462 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d591ba5b-cdc7-4f23-85c6-28418b0abefe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj\" (UID: \"d591ba5b-cdc7-4f23-85c6-28418b0abefe\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" Oct 30 04:56:43.463601 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.463580 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6qs\" (UniqueName: \"kubernetes.io/projected/d591ba5b-cdc7-4f23-85c6-28418b0abefe-kube-api-access-vc6qs\") pod \"payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj\" (UID: \"d591ba5b-cdc7-4f23-85c6-28418b0abefe\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" Oct 30 04:56:43.564290 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.564255 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d591ba5b-cdc7-4f23-85c6-28418b0abefe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj\" (UID: \"d591ba5b-cdc7-4f23-85c6-28418b0abefe\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" Oct 30 04:56:43.564504 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.564296 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbmvk\" (UniqueName: \"kubernetes.io/projected/46d11379-6d13-4b4b-a36b-4d418b490323-kube-api-access-pbmvk\") pod \"payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk\" (UID: \"46d11379-6d13-4b4b-a36b-4d418b490323\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" Oct 30 04:56:43.564504 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.564346 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6qs\" (UniqueName: \"kubernetes.io/projected/d591ba5b-cdc7-4f23-85c6-28418b0abefe-kube-api-access-vc6qs\") pod \"payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj\" (UID: \"d591ba5b-cdc7-4f23-85c6-28418b0abefe\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" Oct 30 04:56:43.564504 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.564413 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d11379-6d13-4b4b-a36b-4d418b490323-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk\" (UID: \"46d11379-6d13-4b4b-a36b-4d418b490323\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" Oct 30 04:56:43.565063 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.565024 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d591ba5b-cdc7-4f23-85c6-28418b0abefe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj\" (UID: \"d591ba5b-cdc7-4f23-85c6-28418b0abefe\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" Oct 30 04:56:43.584497 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.584473 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6qs\" (UniqueName: \"kubernetes.io/projected/d591ba5b-cdc7-4f23-85c6-28418b0abefe-kube-api-access-vc6qs\") pod \"payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj\" (UID: \"d591ba5b-cdc7-4f23-85c6-28418b0abefe\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" Oct 30 04:56:43.660019 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.659946 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" Oct 30 04:56:43.665582 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.665550 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbmvk\" (UniqueName: \"kubernetes.io/projected/46d11379-6d13-4b4b-a36b-4d418b490323-kube-api-access-pbmvk\") pod \"payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk\" (UID: \"46d11379-6d13-4b4b-a36b-4d418b490323\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" Oct 30 04:56:43.665761 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.665742 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d11379-6d13-4b4b-a36b-4d418b490323-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk\" (UID: \"46d11379-6d13-4b4b-a36b-4d418b490323\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" Oct 30 04:56:43.666497 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.666476 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d11379-6d13-4b4b-a36b-4d418b490323-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk\" (UID: \"46d11379-6d13-4b4b-a36b-4d418b490323\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" Oct 30 04:56:43.685644 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.685618 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbmvk\" (UniqueName: \"kubernetes.io/projected/46d11379-6d13-4b4b-a36b-4d418b490323-kube-api-access-pbmvk\") pod \"payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk\" (UID: \"46d11379-6d13-4b4b-a36b-4d418b490323\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" Oct 30 04:56:43.716641 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.716544 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" Oct 30 04:56:43.950896 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:43.950866 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj"] Oct 30 04:56:43.951574 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:43.951551 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd591ba5b_cdc7_4f23_85c6_28418b0abefe.slice/crio-ae1cd1abd3bef7971cec0101eee4479ac00e7e106a813fbf36cf92153553eb2a WatchSource:0}: Error finding container ae1cd1abd3bef7971cec0101eee4479ac00e7e106a813fbf36cf92153553eb2a: Status 404 returned error can't find the container with id ae1cd1abd3bef7971cec0101eee4479ac00e7e106a813fbf36cf92153553eb2a Oct 30 04:56:44.016209 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:44.016171 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d11379_6d13_4b4b_a36b_4d418b490323.slice/crio-b5c9b8e51974fff9840c69c8cdeab46a8ab4a32715b4c2430dda3785c9b35b6f WatchSource:0}: Error finding container b5c9b8e51974fff9840c69c8cdeab46a8ab4a32715b4c2430dda3785c9b35b6f: Status 404 returned error can't find the container with id b5c9b8e51974fff9840c69c8cdeab46a8ab4a32715b4c2430dda3785c9b35b6f Oct 30 04:56:44.016330 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:44.016306 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk"] Oct 30 04:56:44.674416 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:44.671784 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" event={"ID":"46d11379-6d13-4b4b-a36b-4d418b490323","Type":"ContainerStarted","Data":"70940e2c3d864d54730995d4b39b65836deabb4454d8d72e326b65aa8a5ad835"} Oct 30 04:56:44.674416 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:44.671946 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" Oct 30 04:56:44.674416 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:44.671984 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" event={"ID":"46d11379-6d13-4b4b-a36b-4d418b490323","Type":"ContainerStarted","Data":"b5c9b8e51974fff9840c69c8cdeab46a8ab4a32715b4c2430dda3785c9b35b6f"} Oct 30 04:56:44.674416 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:44.673547 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" event={"ID":"d591ba5b-cdc7-4f23-85c6-28418b0abefe","Type":"ContainerStarted","Data":"e1cf2dd39fd53f68e4c349d1007fe123ce6e5fad721c1db4be4aa7a1e4b1a7e3"} Oct 30 04:56:44.674416 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:44.673585 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" event={"ID":"d591ba5b-cdc7-4f23-85c6-28418b0abefe","Type":"ContainerStarted","Data":"ae1cd1abd3bef7971cec0101eee4479ac00e7e106a813fbf36cf92153553eb2a"} Oct 30 04:56:44.674416 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:44.673606 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" Oct 30 04:56:44.711816 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:44.711759 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" podStartSLOduration=1.7117454840000002 podStartE2EDuration="1.711745484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:44.711209266 +0000 UTC m=+729.238976056" watchObservedRunningTime="2026-10-30 04:56:44.711745484 +0000 UTC m=+729.239512271" Oct 30 04:56:44.752076 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:44.752027 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" podStartSLOduration=1.752014271 podStartE2EDuration="1.752014271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:44.750241493 +0000 UTC m=+729.278008280" watchObservedRunningTime="2026-10-30 04:56:44.752014271 +0000 UTC m=+729.279781056" Oct 30 04:56:45.624258 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.624227 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr"] Oct 30 04:56:45.628099 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.628077 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" Oct 30 04:56:45.647334 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.647312 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr"] Oct 30 04:56:45.677535 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.677502 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-cff4ac\"" Oct 30 04:56:45.698773 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.698747 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn"] Oct 30 04:56:45.702721 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.702699 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" Oct 30 04:56:45.721588 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.721562 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-cff4ac-dockercfg-5zkgr\"" Oct 30 04:56:45.722689 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.722668 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn"] Oct 30 04:56:45.788828 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.788782 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scjrh\" (UniqueName: \"kubernetes.io/projected/06b19257-a9ac-42dc-a991-ce01ca2d6414-kube-api-access-scjrh\") pod \"payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr\" (UID: \"06b19257-a9ac-42dc-a991-ce01ca2d6414\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" Oct 30 04:56:45.789048 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.789026 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b19257-a9ac-42dc-a991-ce01ca2d6414-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr\" (UID: \"06b19257-a9ac-42dc-a991-ce01ca2d6414\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" Oct 30 04:56:45.890221 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.890133 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263665e1-3eeb-43a5-adb8-82a0d9269a0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn\" (UID: \"263665e1-3eeb-43a5-adb8-82a0d9269a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" Oct 30 04:56:45.890221 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.890183 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjrh\" (UniqueName: \"kubernetes.io/projected/06b19257-a9ac-42dc-a991-ce01ca2d6414-kube-api-access-scjrh\") pod \"payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr\" (UID: \"06b19257-a9ac-42dc-a991-ce01ca2d6414\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" Oct 30 04:56:45.890506 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.890327 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b19257-a9ac-42dc-a991-ce01ca2d6414-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr\" (UID: \"06b19257-a9ac-42dc-a991-ce01ca2d6414\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" Oct 30 04:56:45.890506 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.890442 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k79rx\" (UniqueName: \"kubernetes.io/projected/263665e1-3eeb-43a5-adb8-82a0d9269a0b-kube-api-access-k79rx\") pod \"payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn\" (UID: \"263665e1-3eeb-43a5-adb8-82a0d9269a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" Oct 30 04:56:45.891016 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.890997 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b19257-a9ac-42dc-a991-ce01ca2d6414-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr\" (UID: \"06b19257-a9ac-42dc-a991-ce01ca2d6414\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" Oct 30 04:56:45.910799 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.910769 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scjrh\" (UniqueName: \"kubernetes.io/projected/06b19257-a9ac-42dc-a991-ce01ca2d6414-kube-api-access-scjrh\") pod \"payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr\" (UID: \"06b19257-a9ac-42dc-a991-ce01ca2d6414\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" Oct 30 04:56:45.941544 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.941515 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" Oct 30 04:56:45.991915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.991306 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263665e1-3eeb-43a5-adb8-82a0d9269a0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn\" (UID: \"263665e1-3eeb-43a5-adb8-82a0d9269a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" Oct 30 04:56:45.991915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.991509 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k79rx\" (UniqueName: \"kubernetes.io/projected/263665e1-3eeb-43a5-adb8-82a0d9269a0b-kube-api-access-k79rx\") pod \"payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn\" (UID: \"263665e1-3eeb-43a5-adb8-82a0d9269a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" Oct 30 04:56:45.993738 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:45.992132 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263665e1-3eeb-43a5-adb8-82a0d9269a0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn\" (UID: \"263665e1-3eeb-43a5-adb8-82a0d9269a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" Oct 30 04:56:46.012419 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.011724 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k79rx\" (UniqueName: \"kubernetes.io/projected/263665e1-3eeb-43a5-adb8-82a0d9269a0b-kube-api-access-k79rx\") pod \"payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn\" (UID: \"263665e1-3eeb-43a5-adb8-82a0d9269a0b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" Oct 30 04:56:46.016804 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.016780 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" Oct 30 04:56:46.235724 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:46.235685 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b19257_a9ac_42dc_a991_ce01ca2d6414.slice/crio-4d4455fa0d4937391196f49ca8aecbb628c32e5c3aef25416d80ddb4c1eaae9e WatchSource:0}: Error finding container 4d4455fa0d4937391196f49ca8aecbb628c32e5c3aef25416d80ddb4c1eaae9e: Status 404 returned error can't find the container with id 4d4455fa0d4937391196f49ca8aecbb628c32e5c3aef25416d80ddb4c1eaae9e Oct 30 04:56:46.235922 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.235906 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr"] Oct 30 04:56:46.326096 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.326069 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn"] Oct 30 04:56:46.327347 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:46.327309 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263665e1_3eeb_43a5_adb8_82a0d9269a0b.slice/crio-5f2a0a9eb607dea9f5913925dd1cf1beab7f7fc604903878ffe6a69b3e847cb2 WatchSource:0}: Error finding container 5f2a0a9eb607dea9f5913925dd1cf1beab7f7fc604903878ffe6a69b3e847cb2: Status 404 returned error can't find the container with id 5f2a0a9eb607dea9f5913925dd1cf1beab7f7fc604903878ffe6a69b3e847cb2 Oct 30 04:56:46.467433 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.467403 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp"] Oct 30 04:56:46.471321 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.471296 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" Oct 30 04:56:46.493324 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.493254 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp"] Oct 30 04:56:46.494149 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.494125 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-0b21ba-dockercfg-jfdrj\"" Oct 30 04:56:46.568353 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.568321 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-0b21ba\"" Oct 30 04:56:46.600119 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.600054 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbtnc\" (UniqueName: \"kubernetes.io/projected/82856c63-29d3-42d2-9078-3880374c9ebc-kube-api-access-xbtnc\") pod \"payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp\" (UID: \"82856c63-29d3-42d2-9078-3880374c9ebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" Oct 30 04:56:46.600292 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.600215 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82856c63-29d3-42d2-9078-3880374c9ebc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp\" (UID: \"82856c63-29d3-42d2-9078-3880374c9ebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" Oct 30 04:56:46.706072 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.705980 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbtnc\" (UniqueName: \"kubernetes.io/projected/82856c63-29d3-42d2-9078-3880374c9ebc-kube-api-access-xbtnc\") pod \"payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp\" (UID: \"82856c63-29d3-42d2-9078-3880374c9ebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" Oct 30 04:56:46.712518 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.712479 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82856c63-29d3-42d2-9078-3880374c9ebc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp\" (UID: \"82856c63-29d3-42d2-9078-3880374c9ebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" Oct 30 04:56:46.713521 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.713497 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82856c63-29d3-42d2-9078-3880374c9ebc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp\" (UID: \"82856c63-29d3-42d2-9078-3880374c9ebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" Oct 30 04:56:46.720442 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.720340 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" event={"ID":"263665e1-3eeb-43a5-adb8-82a0d9269a0b","Type":"ContainerStarted","Data":"a41cac8a4f65730abbc6f8933a631e2eaee9e3d7a691cc9b8f9aa8f609201151"} Oct 30 04:56:46.720645 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.720629 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" event={"ID":"263665e1-3eeb-43a5-adb8-82a0d9269a0b","Type":"ContainerStarted","Data":"5f2a0a9eb607dea9f5913925dd1cf1beab7f7fc604903878ffe6a69b3e847cb2"} Oct 30 04:56:46.720824 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.720810 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" Oct 30 04:56:46.722694 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.722598 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" event={"ID":"06b19257-a9ac-42dc-a991-ce01ca2d6414","Type":"ContainerStarted","Data":"cce08864701e4f421e69d4ff6f0eba5ea38aa4a180c8d3bbe11f41b9daa854a0"} Oct 30 04:56:46.722694 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.722640 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" event={"ID":"06b19257-a9ac-42dc-a991-ce01ca2d6414","Type":"ContainerStarted","Data":"4d4455fa0d4937391196f49ca8aecbb628c32e5c3aef25416d80ddb4c1eaae9e"} Oct 30 04:56:46.722859 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.722837 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" Oct 30 04:56:46.729983 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.729960 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbtnc\" (UniqueName: \"kubernetes.io/projected/82856c63-29d3-42d2-9078-3880374c9ebc-kube-api-access-xbtnc\") pod \"payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp\" (UID: \"82856c63-29d3-42d2-9078-3880374c9ebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" Oct 30 04:56:46.786485 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.786406 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" Oct 30 04:56:46.821222 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.820180 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" podStartSLOduration=1.820160835 podStartE2EDuration="1.820160835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:46.776686928 +0000 UTC m=+731.304453717" watchObservedRunningTime="2026-10-30 04:56:46.820160835 +0000 UTC m=+731.347927619" Oct 30 04:56:46.821222 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:46.821037 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" podStartSLOduration=1.82103039 podStartE2EDuration="1.82103039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:46.817636499 +0000 UTC m=+731.345403286" watchObservedRunningTime="2026-10-30 04:56:46.82103039 +0000 UTC m=+731.348797175" Oct 30 04:56:47.089497 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:47.089465 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp"] Oct 30 04:56:47.091794 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:47.091761 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82856c63_29d3_42d2_9078_3880374c9ebc.slice/crio-be6e6e717c1d05c257ede3475511b32c0ac2b2c3de495736262b2a7dead8794b WatchSource:0}: Error finding container be6e6e717c1d05c257ede3475511b32c0ac2b2c3de495736262b2a7dead8794b: Status 404 returned error can't find the container with id be6e6e717c1d05c257ede3475511b32c0ac2b2c3de495736262b2a7dead8794b Oct 30 04:56:47.733765 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:47.733725 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" event={"ID":"82856c63-29d3-42d2-9078-3880374c9ebc","Type":"ContainerStarted","Data":"e6bae3b936acd7fd399ca27429e6e33590a0d228e516f3573c88fd3eecc64e8f"} Oct 30 04:56:47.734255 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:47.734003 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" Oct 30 04:56:47.734255 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:47.734038 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" event={"ID":"82856c63-29d3-42d2-9078-3880374c9ebc","Type":"ContainerStarted","Data":"be6e6e717c1d05c257ede3475511b32c0ac2b2c3de495736262b2a7dead8794b"} Oct 30 04:56:47.773246 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:47.773196 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" podStartSLOduration=1.773182821 podStartE2EDuration="1.773182821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:47.772160877 +0000 UTC m=+732.299927663" watchObservedRunningTime="2026-10-30 04:56:47.773182821 +0000 UTC m=+732.300949607" Oct 30 04:56:48.963585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:48.963547 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2"] Oct 30 04:56:48.967473 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:48.967454 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" Oct 30 04:56:48.997081 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:48.997044 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-d55037-dockercfg-mh8tn\"" Oct 30 04:56:48.997081 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:48.997062 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-d55037\"" Oct 30 04:56:48.999108 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:48.999081 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2"] Oct 30 04:56:49.010112 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.010084 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v"] Oct 30 04:56:49.014733 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.014712 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" Oct 30 04:56:49.032599 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.032572 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v"] Oct 30 04:56:49.037864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.037839 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f557a246-4b9a-4475-b2f1-5b7df159e4ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2\" (UID: \"f557a246-4b9a-4475-b2f1-5b7df159e4ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" Oct 30 04:56:49.037989 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.037936 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2266h\" (UniqueName: \"kubernetes.io/projected/a00441e4-210e-48f0-bd77-057db2fe849d-kube-api-access-2266h\") pod \"payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v\" (UID: \"a00441e4-210e-48f0-bd77-057db2fe849d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" Oct 30 04:56:49.038059 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.038038 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a00441e4-210e-48f0-bd77-057db2fe849d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v\" (UID: \"a00441e4-210e-48f0-bd77-057db2fe849d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" Oct 30 04:56:49.038114 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.038079 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwsrq\" (UniqueName: \"kubernetes.io/projected/f557a246-4b9a-4475-b2f1-5b7df159e4ac-kube-api-access-rwsrq\") pod \"payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2\" (UID: \"f557a246-4b9a-4475-b2f1-5b7df159e4ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" Oct 30 04:56:49.139261 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.139229 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a00441e4-210e-48f0-bd77-057db2fe849d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v\" (UID: \"a00441e4-210e-48f0-bd77-057db2fe849d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" Oct 30 04:56:49.139261 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.139268 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwsrq\" (UniqueName: \"kubernetes.io/projected/f557a246-4b9a-4475-b2f1-5b7df159e4ac-kube-api-access-rwsrq\") pod \"payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2\" (UID: \"f557a246-4b9a-4475-b2f1-5b7df159e4ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" Oct 30 04:56:49.139568 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.139417 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f557a246-4b9a-4475-b2f1-5b7df159e4ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2\" (UID: \"f557a246-4b9a-4475-b2f1-5b7df159e4ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" Oct 30 04:56:49.139568 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.139444 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2266h\" (UniqueName: \"kubernetes.io/projected/a00441e4-210e-48f0-bd77-057db2fe849d-kube-api-access-2266h\") pod \"payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v\" (UID: \"a00441e4-210e-48f0-bd77-057db2fe849d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" Oct 30 04:56:49.140186 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.140160 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a00441e4-210e-48f0-bd77-057db2fe849d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v\" (UID: \"a00441e4-210e-48f0-bd77-057db2fe849d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" Oct 30 04:56:49.140253 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.140208 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f557a246-4b9a-4475-b2f1-5b7df159e4ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2\" (UID: \"f557a246-4b9a-4475-b2f1-5b7df159e4ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" Oct 30 04:56:49.159300 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.159277 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwsrq\" (UniqueName: \"kubernetes.io/projected/f557a246-4b9a-4475-b2f1-5b7df159e4ac-kube-api-access-rwsrq\") pod \"payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2\" (UID: \"f557a246-4b9a-4475-b2f1-5b7df159e4ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" Oct 30 04:56:49.159500 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.159482 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2266h\" (UniqueName: \"kubernetes.io/projected/a00441e4-210e-48f0-bd77-057db2fe849d-kube-api-access-2266h\") pod \"payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v\" (UID: \"a00441e4-210e-48f0-bd77-057db2fe849d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" Oct 30 04:56:49.280497 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.280417 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" Oct 30 04:56:49.328560 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.328526 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" Oct 30 04:56:49.575989 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:49.575961 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf557a246_4b9a_4475_b2f1_5b7df159e4ac.slice/crio-e8c40b3773582c9b11b9ee8d5cde9d81945edd1cb528d930ca077bd745c9f359 WatchSource:0}: Error finding container e8c40b3773582c9b11b9ee8d5cde9d81945edd1cb528d930ca077bd745c9f359: Status 404 returned error can't find the container with id e8c40b3773582c9b11b9ee8d5cde9d81945edd1cb528d930ca077bd745c9f359 Oct 30 04:56:49.581578 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.581543 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2"] Oct 30 04:56:49.631611 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:49.631586 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00441e4_210e_48f0_bd77_057db2fe849d.slice/crio-04d6edbe680bc68a3750de3ce26d26512a64ceeec09e39884a866f159643b9de WatchSource:0}: Error finding container 04d6edbe680bc68a3750de3ce26d26512a64ceeec09e39884a866f159643b9de: Status 404 returned error can't find the container with id 04d6edbe680bc68a3750de3ce26d26512a64ceeec09e39884a866f159643b9de Oct 30 04:56:49.631872 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.631848 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v"] Oct 30 04:56:49.747280 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.747241 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" event={"ID":"f557a246-4b9a-4475-b2f1-5b7df159e4ac","Type":"ContainerStarted","Data":"df893f2c6caf6dcbafc1e2100e09b6625787783476c4a33f29f077cf43972896"} Oct 30 04:56:49.747459 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.747297 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" event={"ID":"f557a246-4b9a-4475-b2f1-5b7df159e4ac","Type":"ContainerStarted","Data":"e8c40b3773582c9b11b9ee8d5cde9d81945edd1cb528d930ca077bd745c9f359"} Oct 30 04:56:49.747459 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.747390 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" Oct 30 04:56:49.748897 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.748866 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" event={"ID":"a00441e4-210e-48f0-bd77-057db2fe849d","Type":"ContainerStarted","Data":"65a69e5de952121f728ec6ec17b89b30295e41552e317b849de694e9574bc2f3"} Oct 30 04:56:49.748897 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.748898 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" event={"ID":"a00441e4-210e-48f0-bd77-057db2fe849d","Type":"ContainerStarted","Data":"04d6edbe680bc68a3750de3ce26d26512a64ceeec09e39884a866f159643b9de"} Oct 30 04:56:49.749070 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.748994 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" Oct 30 04:56:49.786249 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.786194 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" podStartSLOduration=1.786180035 podStartE2EDuration="1.786180035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:49.785805158 +0000 UTC m=+734.313571949" watchObservedRunningTime="2026-10-30 04:56:49.786180035 +0000 UTC m=+734.313946825" Oct 30 04:56:49.826064 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:49.825969 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" podStartSLOduration=1.8259523610000001 podStartE2EDuration="1.825952361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:49.823897342 +0000 UTC m=+734.351664129" watchObservedRunningTime="2026-10-30 04:56:49.825952361 +0000 UTC m=+734.353719146" Oct 30 04:56:50.527098 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.527060 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5"] Oct 30 04:56:50.531709 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.531690 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" Oct 30 04:56:50.549737 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.549705 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5"] Oct 30 04:56:50.552484 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.552464 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-83e83e\"" Oct 30 04:56:50.560050 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.560025 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5xs\" (UniqueName: \"kubernetes.io/projected/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-kube-api-access-cz5xs\") pod \"payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5\" (UID: \"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" Oct 30 04:56:50.560180 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.560081 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5\" (UID: \"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" Oct 30 04:56:50.573397 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.573373 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z"] Oct 30 04:56:50.578030 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.578002 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" Oct 30 04:56:50.579064 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.579042 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-83e83e-dockercfg-7wjdd\"" Oct 30 04:56:50.594508 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.594480 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z"] Oct 30 04:56:50.663678 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.663633 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c7bf20-a8c4-426e-b692-41e564879c82-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z\" (UID: \"82c7bf20-a8c4-426e-b692-41e564879c82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" Oct 30 04:56:50.663834 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.663700 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5xs\" (UniqueName: \"kubernetes.io/projected/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-kube-api-access-cz5xs\") pod \"payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5\" (UID: \"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" Oct 30 04:56:50.663834 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.663761 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5\" (UID: \"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" Oct 30 04:56:50.663834 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.663789 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcb75\" (UniqueName: \"kubernetes.io/projected/82c7bf20-a8c4-426e-b692-41e564879c82-kube-api-access-rcb75\") pod \"payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z\" (UID: \"82c7bf20-a8c4-426e-b692-41e564879c82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" Oct 30 04:56:50.665386 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.665311 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5\" (UID: \"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" Oct 30 04:56:50.684690 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.684657 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5xs\" (UniqueName: \"kubernetes.io/projected/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-kube-api-access-cz5xs\") pod \"payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5\" (UID: \"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" Oct 30 04:56:50.764984 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.764946 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c7bf20-a8c4-426e-b692-41e564879c82-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z\" (UID: \"82c7bf20-a8c4-426e-b692-41e564879c82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" Oct 30 04:56:50.765180 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.765021 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcb75\" (UniqueName: \"kubernetes.io/projected/82c7bf20-a8c4-426e-b692-41e564879c82-kube-api-access-rcb75\") pod \"payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z\" (UID: \"82c7bf20-a8c4-426e-b692-41e564879c82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" Oct 30 04:56:50.765672 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.765650 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c7bf20-a8c4-426e-b692-41e564879c82-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z\" (UID: \"82c7bf20-a8c4-426e-b692-41e564879c82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" Oct 30 04:56:50.784787 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.784727 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcb75\" (UniqueName: \"kubernetes.io/projected/82c7bf20-a8c4-426e-b692-41e564879c82-kube-api-access-rcb75\") pod \"payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z\" (UID: \"82c7bf20-a8c4-426e-b692-41e564879c82\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" Oct 30 04:56:50.846487 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.846449 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" Oct 30 04:56:50.896442 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:50.896415 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" Oct 30 04:56:51.154870 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:51.154840 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5"] Oct 30 04:56:51.155822 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:51.155797 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02adafe_f0d8_406c_b9d0_3dc6b6f51ebc.slice/crio-4df78f7eb7eaf327e24777ea3967422d96134ae94ce8b68ea58dc908ec645793 WatchSource:0}: Error finding container 4df78f7eb7eaf327e24777ea3967422d96134ae94ce8b68ea58dc908ec645793: Status 404 returned error can't find the container with id 4df78f7eb7eaf327e24777ea3967422d96134ae94ce8b68ea58dc908ec645793 Oct 30 04:56:51.248004 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:51.247971 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z"] Oct 30 04:56:51.249753 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:56:51.249595 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c7bf20_a8c4_426e_b692_41e564879c82.slice/crio-f7e119773228bc8d12b494766fc923b0fbaa22729345d2e59f552ec3a4c13a64 WatchSource:0}: Error finding container f7e119773228bc8d12b494766fc923b0fbaa22729345d2e59f552ec3a4c13a64: Status 404 returned error can't find the container with id f7e119773228bc8d12b494766fc923b0fbaa22729345d2e59f552ec3a4c13a64 Oct 30 04:56:51.771757 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:51.771717 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" event={"ID":"82c7bf20-a8c4-426e-b692-41e564879c82","Type":"ContainerStarted","Data":"1a195695ce00996f05c6d3bd2cd7a0843043f2f61972e773352e231857e244ce"} Oct 30 04:56:51.772342 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:51.772316 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" event={"ID":"82c7bf20-a8c4-426e-b692-41e564879c82","Type":"ContainerStarted","Data":"f7e119773228bc8d12b494766fc923b0fbaa22729345d2e59f552ec3a4c13a64"} Oct 30 04:56:51.773291 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:51.773263 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" Oct 30 04:56:51.774962 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:51.774938 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" event={"ID":"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc","Type":"ContainerStarted","Data":"f3bf091bde286d02bf2ea48d77d6baf32ded0e349c09825f5935b16269d7a09b"} Oct 30 04:56:51.775055 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:51.774983 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" event={"ID":"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc","Type":"ContainerStarted","Data":"4df78f7eb7eaf327e24777ea3967422d96134ae94ce8b68ea58dc908ec645793"} Oct 30 04:56:51.778423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:51.775515 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" Oct 30 04:56:51.821382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:51.817141 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" podStartSLOduration=1.817120053 podStartE2EDuration="1.817120053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:51.815519645 +0000 UTC m=+736.343286434" watchObservedRunningTime="2026-10-30 04:56:51.817120053 +0000 UTC m=+736.344886841" Oct 30 04:56:51.858604 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:51.858545 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" podStartSLOduration=1.85853003 podStartE2EDuration="1.85853003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:51.856376914 +0000 UTC m=+736.384143697" watchObservedRunningTime="2026-10-30 04:56:51.85853003 +0000 UTC m=+736.386296815" Oct 30 04:56:53.668839 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:53.668812 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" Oct 30 04:56:53.669235 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:53.668884 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" Oct 30 04:56:55.682798 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:55.682763 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" Oct 30 04:56:55.683291 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:55.682837 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" Oct 30 04:56:55.984593 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:56:55.984508 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:56:57.737270 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:57.737194 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" Oct 30 04:56:57.738519 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:57.738501 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" Oct 30 04:56:58.743471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:56:58.743443 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" Oct 30 04:57:00.762028 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:00.761999 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" Oct 30 04:57:00.762431 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:00.762214 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" Oct 30 04:57:01.868136 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:01.868103 6587 generic.go:356] "Generic (PLEG): container finished" podID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerID="3d037236381e06aefd12320ec64df2c9a5de69c2703e82cf9f9d1de25efc776f" exitCode=0 Oct 30 04:57:01.868676 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:01.868174 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" event={"ID":"a3a110c9-725e-4466-b182-1f608f0d0ba0","Type":"ContainerDied","Data":"3d037236381e06aefd12320ec64df2c9a5de69c2703e82cf9f9d1de25efc776f"} Oct 30 04:57:01.868676 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:01.868572 6587 scope.go:122] "RemoveContainer" containerID="3d037236381e06aefd12320ec64df2c9a5de69c2703e82cf9f9d1de25efc776f" Oct 30 04:57:01.869939 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:01.869912 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:01.870038 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:01.869976 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:01.870038 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:01.870010 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:02.889381 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:02.888410 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" event={"ID":"a3a110c9-725e-4466-b182-1f608f0d0ba0","Type":"ContainerStarted","Data":"ff42e523667a3265eaf4d4690177625ea88e0a5a58a4f99ef10c6b385e908dcc"} Oct 30 04:57:03.715388 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:03.715240 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:57:03.715599 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:03.715484 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 04:57:03.715599 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:03.715550 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 04:57:03.796566 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:03.796478 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" Oct 30 04:57:03.797055 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:03.797037 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" Oct 30 04:57:04.719383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:04.715439 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 04:57:04.719383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:04.715513 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 04:57:05.726385 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:05.719497 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 04:57:05.726385 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:05.719603 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 04:57:06.715226 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:06.714740 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 04:57:06.715226 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:06.714832 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 04:57:07.714644 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:07.714610 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 04:57:07.715113 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:07.714817 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 04:57:08.720677 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:08.720642 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:57:08.721129 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:08.721079 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:57:08.722250 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:08.722232 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 04:57:09.452921 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:09.452881 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:43158->10.134.0.44:15021: read: connection reset by peer" start-of-body= Oct 30 04:57:09.453114 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:09.452950 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:43158->10.134.0.44:15021: read: connection reset by peer" Oct 30 04:57:09.678891 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:09.678503 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:57:09.678891 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:09.678572 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:57:10.003007 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:10.002975 6587 generic.go:356] "Generic (PLEG): container finished" podID="6ff505e8-bee4-4f87-93e7-113497e13649" containerID="93a935b2fdbd1e32c26a7cf0c6d6b6e26ff57dfe72f43e7aba3fcfa9474d928f" exitCode=0 Oct 30 04:57:10.003428 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:10.003049 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" event={"ID":"6ff505e8-bee4-4f87-93e7-113497e13649","Type":"ContainerDied","Data":"93a935b2fdbd1e32c26a7cf0c6d6b6e26ff57dfe72f43e7aba3fcfa9474d928f"} Oct 30 04:57:10.003638 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:10.003616 6587 scope.go:122] "RemoveContainer" containerID="93a935b2fdbd1e32c26a7cf0c6d6b6e26ff57dfe72f43e7aba3fcfa9474d928f" Oct 30 04:57:10.005023 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:10.004991 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:10.005115 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:10.005085 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:10.005162 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:10.005132 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:11.010413 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:11.010375 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" event={"ID":"6ff505e8-bee4-4f87-93e7-113497e13649","Type":"ContainerStarted","Data":"f380abee44cda86058a7992ea1f698fb29eac236235ea2bb5a6bd997bc30cd67"} Oct 30 04:57:11.014939 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:11.014920 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:57:11.015131 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:11.015110 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 04:57:11.015200 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:11.015156 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 04:57:12.014160 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:12.014129 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 04:57:12.014592 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:12.014177 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 04:57:13.015063 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:13.015026 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 04:57:13.015532 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:13.015079 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 04:57:13.050575 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:13.050546 6587 generic.go:356] "Generic (PLEG): container finished" podID="6df4067a-ac1c-496d-83d1-470dd606212d" containerID="3b6ecc9a07c19efada4e684d89ea5f4d5e2f37d9dbd7e4e2804f4542c7b55b20" exitCode=0 Oct 30 04:57:13.050794 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:13.050703 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" event={"ID":"6df4067a-ac1c-496d-83d1-470dd606212d","Type":"ContainerDied","Data":"3b6ecc9a07c19efada4e684d89ea5f4d5e2f37d9dbd7e4e2804f4542c7b55b20"} Oct 30 04:57:13.051302 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:13.051284 6587 scope.go:122] "RemoveContainer" containerID="3b6ecc9a07c19efada4e684d89ea5f4d5e2f37d9dbd7e4e2804f4542c7b55b20" Oct 30 04:57:13.052850 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:13.052823 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:13.052929 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:13.052889 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:13.052929 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:13.052922 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:14.015214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:14.015171 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 04:57:14.015722 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:14.015273 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 04:57:14.076801 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:14.076765 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" event={"ID":"6df4067a-ac1c-496d-83d1-470dd606212d","Type":"ContainerStarted","Data":"21cdf1c63cef0ef58b0dfd95a16d5ebaf6d6c8af91a66a0d08e1f414aa8dd262"} Oct 30 04:57:14.322134 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:14.322102 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:57:14.346422 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:14.322496 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 04:57:14.346692 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:14.346660 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 04:57:15.015000 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:15.014963 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 04:57:15.015190 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:15.015016 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 04:57:15.319804 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:15.319765 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 04:57:15.320799 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:15.320395 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 04:57:16.015225 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:16.015195 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 04:57:16.015438 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:16.015246 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 04:57:16.321380 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:16.320741 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 04:57:16.321380 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:16.320802 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 04:57:17.029719 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:17.029684 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:57:17.032107 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:17.032059 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:57:17.038607 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:17.038582 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 04:57:17.319893 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:17.319858 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 04:57:17.320208 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:17.320178 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 04:57:18.141381 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.132762 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst"] Oct 30 04:57:18.158536 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.158507 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst"] Oct 30 04:57:18.158792 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.158774 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.176306 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.176278 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cff4ac\"/\"openshift-service-ca.crt\"" Oct 30 04:57:18.176306 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.176287 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cff4ac\"/\"kube-root-ca.crt\"" Oct 30 04:57:18.189880 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.189856 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cff4ac\"/\"default-dockercfg-7dcgl\"" Oct 30 04:57:18.190159 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.190142 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cff4ac\"/\"simulator-w1-cff4ac-kserve-self-signed-certs\"" Oct 30 04:57:18.270900 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.270865 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tls-certs\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.271070 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.270958 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrwg\" (UniqueName: \"kubernetes.io/projected/faad5ec5-3afc-4be7-8a65-6072b41c51a7-kube-api-access-fwrwg\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.271070 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.270983 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tmp-dir\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.271070 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.271006 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-home\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.271070 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.271030 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-dshm\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.271070 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.271047 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-model-cache\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.328383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.323747 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 04:57:18.328383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.323817 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 04:57:18.373980 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.373941 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-dshm\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.374145 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.373992 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-model-cache\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.374145 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.374128 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tls-certs\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.374222 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.374207 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrwg\" (UniqueName: \"kubernetes.io/projected/faad5ec5-3afc-4be7-8a65-6072b41c51a7-kube-api-access-fwrwg\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.374264 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.374251 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tmp-dir\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.374298 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.374285 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-home\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.374727 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.374699 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tmp-dir\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.374894 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.374753 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-model-cache\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.374894 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.374847 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-home\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.378926 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.378903 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-dshm\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.381130 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.381102 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tls-certs\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.394231 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.394169 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrwg\" (UniqueName: \"kubernetes.io/projected/faad5ec5-3afc-4be7-8a65-6072b41c51a7-kube-api-access-fwrwg\") pod \"simulator-w1-cff4ac-kserve-5756c7bc77-wlvst\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.473754 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.473716 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:18.793510 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:57:18.793464 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaad5ec5_3afc_4be7_8a65_6072b41c51a7.slice/crio-d37c4fdbaa5418237112e1b1a35afd0b9e9e0c68dedbb771234f9a613c77f42a WatchSource:0}: Error finding container d37c4fdbaa5418237112e1b1a35afd0b9e9e0c68dedbb771234f9a613c77f42a: Status 404 returned error can't find the container with id d37c4fdbaa5418237112e1b1a35afd0b9e9e0c68dedbb771234f9a613c77f42a Oct 30 04:57:18.794385 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:18.794340 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst"] Oct 30 04:57:19.107790 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.107758 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" event={"ID":"faad5ec5-3afc-4be7-8a65-6072b41c51a7","Type":"ContainerStarted","Data":"16ce9a6d9bbae1d4dd4a9ea05fd5020eae94c018ffe07f56cdfee6835856c975"} Oct 30 04:57:19.107790 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.107796 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" event={"ID":"faad5ec5-3afc-4be7-8a65-6072b41c51a7","Type":"ContainerStarted","Data":"d37c4fdbaa5418237112e1b1a35afd0b9e9e0c68dedbb771234f9a613c77f42a"} Oct 30 04:57:19.148984 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.148933 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podStartSLOduration=1.148917794 podStartE2EDuration="1.148917794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:19.147422849 +0000 UTC m=+763.675189635" watchObservedRunningTime="2026-10-30 04:57:19.148917794 +0000 UTC m=+763.676684580" Oct 30 04:57:19.394531 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.394428 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:57:19.394994 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.394971 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:57:19.396071 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.396041 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 04:57:19.474025 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.473995 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:19.731817 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.731741 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8"] Oct 30 04:57:19.735463 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.735376 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.752285 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.752259 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0b21ba\"/\"openshift-service-ca.crt\"" Oct 30 04:57:19.753468 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.753446 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8"] Oct 30 04:57:19.753739 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.753611 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0b21ba\"/\"default-dockercfg-qgzhb\"" Oct 30 04:57:19.753739 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.753709 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0b21ba\"/\"kube-root-ca.crt\"" Oct 30 04:57:19.764227 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.764206 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0b21ba\"/\"simulator-w5-0b21ba-kserve-self-signed-certs\"" Oct 30 04:57:19.788967 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.788941 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktthr\" (UniqueName: \"kubernetes.io/projected/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-kube-api-access-ktthr\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.789054 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.788985 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-dshm\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.789098 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.789067 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-model-cache\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.789137 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.789113 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tls-certs\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.789180 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.789144 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-home\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.789223 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.789192 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tmp-dir\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.889915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.889883 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tls-certs\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.890100 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.889926 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-home\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.890100 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.889945 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tmp-dir\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.890100 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.890005 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktthr\" (UniqueName: \"kubernetes.io/projected/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-kube-api-access-ktthr\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.890100 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.890041 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-dshm\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.890100 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.890069 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-model-cache\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.890463 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.890441 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-home\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.890553 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.890478 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tmp-dir\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.890553 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.890507 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-model-cache\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.892281 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.892263 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-dshm\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.892489 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.892473 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tls-certs\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:19.910250 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:19.910215 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktthr\" (UniqueName: \"kubernetes.io/projected/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-kube-api-access-ktthr\") pod \"simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:20.049255 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:20.049231 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:20.346262 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:57:20.346226 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfef5d50e_c6e1_4f7e_ad51_af4edc5c1a31.slice/crio-25f5cdf2a1d61b4ea5fdbe406920ab4f55e5d6b2390c64d2e0a5adb960afe40f WatchSource:0}: Error finding container 25f5cdf2a1d61b4ea5fdbe406920ab4f55e5d6b2390c64d2e0a5adb960afe40f: Status 404 returned error can't find the container with id 25f5cdf2a1d61b4ea5fdbe406920ab4f55e5d6b2390c64d2e0a5adb960afe40f Oct 30 04:57:20.347855 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:20.346307 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8"] Oct 30 04:57:21.131962 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:21.131918 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" event={"ID":"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31","Type":"ContainerStarted","Data":"b45cb13399d9a75e6266dd02367a70f6835d224d4e9f9d12368aecfd15c9a3ac"} Oct 30 04:57:21.132154 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:21.131971 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" event={"ID":"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31","Type":"ContainerStarted","Data":"25f5cdf2a1d61b4ea5fdbe406920ab4f55e5d6b2390c64d2e0a5adb960afe40f"} Oct 30 04:57:21.173598 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:21.173541 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podStartSLOduration=2.173523004 podStartE2EDuration="2.173523004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:21.172303876 +0000 UTC m=+765.700070663" watchObservedRunningTime="2026-10-30 04:57:21.173523004 +0000 UTC m=+765.701289792" Oct 30 04:57:22.049953 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:22.049918 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:25.903773 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:25.903728 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj"] Oct 30 04:57:25.907312 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:25.907288 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:25.919545 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:25.919519 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj"] Oct 30 04:57:25.930207 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:25.930173 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-889ca9\"/\"openshift-service-ca.crt\"" Oct 30 04:57:25.936153 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:25.936117 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-889ca9\"/\"kube-root-ca.crt\"" Oct 30 04:57:25.936345 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:25.936322 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-889ca9\"/\"simulator-w3-889ca9-kserve-self-signed-certs\"" Oct 30 04:57:25.936459 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:25.936121 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-889ca9\"/\"default-dockercfg-gf5t6\"" Oct 30 04:57:25.968430 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:25.968402 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tmp-dir\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:25.968430 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:25.968434 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-dshm\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:25.968630 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:25.968467 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tls-certs\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:25.968630 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:25.968603 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74l8c\" (UniqueName: \"kubernetes.io/projected/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-kube-api-access-74l8c\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:25.968737 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:25.968722 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-home\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:25.968796 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:25.968782 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-model-cache\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:25.985845 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:57:25.985809 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:57:26.069642 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.069605 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-home\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:26.069819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.069654 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-model-cache\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:26.069819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.069689 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tmp-dir\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:26.069819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.069704 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-dshm\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:26.069819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.069727 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tls-certs\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:26.069819 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.069763 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74l8c\" (UniqueName: \"kubernetes.io/projected/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-kube-api-access-74l8c\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:26.070102 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.070023 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-home\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:26.070102 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.070075 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tmp-dir\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:26.070197 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.070149 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-model-cache\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:26.072025 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.072002 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-dshm\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:26.072304 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.072285 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tls-certs\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:26.089960 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.089937 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74l8c\" (UniqueName: \"kubernetes.io/projected/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-kube-api-access-74l8c\") pod \"simulator-w3-889ca9-kserve-64bd59c796-n58vj\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:26.225092 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.224998 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:26.526798 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:57:26.526766 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9889ac0a_17cf_41c2_b40b_b5ed537cc94c.slice/crio-934403b52864d11d81944389ecf798b90fbd7458b7d34299e4ec243b688a7314 WatchSource:0}: Error finding container 934403b52864d11d81944389ecf798b90fbd7458b7d34299e4ec243b688a7314: Status 404 returned error can't find the container with id 934403b52864d11d81944389ecf798b90fbd7458b7d34299e4ec243b688a7314 Oct 30 04:57:26.527157 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:26.527128 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj"] Oct 30 04:57:27.176685 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:27.176203 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" event={"ID":"9889ac0a-17cf-41c2-b40b-b5ed537cc94c","Type":"ContainerStarted","Data":"f6a94b57388cc1058fefd5617f7e85408ffa91605d5fcda533a88d934be6cdcd"} Oct 30 04:57:27.176685 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:27.176256 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" event={"ID":"9889ac0a-17cf-41c2-b40b-b5ed537cc94c","Type":"ContainerStarted","Data":"934403b52864d11d81944389ecf798b90fbd7458b7d34299e4ec243b688a7314"} Oct 30 04:57:27.218954 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:27.218898 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podStartSLOduration=2.218883538 podStartE2EDuration="2.218883538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:27.216835428 +0000 UTC m=+771.744602214" watchObservedRunningTime="2026-10-30 04:57:27.218883538 +0000 UTC m=+771.746650323" Oct 30 04:57:27.225924 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:27.225899 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:28.476377 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:28.476027 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:28.515378 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:28.512965 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:29.200243 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:29.200216 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 04:57:30.049989 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:30.049947 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:30.069324 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:30.069297 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:30.205772 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:30.205747 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:57:36.078725 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.078689 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk"] Oct 30 04:57:36.082002 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.081984 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.103732 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.103708 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk"] Oct 30 04:57:36.114272 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.114244 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-397179-openshift-default-dockercfg-q4dsf\"" Oct 30 04:57:36.189475 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.189436 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.189633 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.189483 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.189633 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.189597 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.189716 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.189663 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dabb599-1beb-451d-b03d-87ed3942f5ff-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.189751 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.189724 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.189793 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.189761 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.189845 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.189832 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.189883 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.189869 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4rdt\" (UniqueName: \"kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-kube-api-access-j4rdt\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.189962 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.189948 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.225243 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.225217 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:36.238674 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.238654 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:36.249846 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.249828 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 04:57:36.291137 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.291113 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.291268 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.291172 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.291268 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.291219 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.291268 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.291261 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4rdt\" (UniqueName: \"kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-kube-api-access-j4rdt\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.291477 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.291451 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.291631 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.291597 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.291631 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.291626 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.291808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.291654 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.291808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.291624 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.291808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.291707 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dabb599-1beb-451d-b03d-87ed3942f5ff-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.291966 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.291800 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.291966 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.291851 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.291966 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.291920 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.292266 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.292248 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dabb599-1beb-451d-b03d-87ed3942f5ff-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.293822 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.293799 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.293963 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.293943 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.310838 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.310818 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.311014 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.310991 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4rdt\" (UniqueName: \"kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-kube-api-access-j4rdt\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.394353 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.394274 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:36.710943 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.710914 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk"] Oct 30 04:57:36.711740 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:57:36.711706 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dabb599_1beb_451d_b03d_87ed3942f5ff.slice/crio-c640ab3ca3717f769cbfd4fb5c85c1e6403d7cd389ca198442848196c5f74dea WatchSource:0}: Error finding container c640ab3ca3717f769cbfd4fb5c85c1e6403d7cd389ca198442848196c5f74dea: Status 404 returned error can't find the container with id c640ab3ca3717f769cbfd4fb5c85c1e6403d7cd389ca198442848196c5f74dea Oct 30 04:57:36.714185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.714151 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:36.714282 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.714216 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:36.714282 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:36.714244 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:57:37.250157 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:37.250122 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" event={"ID":"1dabb599-1beb-451d-b03d-87ed3942f5ff","Type":"ContainerStarted","Data":"5e5453bebf90d6e50081d246ea1513b63fb55c71dddf9011ed4a3c419c26a4bc"} Oct 30 04:57:37.250585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:37.250192 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" event={"ID":"1dabb599-1beb-451d-b03d-87ed3942f5ff","Type":"ContainerStarted","Data":"c640ab3ca3717f769cbfd4fb5c85c1e6403d7cd389ca198442848196c5f74dea"} Oct 30 04:57:37.296874 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:37.296817 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" podStartSLOduration=1.2968013 podStartE2EDuration="1.2968013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:37.294324619 +0000 UTC m=+781.822091407" watchObservedRunningTime="2026-10-30 04:57:37.2968013 +0000 UTC m=+781.824568086" Oct 30 04:57:37.394639 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:37.394608 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:37.396130 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:37.396105 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 04:57:37.396259 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:37.396152 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" podUID="1dabb599-1beb-451d-b03d-87ed3942f5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 04:57:38.394728 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:38.394683 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 04:57:38.395210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:38.394747 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" podUID="1dabb599-1beb-451d-b03d-87ed3942f5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 04:57:39.395123 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:39.395088 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 04:57:39.395548 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:39.395143 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" podUID="1dabb599-1beb-451d-b03d-87ed3942f5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 04:57:40.395123 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:40.395089 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 04:57:40.395300 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:40.395142 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" podUID="1dabb599-1beb-451d-b03d-87ed3942f5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 04:57:41.396749 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:41.396712 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 04:57:41.397372 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:41.397321 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" podUID="1dabb599-1beb-451d-b03d-87ed3942f5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 04:57:42.394936 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:42.394898 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 04:57:42.395118 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:42.394960 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" podUID="1dabb599-1beb-451d-b03d-87ed3942f5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 04:57:43.399553 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:43.399520 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:43.399953 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:43.399810 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:43.400491 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:57:43.400465 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 04:57:55.985054 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:57:55.985011 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:58:05.350226 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.350193 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw"] Oct 30 04:58:05.355820 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.355796 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.374663 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.374632 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw"] Oct 30 04:58:05.384114 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.384077 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-83e83e\"/\"kube-root-ca.crt\"" Oct 30 04:58:05.384997 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.384975 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-83e83e\"/\"premium-w6-83e83e-kserve-self-signed-certs\"" Oct 30 04:58:05.385158 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.385002 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-83e83e\"/\"openshift-service-ca.crt\"" Oct 30 04:58:05.385224 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.385163 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-83e83e\"/\"default-dockercfg-m9psh\"" Oct 30 04:58:05.475420 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.475350 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-home\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.475586 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.475508 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-dshm\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.475586 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.475570 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607c7ab6-765b-494e-b2b5-e40f0a16e956-tls-certs\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.475717 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.475691 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-tmp-dir\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.475752 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.475727 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh82t\" (UniqueName: \"kubernetes.io/projected/607c7ab6-765b-494e-b2b5-e40f0a16e956-kube-api-access-kh82t\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.475865 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.475850 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-model-cache\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.577028 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.576989 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-model-cache\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.577409 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.577389 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-home\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.577591 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.577492 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-model-cache\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.577752 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.577728 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-dshm\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.577752 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.577739 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-home\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.577941 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.577811 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607c7ab6-765b-494e-b2b5-e40f0a16e956-tls-certs\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.577941 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.577863 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-tmp-dir\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.577941 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.577884 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh82t\" (UniqueName: \"kubernetes.io/projected/607c7ab6-765b-494e-b2b5-e40f0a16e956-kube-api-access-kh82t\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.578278 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.578258 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-tmp-dir\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.581026 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.580446 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-dshm\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.582000 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.581978 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607c7ab6-765b-494e-b2b5-e40f0a16e956-tls-certs\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.598746 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.598715 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh82t\" (UniqueName: \"kubernetes.io/projected/607c7ab6-765b-494e-b2b5-e40f0a16e956-kube-api-access-kh82t\") pod \"premium-w6-83e83e-kserve-5c8b9f9944-pmprw\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.671428 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.671333 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:05.974697 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:58:05.974662 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod607c7ab6_765b_494e_b2b5_e40f0a16e956.slice/crio-13ec8cb8ce962deb5b2bef6a64a9e17f474ea7bfc7237663927f1cc3644ec0d7 WatchSource:0}: Error finding container 13ec8cb8ce962deb5b2bef6a64a9e17f474ea7bfc7237663927f1cc3644ec0d7: Status 404 returned error can't find the container with id 13ec8cb8ce962deb5b2bef6a64a9e17f474ea7bfc7237663927f1cc3644ec0d7 Oct 30 04:58:05.975292 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:05.975264 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw"] Oct 30 04:58:06.431785 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:06.431751 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" event={"ID":"607c7ab6-765b-494e-b2b5-e40f0a16e956","Type":"ContainerStarted","Data":"45de0ce49422efb26c90e99737ef5e3804f30612950e14e7c4ecfaf7ce645f21"} Oct 30 04:58:06.431785 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:06.431790 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" event={"ID":"607c7ab6-765b-494e-b2b5-e40f0a16e956","Type":"ContainerStarted","Data":"13ec8cb8ce962deb5b2bef6a64a9e17f474ea7bfc7237663927f1cc3644ec0d7"} Oct 30 04:58:06.472765 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:06.472717 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podStartSLOduration=1.4727023240000001 podStartE2EDuration="1.472702324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:06.470857941 +0000 UTC m=+810.998624726" watchObservedRunningTime="2026-10-30 04:58:06.472702324 +0000 UTC m=+811.000469109" Oct 30 04:58:06.671940 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:06.671901 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:15.672382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:15.672337 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:15.685500 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:15.685470 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:16.520523 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:16.520491 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:58:21.503195 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.503163 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl"] Oct 30 04:58:21.507539 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.507519 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.523339 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.523303 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b21507-openshift-default-dockercfg-q4ptg\"" Oct 30 04:58:21.540409 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.540377 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl"] Oct 30 04:58:21.574080 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.574034 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.574247 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.574159 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxrm4\" (UniqueName: \"kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-kube-api-access-hxrm4\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.574247 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.574185 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.574247 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.574208 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.574384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.574261 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.574384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.574352 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.574474 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.574439 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.574565 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.574542 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.574629 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.574577 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.675644 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.675606 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.675860 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.675755 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.675860 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.675787 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.675980 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.675870 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.675980 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.675924 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrm4\" (UniqueName: \"kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-kube-api-access-hxrm4\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.675980 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.675943 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.675980 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.675972 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.676257 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.676016 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.676257 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.676068 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.676400 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.676290 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.676471 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.676400 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.676528 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.676480 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.676653 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.676634 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.676936 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.676912 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.678483 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.678461 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.678645 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.678631 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.696044 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.696021 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.696153 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.696136 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxrm4\" (UniqueName: \"kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-kube-api-access-hxrm4\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.821915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.821876 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:21.886608 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.883613 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8"] Oct 30 04:58:21.892660 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.892640 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:21.906675 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.906648 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8"] Oct 30 04:58:21.910525 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.910504 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-0b21ba-openshift-default-dockercfg-5n9kc\"" Oct 30 04:58:21.983495 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.983467 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:21.983649 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.983514 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h6jz\" (UniqueName: \"kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-kube-api-access-6h6jz\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:21.983649 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.983599 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:21.983649 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.983631 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:21.983756 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.983652 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:21.983826 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.983803 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:21.983861 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.983846 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:21.983894 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.983867 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:21.983946 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:21.983935 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.085314 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.085215 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.085314 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.085278 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.085557 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.085339 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.085557 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.085403 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.085557 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.085483 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.085557 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.085527 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h6jz\" (UniqueName: \"kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-kube-api-access-6h6jz\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.085763 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.085582 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.085763 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.085621 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.085763 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.085658 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.085763 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.085669 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.085763 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.085733 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.086011 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.085891 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.086069 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.086038 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.086236 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.086218 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.087603 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.087584 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.088005 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.087982 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.105565 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.105540 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h6jz\" (UniqueName: \"kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-kube-api-access-6h6jz\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.105661 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.105582 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.129431 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:58:22.129403 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9f1cd1_5271_4630_b240_27b4cf22b9a8.slice/crio-1d37eb7c051aad34ea5c9435d3fe7fb87986f33bd46a944f75f5c89618593890 WatchSource:0}: Error finding container 1d37eb7c051aad34ea5c9435d3fe7fb87986f33bd46a944f75f5c89618593890: Status 404 returned error can't find the container with id 1d37eb7c051aad34ea5c9435d3fe7fb87986f33bd46a944f75f5c89618593890 Oct 30 04:58:22.130236 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.130217 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl"] Oct 30 04:58:22.132238 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.132201 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:22.132318 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.132282 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:22.132379 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.132324 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:22.212974 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.212950 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:22.533954 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.533925 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8"] Oct 30 04:58:22.536453 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:58:22.536412 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e81da2_cc1a_4b99_9a85_ee73d843b8ad.slice/crio-b801ecf3309d16f21201e9d1bea251dcef14cb3211ef85b2004e3de842cc4066 WatchSource:0}: Error finding container b801ecf3309d16f21201e9d1bea251dcef14cb3211ef85b2004e3de842cc4066: Status 404 returned error can't find the container with id b801ecf3309d16f21201e9d1bea251dcef14cb3211ef85b2004e3de842cc4066 Oct 30 04:58:22.539687 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.539648 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:22.539808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.539731 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:22.539808 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.539772 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:22.563851 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.563821 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" event={"ID":"7d9f1cd1-5271-4630-b240-27b4cf22b9a8","Type":"ContainerStarted","Data":"ded0e003c4ae947b4f371f58fb875a6980f402d6178c554c114531c735a5d8fb"} Oct 30 04:58:22.563967 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.563878 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" event={"ID":"7d9f1cd1-5271-4630-b240-27b4cf22b9a8","Type":"ContainerStarted","Data":"1d37eb7c051aad34ea5c9435d3fe7fb87986f33bd46a944f75f5c89618593890"} Oct 30 04:58:22.565199 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.565171 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" event={"ID":"14e81da2-cc1a-4b99-9a85-ee73d843b8ad","Type":"ContainerStarted","Data":"b801ecf3309d16f21201e9d1bea251dcef14cb3211ef85b2004e3de842cc4066"} Oct 30 04:58:22.610424 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.610352 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" podStartSLOduration=1.610340299 podStartE2EDuration="1.610340299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:22.609054718 +0000 UTC m=+827.136821504" watchObservedRunningTime="2026-10-30 04:58:22.610340299 +0000 UTC m=+827.138107079" Oct 30 04:58:22.832388 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:22.827455 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:23.601772 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:23.601742 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" event={"ID":"14e81da2-cc1a-4b99-9a85-ee73d843b8ad","Type":"ContainerStarted","Data":"c3d8715d05837259ecd239198bac5a25f074df2968188eb811e82ad8d2fc37ef"} Oct 30 04:58:23.682848 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:23.682782 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" podStartSLOduration=2.682760855 podStartE2EDuration="2.682760855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:23.658351776 +0000 UTC m=+828.186118563" watchObservedRunningTime="2026-10-30 04:58:23.682760855 +0000 UTC m=+828.210527642" Oct 30 04:58:23.827384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:23.824476 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 04:58:23.827384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:23.824540 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" podUID="7d9f1cd1-5271-4630-b240-27b4cf22b9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 04:58:24.046563 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.034347 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94"] Oct 30 04:58:24.054861 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.054832 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" Oct 30 04:58:24.071383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.070760 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94"] Oct 30 04:58:24.106559 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.101041 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:58:24.106559 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.101181 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 04:58:24.106559 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.101234 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:58:24.106559 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.101305 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:58:24.177401 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.177189 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08e188d6-9mp94\" (UID: \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" Oct 30 04:58:24.177401 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.177320 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjpxr\" (UniqueName: \"kubernetes.io/projected/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-kube-api-access-wjpxr\") pod \"maas-api-revoke-sub-08e188d6-9mp94\" (UID: \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" Oct 30 04:58:24.182679 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.177768 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08e188d6-9mp94\" (UID: \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" Oct 30 04:58:24.237392 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.215193 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:24.247386 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.238088 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 04:58:24.247724 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.247628 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" podUID="14e81da2-cc1a-4b99-9a85-ee73d843b8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 04:58:24.284942 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.284909 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08e188d6-9mp94\" (UID: \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" Oct 30 04:58:24.286435 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.286409 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjpxr\" (UniqueName: \"kubernetes.io/projected/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-kube-api-access-wjpxr\") pod \"maas-api-revoke-sub-08e188d6-9mp94\" (UID: \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" Oct 30 04:58:24.286649 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.286633 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08e188d6-9mp94\" (UID: \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" Oct 30 04:58:24.287163 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.286255 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08e188d6-9mp94\" (UID: \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" Oct 30 04:58:24.295619 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.292979 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08e188d6-9mp94\" (UID: \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" Oct 30 04:58:24.311509 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.311484 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjpxr\" (UniqueName: \"kubernetes.io/projected/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-kube-api-access-wjpxr\") pod \"maas-api-revoke-sub-08e188d6-9mp94\" (UID: \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" Oct 30 04:58:24.509381 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.508562 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" Oct 30 04:58:24.835773 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.835735 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 04:58:24.868537 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:24.868484 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" podUID="7d9f1cd1-5271-4630-b240-27b4cf22b9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 04:58:25.013335 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:25.013284 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94"] Oct 30 04:58:25.237382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:25.236735 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 04:58:25.237382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:25.236796 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" podUID="14e81da2-cc1a-4b99-9a85-ee73d843b8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 04:58:25.670384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:25.667547 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" event={"ID":"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad","Type":"ContainerStarted","Data":"0a3d286124acd66781b2232486429b56c933c9551b7e2b759505d9d10ac57cc8"} Oct 30 04:58:25.822401 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:25.822342 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 04:58:25.822648 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:25.822618 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" podUID="7d9f1cd1-5271-4630-b240-27b4cf22b9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 04:58:26.007507 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:58:26.007421 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:58:26.233461 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:26.233419 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 04:58:26.233760 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:26.233712 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" podUID="14e81da2-cc1a-4b99-9a85-ee73d843b8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 04:58:26.831905 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:26.831468 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 04:58:26.831905 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:26.831529 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" podUID="7d9f1cd1-5271-4630-b240-27b4cf22b9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 04:58:27.213582 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:27.213507 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 04:58:27.213582 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:27.213558 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" podUID="14e81da2-cc1a-4b99-9a85-ee73d843b8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 04:58:27.823228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:27.822792 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 04:58:27.823228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:27.822853 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" podUID="7d9f1cd1-5271-4630-b240-27b4cf22b9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 04:58:28.214011 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:28.213895 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 04:58:28.214011 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:28.213948 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" podUID="14e81da2-cc1a-4b99-9a85-ee73d843b8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 04:58:28.822866 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:28.822820 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 04:58:28.823053 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:28.822895 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" podUID="7d9f1cd1-5271-4630-b240-27b4cf22b9a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 04:58:29.215972 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:29.215603 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 04:58:29.215972 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:29.215657 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" podUID="14e81da2-cc1a-4b99-9a85-ee73d843b8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 04:58:30.668559 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:30.668448 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:30.669083 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:30.668903 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:30.670181 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:30.670154 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 04:58:30.733185 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:30.732047 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" event={"ID":"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad","Type":"ContainerStarted","Data":"4cc2fc67f9ab0a457e6d6ecb6a2bd7dfb4803da88b4a3c5806435235bc1e91d9"} Oct 30 04:58:30.812721 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:30.812338 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" podStartSLOduration=2.190699429 podStartE2EDuration="6.812317657s" totalImagesPullingTime="4.621618228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:30.810232049 +0000 UTC m=+835.337998835" watchObservedRunningTime="2026-10-30 04:58:30.812317657 +0000 UTC m=+835.340084444" Oct 30 04:58:30.981380 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:30.980682 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:30.981947 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:30.981857 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:30.983706 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:30.983674 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 04:58:31.738344 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:31.738309 6587 generic.go:356] "Generic (PLEG): container finished" podID="f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad" containerID="4cc2fc67f9ab0a457e6d6ecb6a2bd7dfb4803da88b4a3c5806435235bc1e91d9" exitCode=0 Oct 30 04:58:31.738787 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:31.738395 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" event={"ID":"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad","Type":"ContainerDied","Data":"4cc2fc67f9ab0a457e6d6ecb6a2bd7dfb4803da88b4a3c5806435235bc1e91d9"} Oct 30 04:58:32.923972 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:32.923946 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" Oct 30 04:58:32.984082 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:32.984057 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-openshift-ca\") pod \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\" (UID: \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\") " Oct 30 04:58:32.984232 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:32.984125 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-xks-ca\") pod \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\" (UID: \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\") " Oct 30 04:58:32.984232 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:32.984231 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-kube-api-access-wjpxr\" (UniqueName: \"kubernetes.io/projected/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-kube-api-access-wjpxr\") pod \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\" (UID: \"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad\") " Oct 30 04:58:32.984527 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:32.984493 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-openshift-ca" pod "f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad" (UID: "f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:58:32.984723 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:32.984702 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:32.986406 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:32.986341 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-xks-ca" pod "f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad" (UID: "f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:58:32.986406 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:32.986395 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-kube-api-access-wjpxr" pod "f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad" (UID: "f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad"). InnerVolumeSpecName "kube-api-access-wjpxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:58:33.086222 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:33.086180 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjpxr\" (UniqueName: \"kubernetes.io/projected/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-kube-api-access-wjpxr\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:33.086222 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:33.086222 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:33.752714 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:33.752678 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" event={"ID":"f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad","Type":"ContainerDied","Data":"0a3d286124acd66781b2232486429b56c933c9551b7e2b759505d9d10ac57cc8"} Oct 30 04:58:33.752944 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:33.752705 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94" Oct 30 04:58:33.754294 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:33.753980 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a3d286124acd66781b2232486429b56c933c9551b7e2b759505d9d10ac57cc8" Oct 30 04:58:34.983163 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:34.983129 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m"] Oct 30 04:58:34.983939 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:34.983916 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad" containerName="revoke-keys" Oct 30 04:58:34.984030 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:34.983941 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad" containerName="revoke-keys" Oct 30 04:58:34.984094 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:34.984064 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad" containerName="revoke-keys" Oct 30 04:58:34.987630 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:34.987609 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" Oct 30 04:58:35.003791 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.003761 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m"] Oct 30 04:58:35.019401 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.013884 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:58:35.019401 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.014054 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:58:35.019401 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.014051 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:58:35.019401 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.017483 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 04:58:35.063872 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.063826 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b7d98dd-wsj4m\" (UID: \"dae14d57-3dea-4327-a1de-28239d35fae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" Oct 30 04:58:35.063872 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.063865 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzn6h\" (UniqueName: \"kubernetes.io/projected/dae14d57-3dea-4327-a1de-28239d35fae6-kube-api-access-kzn6h\") pod \"maas-api-revoke-sub-4b7d98dd-wsj4m\" (UID: \"dae14d57-3dea-4327-a1de-28239d35fae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" Oct 30 04:58:35.064115 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.063896 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b7d98dd-wsj4m\" (UID: \"dae14d57-3dea-4327-a1de-28239d35fae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" Oct 30 04:58:35.165720 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.165677 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b7d98dd-wsj4m\" (UID: \"dae14d57-3dea-4327-a1de-28239d35fae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" Oct 30 04:58:35.165916 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.165750 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzn6h\" (UniqueName: \"kubernetes.io/projected/dae14d57-3dea-4327-a1de-28239d35fae6-kube-api-access-kzn6h\") pod \"maas-api-revoke-sub-4b7d98dd-wsj4m\" (UID: \"dae14d57-3dea-4327-a1de-28239d35fae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" Oct 30 04:58:35.165916 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.165817 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b7d98dd-wsj4m\" (UID: \"dae14d57-3dea-4327-a1de-28239d35fae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" Oct 30 04:58:35.166902 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.166827 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b7d98dd-wsj4m\" (UID: \"dae14d57-3dea-4327-a1de-28239d35fae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" Oct 30 04:58:35.169506 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.169482 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b7d98dd-wsj4m\" (UID: \"dae14d57-3dea-4327-a1de-28239d35fae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" Oct 30 04:58:35.186876 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.186852 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzn6h\" (UniqueName: \"kubernetes.io/projected/dae14d57-3dea-4327-a1de-28239d35fae6-kube-api-access-kzn6h\") pod \"maas-api-revoke-sub-4b7d98dd-wsj4m\" (UID: \"dae14d57-3dea-4327-a1de-28239d35fae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" Oct 30 04:58:35.301106 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.301075 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" Oct 30 04:58:35.608535 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.608507 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m"] Oct 30 04:58:35.609317 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:58:35.609283 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae14d57_3dea_4327_a1de_28239d35fae6.slice/crio-d46e964a66a2b3127f12b63796a03dd071c2b890554e4764a27254b95e4c72d6 WatchSource:0}: Error finding container d46e964a66a2b3127f12b63796a03dd071c2b890554e4764a27254b95e4c72d6: Status 404 returned error can't find the container with id d46e964a66a2b3127f12b63796a03dd071c2b890554e4764a27254b95e4c72d6 Oct 30 04:58:35.775417 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:35.775345 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" event={"ID":"dae14d57-3dea-4327-a1de-28239d35fae6","Type":"ContainerStarted","Data":"d46e964a66a2b3127f12b63796a03dd071c2b890554e4764a27254b95e4c72d6"} Oct 30 04:58:36.627290 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.627258 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml"] Oct 30 04:58:36.631567 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.631543 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.652906 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.652868 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-889ca9-openshift-default-dockercfg-t56wr\"" Oct 30 04:58:36.665782 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.665756 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml"] Oct 30 04:58:36.784321 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.784285 6587 generic.go:356] "Generic (PLEG): container finished" podID="dae14d57-3dea-4327-a1de-28239d35fae6" containerID="5751eb81aadc2d205b241ef9008f869e0646d227790cfe03b37b1085475e2bff" exitCode=0 Oct 30 04:58:36.784542 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.784343 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" event={"ID":"dae14d57-3dea-4327-a1de-28239d35fae6","Type":"ContainerDied","Data":"5751eb81aadc2d205b241ef9008f869e0646d227790cfe03b37b1085475e2bff"} Oct 30 04:58:36.786018 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.785988 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.786149 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.786045 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.786223 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.786193 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.786282 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.786247 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e011432d-3991-4d45-9926-cf7ca5e50194-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.786385 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.786366 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.786469 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.786456 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.786528 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.786515 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsv9c\" (UniqueName: \"kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-kube-api-access-tsv9c\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.786610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.786587 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e011432d-3991-4d45-9926-cf7ca5e50194-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.786678 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.786662 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.887612 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.887519 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e011432d-3991-4d45-9926-cf7ca5e50194-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.887612 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.887578 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.887800 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.887652 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.887800 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.887707 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsv9c\" (UniqueName: \"kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-kube-api-access-tsv9c\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.887800 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.887728 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e011432d-3991-4d45-9926-cf7ca5e50194-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.887800 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.887757 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.887800 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.887788 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.887965 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.887814 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.887965 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.887873 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.888307 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.888278 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.888585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.888397 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.888733 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.888428 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.888733 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.888680 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.889256 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.889228 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e011432d-3991-4d45-9926-cf7ca5e50194-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.891056 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.891027 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.891532 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.891513 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e011432d-3991-4d45-9926-cf7ca5e50194-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.908236 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.908211 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.908812 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.908791 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsv9c\" (UniqueName: \"kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-kube-api-access-tsv9c\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:36.945989 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:36.945948 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:37.282503 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:58:37.282468 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode011432d_3991_4d45_9926_cf7ca5e50194.slice/crio-bebf2841b533508688bdce7ed09547a947bf1673c9f17bff844407d9ae0a45dd WatchSource:0}: Error finding container bebf2841b533508688bdce7ed09547a947bf1673c9f17bff844407d9ae0a45dd: Status 404 returned error can't find the container with id bebf2841b533508688bdce7ed09547a947bf1673c9f17bff844407d9ae0a45dd Oct 30 04:58:37.282806 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:37.282780 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml"] Oct 30 04:58:37.285952 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:37.285920 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:37.286050 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:37.285984 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:37.286050 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:37.286033 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:58:37.795230 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:37.795184 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" event={"ID":"e011432d-3991-4d45-9926-cf7ca5e50194","Type":"ContainerStarted","Data":"97a9040b91f5f71d100b7fdac094b9780f69b83988b8f33d385979dd28b36894"} Oct 30 04:58:37.795752 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:37.795251 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" event={"ID":"e011432d-3991-4d45-9926-cf7ca5e50194","Type":"ContainerStarted","Data":"bebf2841b533508688bdce7ed09547a947bf1673c9f17bff844407d9ae0a45dd"} Oct 30 04:58:37.846509 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:37.838706 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" podStartSLOduration=1.8386858579999998 podStartE2EDuration="1.838685858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:37.83686389 +0000 UTC m=+842.364630681" watchObservedRunningTime="2026-10-30 04:58:37.838685858 +0000 UTC m=+842.366452644" Oct 30 04:58:37.955551 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:37.955510 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:38.002870 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.002839 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" Oct 30 04:58:38.135382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.123527 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae14d57-3dea-4327-a1de-28239d35fae6-kube-api-access-kzn6h\" (UniqueName: \"kubernetes.io/projected/dae14d57-3dea-4327-a1de-28239d35fae6-kube-api-access-kzn6h\") pod \"dae14d57-3dea-4327-a1de-28239d35fae6\" (UID: \"dae14d57-3dea-4327-a1de-28239d35fae6\") " Oct 30 04:58:38.135382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.123725 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-xks-ca\") pod \"dae14d57-3dea-4327-a1de-28239d35fae6\" (UID: \"dae14d57-3dea-4327-a1de-28239d35fae6\") " Oct 30 04:58:38.135382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.123851 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-openshift-ca\") pod \"dae14d57-3dea-4327-a1de-28239d35fae6\" (UID: \"dae14d57-3dea-4327-a1de-28239d35fae6\") " Oct 30 04:58:38.150382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.147899 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-openshift-ca" pod "dae14d57-3dea-4327-a1de-28239d35fae6" (UID: "dae14d57-3dea-4327-a1de-28239d35fae6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:58:38.153423 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.151720 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-xks-ca" pod "dae14d57-3dea-4327-a1de-28239d35fae6" (UID: "dae14d57-3dea-4327-a1de-28239d35fae6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:58:38.153760 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.153721 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae14d57-3dea-4327-a1de-28239d35fae6-kube-api-access-kzn6h" pod "dae14d57-3dea-4327-a1de-28239d35fae6" (UID: "dae14d57-3dea-4327-a1de-28239d35fae6"). InnerVolumeSpecName "kube-api-access-kzn6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:58:38.252297 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.252206 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:38.252297 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.252250 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzn6h\" (UniqueName: \"kubernetes.io/projected/dae14d57-3dea-4327-a1de-28239d35fae6-kube-api-access-kzn6h\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:38.252297 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.252266 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae14d57-3dea-4327-a1de-28239d35fae6-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:38.507480 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.505219 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r"] Oct 30 04:58:38.507480 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.506415 6587 cpu_manager.go:414] "containerMap: removing container" podUID="dae14d57-3dea-4327-a1de-28239d35fae6" containerName="revoke-keys" Oct 30 04:58:38.507480 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.506439 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae14d57-3dea-4327-a1de-28239d35fae6" containerName="revoke-keys" Oct 30 04:58:38.507480 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.506758 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae14d57-3dea-4327-a1de-28239d35fae6" containerName="revoke-keys" Oct 30 04:58:38.513739 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.513717 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" Oct 30 04:58:38.559324 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.559275 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r"] Oct 30 04:58:38.658059 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.658017 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl8bs\" (UniqueName: \"kubernetes.io/projected/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-kube-api-access-zl8bs\") pod \"maas-api-revoke-sub-c5000d16-h458r\" (UID: \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" Oct 30 04:58:38.658541 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.658514 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5000d16-h458r\" (UID: \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" Oct 30 04:58:38.658762 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.658743 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5000d16-h458r\" (UID: \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" Oct 30 04:58:38.760996 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.760913 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5000d16-h458r\" (UID: \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" Oct 30 04:58:38.760996 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.760968 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5000d16-h458r\" (UID: \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" Oct 30 04:58:38.761221 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.761027 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5000d16-h458r\" (UID: \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" Oct 30 04:58:38.761460 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.761437 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl8bs\" (UniqueName: \"kubernetes.io/projected/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-kube-api-access-zl8bs\") pod \"maas-api-revoke-sub-c5000d16-h458r\" (UID: \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" Oct 30 04:58:38.764188 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.764164 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5000d16-h458r\" (UID: \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" Oct 30 04:58:38.781959 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.781743 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl8bs\" (UniqueName: \"kubernetes.io/projected/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-kube-api-access-zl8bs\") pod \"maas-api-revoke-sub-c5000d16-h458r\" (UID: \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" Oct 30 04:58:38.802085 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.802061 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" Oct 30 04:58:38.802507 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.802088 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m" event={"ID":"dae14d57-3dea-4327-a1de-28239d35fae6","Type":"ContainerDied","Data":"d46e964a66a2b3127f12b63796a03dd071c2b890554e4764a27254b95e4c72d6"} Oct 30 04:58:38.802507 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.802145 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d46e964a66a2b3127f12b63796a03dd071c2b890554e4764a27254b95e4c72d6" Oct 30 04:58:38.859787 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.859753 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" Oct 30 04:58:38.947906 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.947872 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 04:58:38.948065 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:38.947933 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" podUID="e011432d-3991-4d45-9926-cf7ca5e50194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 04:58:39.143395 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.143315 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr"] Oct 30 04:58:39.150725 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.150691 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" Oct 30 04:58:39.153942 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.153910 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr"] Oct 30 04:58:39.210152 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.198445 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r"] Oct 30 04:58:39.210152 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:58:39.200257 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3193d88_8b5c_4fad_9873_aca7e7ec72ab.slice/crio-03b421f4482d80622ae3306e9445c140c497558695dccda632544bc9f5f973bb WatchSource:0}: Error finding container 03b421f4482d80622ae3306e9445c140c497558695dccda632544bc9f5f973bb: Status 404 returned error can't find the container with id 03b421f4482d80622ae3306e9445c140c497558695dccda632544bc9f5f973bb Oct 30 04:58:39.272395 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.272338 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn9pt\" (UniqueName: \"kubernetes.io/projected/1968b9e2-7abc-44c6-991a-a3f7e64853d6-kube-api-access-nn9pt\") pod \"maas-api-revoke-sub-2c980cf8-kjwnr\" (UID: \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" Oct 30 04:58:39.272500 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.272432 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c980cf8-kjwnr\" (UID: \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" Oct 30 04:58:39.272622 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.272599 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c980cf8-kjwnr\" (UID: \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" Oct 30 04:58:39.373738 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.373703 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn9pt\" (UniqueName: \"kubernetes.io/projected/1968b9e2-7abc-44c6-991a-a3f7e64853d6-kube-api-access-nn9pt\") pod \"maas-api-revoke-sub-2c980cf8-kjwnr\" (UID: \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" Oct 30 04:58:39.373929 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.373777 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c980cf8-kjwnr\" (UID: \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" Oct 30 04:58:39.373929 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.373897 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c980cf8-kjwnr\" (UID: \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" Oct 30 04:58:39.374707 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.374684 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c980cf8-kjwnr\" (UID: \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" Oct 30 04:58:39.431426 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.423901 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c980cf8-kjwnr\" (UID: \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" Oct 30 04:58:39.449083 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.449051 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn9pt\" (UniqueName: \"kubernetes.io/projected/1968b9e2-7abc-44c6-991a-a3f7e64853d6-kube-api-access-nn9pt\") pod \"maas-api-revoke-sub-2c980cf8-kjwnr\" (UID: \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" Oct 30 04:58:39.472177 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.472151 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" Oct 30 04:58:39.794396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.794366 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr"] Oct 30 04:58:39.794669 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:58:39.794649 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1968b9e2_7abc_44c6_991a_a3f7e64853d6.slice/crio-2f46573ef89d596e9c22db6077ddbcfb354cd71583f76c62702e40ced6d6eda0 WatchSource:0}: Error finding container 2f46573ef89d596e9c22db6077ddbcfb354cd71583f76c62702e40ced6d6eda0: Status 404 returned error can't find the container with id 2f46573ef89d596e9c22db6077ddbcfb354cd71583f76c62702e40ced6d6eda0 Oct 30 04:58:39.809074 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.809044 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" event={"ID":"c3193d88-8b5c-4fad-9873-aca7e7ec72ab","Type":"ContainerStarted","Data":"5fc207671721f7f4a5971dcb317eaf6c38f33edc6faccbce02d8c446e645c120"} Oct 30 04:58:39.809507 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.809095 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" event={"ID":"c3193d88-8b5c-4fad-9873-aca7e7ec72ab","Type":"ContainerStarted","Data":"03b421f4482d80622ae3306e9445c140c497558695dccda632544bc9f5f973bb"} Oct 30 04:58:39.810530 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.810510 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" event={"ID":"1968b9e2-7abc-44c6-991a-a3f7e64853d6","Type":"ContainerStarted","Data":"2f46573ef89d596e9c22db6077ddbcfb354cd71583f76c62702e40ced6d6eda0"} Oct 30 04:58:39.851147 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.850325 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" podStartSLOduration=1.850308811 podStartE2EDuration="1.850308811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:39.850155437 +0000 UTC m=+844.377922223" watchObservedRunningTime="2026-10-30 04:58:39.850308811 +0000 UTC m=+844.378075600" Oct 30 04:58:39.947172 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.947095 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 04:58:39.947172 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:39.947155 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" podUID="e011432d-3991-4d45-9926-cf7ca5e50194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 04:58:40.844383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:40.843536 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" event={"ID":"1968b9e2-7abc-44c6-991a-a3f7e64853d6","Type":"ContainerStarted","Data":"05615386e783cd9b03d1c2d807dcb63550edb2dacd88559cc646ea7c759616b7"} Oct 30 04:58:40.846553 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:40.846521 6587 generic.go:356] "Generic (PLEG): container finished" podID="c3193d88-8b5c-4fad-9873-aca7e7ec72ab" containerID="5fc207671721f7f4a5971dcb317eaf6c38f33edc6faccbce02d8c446e645c120" exitCode=0 Oct 30 04:58:40.846688 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:40.846627 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" event={"ID":"c3193d88-8b5c-4fad-9873-aca7e7ec72ab","Type":"ContainerDied","Data":"5fc207671721f7f4a5971dcb317eaf6c38f33edc6faccbce02d8c446e645c120"} Oct 30 04:58:40.887293 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:40.887239 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" podStartSLOduration=1.887223716 podStartE2EDuration="1.887223716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:40.884504762 +0000 UTC m=+845.412271549" watchObservedRunningTime="2026-10-30 04:58:40.887223716 +0000 UTC m=+845.414990501" Oct 30 04:58:40.952433 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:40.952179 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 04:58:40.952433 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:40.952237 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" podUID="e011432d-3991-4d45-9926-cf7ca5e50194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 04:58:41.856691 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:41.856654 6587 generic.go:356] "Generic (PLEG): container finished" podID="1968b9e2-7abc-44c6-991a-a3f7e64853d6" containerID="05615386e783cd9b03d1c2d807dcb63550edb2dacd88559cc646ea7c759616b7" exitCode=0 Oct 30 04:58:41.857172 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:41.856744 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" event={"ID":"1968b9e2-7abc-44c6-991a-a3f7e64853d6","Type":"ContainerDied","Data":"05615386e783cd9b03d1c2d807dcb63550edb2dacd88559cc646ea7c759616b7"} Oct 30 04:58:41.958802 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:41.949106 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 04:58:41.958802 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:41.949164 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" podUID="e011432d-3991-4d45-9926-cf7ca5e50194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 04:58:42.113550 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.113482 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" Oct 30 04:58:42.250381 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.241130 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-openshift-ca\") pod \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\" (UID: \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\") " Oct 30 04:58:42.250381 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.241192 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-kube-api-access-zl8bs\" (UniqueName: \"kubernetes.io/projected/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-kube-api-access-zl8bs\") pod \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\" (UID: \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\") " Oct 30 04:58:42.250381 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.241305 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-xks-ca\") pod \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\" (UID: \"c3193d88-8b5c-4fad-9873-aca7e7ec72ab\") " Oct 30 04:58:42.261380 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.259762 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-openshift-ca" pod "c3193d88-8b5c-4fad-9873-aca7e7ec72ab" (UID: "c3193d88-8b5c-4fad-9873-aca7e7ec72ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:58:42.277535 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.277493 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-kube-api-access-zl8bs" pod "c3193d88-8b5c-4fad-9873-aca7e7ec72ab" (UID: "c3193d88-8b5c-4fad-9873-aca7e7ec72ab"). InnerVolumeSpecName "kube-api-access-zl8bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:58:42.284165 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.284134 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-xks-ca" pod "c3193d88-8b5c-4fad-9873-aca7e7ec72ab" (UID: "c3193d88-8b5c-4fad-9873-aca7e7ec72ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:58:42.350415 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.343194 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:42.350415 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.343235 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl8bs\" (UniqueName: \"kubernetes.io/projected/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-kube-api-access-zl8bs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:42.350415 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.343252 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3193d88-8b5c-4fad-9873-aca7e7ec72ab-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:42.641380 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.640858 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm"] Oct 30 04:58:42.642387 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.641767 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c3193d88-8b5c-4fad-9873-aca7e7ec72ab" containerName="revoke-keys" Oct 30 04:58:42.642387 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.641792 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3193d88-8b5c-4fad-9873-aca7e7ec72ab" containerName="revoke-keys" Oct 30 04:58:42.642387 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.641940 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3193d88-8b5c-4fad-9873-aca7e7ec72ab" containerName="revoke-keys" Oct 30 04:58:42.650742 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.650708 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" Oct 30 04:58:42.659999 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.659439 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm"] Oct 30 04:58:42.748665 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.748617 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7486d1ca-7clsm\" (UID: \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" Oct 30 04:58:42.748816 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.748748 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpb29\" (UniqueName: \"kubernetes.io/projected/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-kube-api-access-xpb29\") pod \"maas-api-revoke-sub-7486d1ca-7clsm\" (UID: \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" Oct 30 04:58:42.748816 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.748799 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7486d1ca-7clsm\" (UID: \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" Oct 30 04:58:42.849654 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.849614 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7486d1ca-7clsm\" (UID: \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" Oct 30 04:58:42.849843 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.849683 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpb29\" (UniqueName: \"kubernetes.io/projected/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-kube-api-access-xpb29\") pod \"maas-api-revoke-sub-7486d1ca-7clsm\" (UID: \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" Oct 30 04:58:42.849843 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.849723 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7486d1ca-7clsm\" (UID: \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" Oct 30 04:58:42.850505 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.850480 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7486d1ca-7clsm\" (UID: \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" Oct 30 04:58:42.854940 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.854910 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7486d1ca-7clsm\" (UID: \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" Oct 30 04:58:42.867735 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.867710 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" Oct 30 04:58:42.868075 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.867733 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r" event={"ID":"c3193d88-8b5c-4fad-9873-aca7e7ec72ab","Type":"ContainerDied","Data":"03b421f4482d80622ae3306e9445c140c497558695dccda632544bc9f5f973bb"} Oct 30 04:58:42.868075 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.867790 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03b421f4482d80622ae3306e9445c140c497558695dccda632544bc9f5f973bb" Oct 30 04:58:42.870125 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.870099 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpb29\" (UniqueName: \"kubernetes.io/projected/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-kube-api-access-xpb29\") pod \"maas-api-revoke-sub-7486d1ca-7clsm\" (UID: \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" Oct 30 04:58:42.949442 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.949032 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 04:58:42.949442 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.949117 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" podUID="e011432d-3991-4d45-9926-cf7ca5e50194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 04:58:42.969197 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:42.969170 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" Oct 30 04:58:43.071682 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.071656 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" Oct 30 04:58:43.170395 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.161694 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-openshift-ca\") pod \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\" (UID: \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\") " Oct 30 04:58:43.170395 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.161783 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-xks-ca\") pod \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\" (UID: \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\") " Oct 30 04:58:43.170395 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.161877 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1968b9e2-7abc-44c6-991a-a3f7e64853d6-kube-api-access-nn9pt\" (UniqueName: \"kubernetes.io/projected/1968b9e2-7abc-44c6-991a-a3f7e64853d6-kube-api-access-nn9pt\") pod \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\" (UID: \"1968b9e2-7abc-44c6-991a-a3f7e64853d6\") " Oct 30 04:58:43.170775 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.170725 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-openshift-ca" pod "1968b9e2-7abc-44c6-991a-a3f7e64853d6" (UID: "1968b9e2-7abc-44c6-991a-a3f7e64853d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:58:43.171404 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.171381 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1968b9e2-7abc-44c6-991a-a3f7e64853d6-kube-api-access-nn9pt" pod "1968b9e2-7abc-44c6-991a-a3f7e64853d6" (UID: "1968b9e2-7abc-44c6-991a-a3f7e64853d6"). InnerVolumeSpecName "kube-api-access-nn9pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:58:43.176625 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.176588 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-xks-ca" pod "1968b9e2-7abc-44c6-991a-a3f7e64853d6" (UID: "1968b9e2-7abc-44c6-991a-a3f7e64853d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:58:43.264399 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.264292 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:43.264620 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.264601 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn9pt\" (UniqueName: \"kubernetes.io/projected/1968b9e2-7abc-44c6-991a-a3f7e64853d6-kube-api-access-nn9pt\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:43.264745 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.264732 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1968b9e2-7abc-44c6-991a-a3f7e64853d6-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:43.325812 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:58:43.324968 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc29a3f_d5e3_4b0b_bf3b_e7f83c5b32b5.slice/crio-f104a5d3378a966169faadc464f769555debedb63c33fd75e3780d98bc1ea68f WatchSource:0}: Error finding container f104a5d3378a966169faadc464f769555debedb63c33fd75e3780d98bc1ea68f: Status 404 returned error can't find the container with id f104a5d3378a966169faadc464f769555debedb63c33fd75e3780d98bc1ea68f Oct 30 04:58:43.326299 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.326266 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm"] Oct 30 04:58:43.885267 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.885237 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" Oct 30 04:58:43.885821 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.885283 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr" event={"ID":"1968b9e2-7abc-44c6-991a-a3f7e64853d6","Type":"ContainerDied","Data":"2f46573ef89d596e9c22db6077ddbcfb354cd71583f76c62702e40ced6d6eda0"} Oct 30 04:58:43.885821 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.885411 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f46573ef89d596e9c22db6077ddbcfb354cd71583f76c62702e40ced6d6eda0" Oct 30 04:58:43.887663 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.887593 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" event={"ID":"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5","Type":"ContainerStarted","Data":"bc3b8ddb058bab276836403a807e2e0d058cd6dff021c8edeef4fd9cbb9895d9"} Oct 30 04:58:43.887663 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.887647 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" event={"ID":"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5","Type":"ContainerStarted","Data":"f104a5d3378a966169faadc464f769555debedb63c33fd75e3780d98bc1ea68f"} Oct 30 04:58:43.930787 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:43.930731 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" podStartSLOduration=1.930716814 podStartE2EDuration="1.930716814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:43.929317007 +0000 UTC m=+848.457083786" watchObservedRunningTime="2026-10-30 04:58:43.930716814 +0000 UTC m=+848.458483599" Oct 30 04:58:44.042008 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:44.041974 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:44.042424 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:44.042405 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:44.043946 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:44.043922 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 04:58:44.898042 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:44.897996 6587 generic.go:356] "Generic (PLEG): container finished" podID="cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5" containerID="bc3b8ddb058bab276836403a807e2e0d058cd6dff021c8edeef4fd9cbb9895d9" exitCode=0 Oct 30 04:58:44.898558 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:44.898100 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" event={"ID":"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5","Type":"ContainerDied","Data":"bc3b8ddb058bab276836403a807e2e0d058cd6dff021c8edeef4fd9cbb9895d9"} Oct 30 04:58:46.181850 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.181788 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" Oct 30 04:58:46.243717 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.243681 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-xks-ca\") pod \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\" (UID: \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\") " Oct 30 04:58:46.244017 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.243993 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-openshift-ca\") pod \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\" (UID: \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\") " Oct 30 04:58:46.244407 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.244347 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-kube-api-access-xpb29\" (UniqueName: \"kubernetes.io/projected/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-kube-api-access-xpb29\") pod \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\" (UID: \"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5\") " Oct 30 04:58:46.253146 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.253122 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-openshift-ca" pod "cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5" (UID: "cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:58:46.262551 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.262521 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-xks-ca" pod "cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5" (UID: "cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:58:46.265566 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.265539 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-kube-api-access-xpb29" pod "cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5" (UID: "cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5"). InnerVolumeSpecName "kube-api-access-xpb29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:58:46.346294 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.346259 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpb29\" (UniqueName: \"kubernetes.io/projected/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-kube-api-access-xpb29\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:46.346294 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.346290 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:46.346294 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.346299 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:46.736864 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.736822 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz"] Oct 30 04:58:46.737524 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.737498 6587 cpu_manager.go:414] "containerMap: removing container" podUID="cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5" containerName="revoke-keys" Oct 30 04:58:46.737524 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.737521 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5" containerName="revoke-keys" Oct 30 04:58:46.737753 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.737571 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1968b9e2-7abc-44c6-991a-a3f7e64853d6" containerName="revoke-keys" Oct 30 04:58:46.737753 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.737579 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1968b9e2-7abc-44c6-991a-a3f7e64853d6" containerName="revoke-keys" Oct 30 04:58:46.737753 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.737707 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5" containerName="revoke-keys" Oct 30 04:58:46.737753 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.737725 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1968b9e2-7abc-44c6-991a-a3f7e64853d6" containerName="revoke-keys" Oct 30 04:58:46.741791 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.741766 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" Oct 30 04:58:46.786385 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.786134 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz"] Oct 30 04:58:46.852159 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.852121 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31ddb39e-x7bcz\" (UID: \"b25ac0aa-86c0-4e22-aff5-490a57f95397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" Oct 30 04:58:46.852159 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.852161 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31ddb39e-x7bcz\" (UID: \"b25ac0aa-86c0-4e22-aff5-490a57f95397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" Oct 30 04:58:46.852401 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.852189 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd2jq\" (UniqueName: \"kubernetes.io/projected/b25ac0aa-86c0-4e22-aff5-490a57f95397-kube-api-access-qd2jq\") pod \"maas-api-revoke-sub-31ddb39e-x7bcz\" (UID: \"b25ac0aa-86c0-4e22-aff5-490a57f95397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" Oct 30 04:58:46.934050 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.934012 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" event={"ID":"cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5","Type":"ContainerDied","Data":"f104a5d3378a966169faadc464f769555debedb63c33fd75e3780d98bc1ea68f"} Oct 30 04:58:46.934213 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.934078 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f104a5d3378a966169faadc464f769555debedb63c33fd75e3780d98bc1ea68f" Oct 30 04:58:46.934213 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.934029 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm" Oct 30 04:58:46.953415 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.953384 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd2jq\" (UniqueName: \"kubernetes.io/projected/b25ac0aa-86c0-4e22-aff5-490a57f95397-kube-api-access-qd2jq\") pod \"maas-api-revoke-sub-31ddb39e-x7bcz\" (UID: \"b25ac0aa-86c0-4e22-aff5-490a57f95397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" Oct 30 04:58:46.953637 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.953620 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31ddb39e-x7bcz\" (UID: \"b25ac0aa-86c0-4e22-aff5-490a57f95397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" Oct 30 04:58:46.953721 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.953654 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31ddb39e-x7bcz\" (UID: \"b25ac0aa-86c0-4e22-aff5-490a57f95397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" Oct 30 04:58:46.954275 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.954247 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31ddb39e-x7bcz\" (UID: \"b25ac0aa-86c0-4e22-aff5-490a57f95397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" Oct 30 04:58:46.956579 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.956554 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31ddb39e-x7bcz\" (UID: \"b25ac0aa-86c0-4e22-aff5-490a57f95397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" Oct 30 04:58:46.973221 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:46.973196 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd2jq\" (UniqueName: \"kubernetes.io/projected/b25ac0aa-86c0-4e22-aff5-490a57f95397-kube-api-access-qd2jq\") pod \"maas-api-revoke-sub-31ddb39e-x7bcz\" (UID: \"b25ac0aa-86c0-4e22-aff5-490a57f95397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" Oct 30 04:58:47.059511 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.059478 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" Oct 30 04:58:47.218770 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.218734 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt"] Oct 30 04:58:47.224789 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.224766 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" Oct 30 04:58:47.240556 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.240486 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt"] Oct 30 04:58:47.261974 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.261939 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a199ab49-bhzwt\" (UID: \"030b0117-0d9a-4370-8a1c-40bd335543c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" Oct 30 04:58:47.262118 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.262072 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a199ab49-bhzwt\" (UID: \"030b0117-0d9a-4370-8a1c-40bd335543c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" Oct 30 04:58:47.262189 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.262176 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbblw\" (UniqueName: \"kubernetes.io/projected/030b0117-0d9a-4370-8a1c-40bd335543c7-kube-api-access-gbblw\") pod \"maas-api-revoke-sub-a199ab49-bhzwt\" (UID: \"030b0117-0d9a-4370-8a1c-40bd335543c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" Oct 30 04:58:47.356286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.356188 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz"] Oct 30 04:58:47.361511 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:58:47.361482 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25ac0aa_86c0_4e22_aff5_490a57f95397.slice/crio-9d7c2c7bd7ba7393b0299cccd0147a1f8a61c4339338abc01cf9981fbc0cf990 WatchSource:0}: Error finding container 9d7c2c7bd7ba7393b0299cccd0147a1f8a61c4339338abc01cf9981fbc0cf990: Status 404 returned error can't find the container with id 9d7c2c7bd7ba7393b0299cccd0147a1f8a61c4339338abc01cf9981fbc0cf990 Oct 30 04:58:47.363095 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.363063 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a199ab49-bhzwt\" (UID: \"030b0117-0d9a-4370-8a1c-40bd335543c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" Oct 30 04:58:47.363209 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.363179 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbblw\" (UniqueName: \"kubernetes.io/projected/030b0117-0d9a-4370-8a1c-40bd335543c7-kube-api-access-gbblw\") pod \"maas-api-revoke-sub-a199ab49-bhzwt\" (UID: \"030b0117-0d9a-4370-8a1c-40bd335543c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" Oct 30 04:58:47.363281 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.363266 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a199ab49-bhzwt\" (UID: \"030b0117-0d9a-4370-8a1c-40bd335543c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" Oct 30 04:58:47.364058 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.364034 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a199ab49-bhzwt\" (UID: \"030b0117-0d9a-4370-8a1c-40bd335543c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" Oct 30 04:58:47.369632 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.366983 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a199ab49-bhzwt\" (UID: \"030b0117-0d9a-4370-8a1c-40bd335543c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" Oct 30 04:58:47.386069 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.386045 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbblw\" (UniqueName: \"kubernetes.io/projected/030b0117-0d9a-4370-8a1c-40bd335543c7-kube-api-access-gbblw\") pod \"maas-api-revoke-sub-a199ab49-bhzwt\" (UID: \"030b0117-0d9a-4370-8a1c-40bd335543c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" Oct 30 04:58:47.547991 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.547951 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" Oct 30 04:58:47.870800 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.870767 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt"] Oct 30 04:58:47.871550 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:58:47.871521 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030b0117_0d9a_4370_8a1c_40bd335543c7.slice/crio-5b5dcb2d4409b1a808f84ea1efb4352feaf222d2d32833039e2e316fe6dff8bd WatchSource:0}: Error finding container 5b5dcb2d4409b1a808f84ea1efb4352feaf222d2d32833039e2e316fe6dff8bd: Status 404 returned error can't find the container with id 5b5dcb2d4409b1a808f84ea1efb4352feaf222d2d32833039e2e316fe6dff8bd Oct 30 04:58:47.941962 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.941933 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" event={"ID":"b25ac0aa-86c0-4e22-aff5-490a57f95397","Type":"ContainerStarted","Data":"6982c48f69306b0f8c4dfe47ba6d13f46a81d78ef26773527d2d774d9a0c20ee"} Oct 30 04:58:47.942106 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.941986 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" event={"ID":"b25ac0aa-86c0-4e22-aff5-490a57f95397","Type":"ContainerStarted","Data":"9d7c2c7bd7ba7393b0299cccd0147a1f8a61c4339338abc01cf9981fbc0cf990"} Oct 30 04:58:47.943210 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.943179 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" event={"ID":"030b0117-0d9a-4370-8a1c-40bd335543c7","Type":"ContainerStarted","Data":"5b5dcb2d4409b1a808f84ea1efb4352feaf222d2d32833039e2e316fe6dff8bd"} Oct 30 04:58:47.985925 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:47.985886 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" podStartSLOduration=1.985872903 podStartE2EDuration="1.985872903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:47.983703243 +0000 UTC m=+852.511470028" watchObservedRunningTime="2026-10-30 04:58:47.985872903 +0000 UTC m=+852.513639689" Oct 30 04:58:48.952390 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:48.952314 6587 generic.go:356] "Generic (PLEG): container finished" podID="b25ac0aa-86c0-4e22-aff5-490a57f95397" containerID="6982c48f69306b0f8c4dfe47ba6d13f46a81d78ef26773527d2d774d9a0c20ee" exitCode=0 Oct 30 04:58:48.952799 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:48.952476 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" event={"ID":"b25ac0aa-86c0-4e22-aff5-490a57f95397","Type":"ContainerDied","Data":"6982c48f69306b0f8c4dfe47ba6d13f46a81d78ef26773527d2d774d9a0c20ee"} Oct 30 04:58:48.954456 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:48.954425 6587 generic.go:356] "Generic (PLEG): container finished" podID="030b0117-0d9a-4370-8a1c-40bd335543c7" containerID="68f221e75c2fb0660e10278c5c67b92e733246c948b5fb5abb7f20043365c5c9" exitCode=0 Oct 30 04:58:48.954618 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:48.954477 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" event={"ID":"030b0117-0d9a-4370-8a1c-40bd335543c7","Type":"ContainerDied","Data":"68f221e75c2fb0660e10278c5c67b92e733246c948b5fb5abb7f20043365c5c9"} Oct 30 04:58:50.132253 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.132225 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" Oct 30 04:58:50.165836 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.165813 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" Oct 30 04:58:50.204041 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.204012 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-openshift-ca\") pod \"b25ac0aa-86c0-4e22-aff5-490a57f95397\" (UID: \"b25ac0aa-86c0-4e22-aff5-490a57f95397\") " Oct 30 04:58:50.204196 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.204065 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-xks-ca\") pod \"030b0117-0d9a-4370-8a1c-40bd335543c7\" (UID: \"030b0117-0d9a-4370-8a1c-40bd335543c7\") " Oct 30 04:58:50.204196 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.204183 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-openshift-ca\") pod \"030b0117-0d9a-4370-8a1c-40bd335543c7\" (UID: \"030b0117-0d9a-4370-8a1c-40bd335543c7\") " Oct 30 04:58:50.204280 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.204226 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-xks-ca\") pod \"b25ac0aa-86c0-4e22-aff5-490a57f95397\" (UID: \"b25ac0aa-86c0-4e22-aff5-490a57f95397\") " Oct 30 04:58:50.204331 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.204292 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/030b0117-0d9a-4370-8a1c-40bd335543c7-kube-api-access-gbblw\" (UniqueName: \"kubernetes.io/projected/030b0117-0d9a-4370-8a1c-40bd335543c7-kube-api-access-gbblw\") pod \"030b0117-0d9a-4370-8a1c-40bd335543c7\" (UID: \"030b0117-0d9a-4370-8a1c-40bd335543c7\") " Oct 30 04:58:50.204403 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.204373 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25ac0aa-86c0-4e22-aff5-490a57f95397-kube-api-access-qd2jq\" (UniqueName: \"kubernetes.io/projected/b25ac0aa-86c0-4e22-aff5-490a57f95397-kube-api-access-qd2jq\") pod \"b25ac0aa-86c0-4e22-aff5-490a57f95397\" (UID: \"b25ac0aa-86c0-4e22-aff5-490a57f95397\") " Oct 30 04:58:50.204476 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.204453 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-openshift-ca" pod "b25ac0aa-86c0-4e22-aff5-490a57f95397" (UID: "b25ac0aa-86c0-4e22-aff5-490a57f95397"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:58:50.204918 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.204896 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:50.206313 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.206288 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-openshift-ca" pod "030b0117-0d9a-4370-8a1c-40bd335543c7" (UID: "030b0117-0d9a-4370-8a1c-40bd335543c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:58:50.208020 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.207992 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-xks-ca" pod "030b0117-0d9a-4370-8a1c-40bd335543c7" (UID: "030b0117-0d9a-4370-8a1c-40bd335543c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:58:50.209322 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.209288 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030b0117-0d9a-4370-8a1c-40bd335543c7-kube-api-access-gbblw" pod "030b0117-0d9a-4370-8a1c-40bd335543c7" (UID: "030b0117-0d9a-4370-8a1c-40bd335543c7"). InnerVolumeSpecName "kube-api-access-gbblw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:58:50.209439 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.209329 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25ac0aa-86c0-4e22-aff5-490a57f95397-kube-api-access-qd2jq" pod "b25ac0aa-86c0-4e22-aff5-490a57f95397" (UID: "b25ac0aa-86c0-4e22-aff5-490a57f95397"). InnerVolumeSpecName "kube-api-access-qd2jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:58:50.209583 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.209558 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-xks-ca" pod "b25ac0aa-86c0-4e22-aff5-490a57f95397" (UID: "b25ac0aa-86c0-4e22-aff5-490a57f95397"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:58:50.307016 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.306982 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:50.307016 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.307013 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/030b0117-0d9a-4370-8a1c-40bd335543c7-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:50.307016 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.307024 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25ac0aa-86c0-4e22-aff5-490a57f95397-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:50.307016 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.307034 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbblw\" (UniqueName: \"kubernetes.io/projected/030b0117-0d9a-4370-8a1c-40bd335543c7-kube-api-access-gbblw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:50.307387 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.307044 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd2jq\" (UniqueName: \"kubernetes.io/projected/b25ac0aa-86c0-4e22-aff5-490a57f95397-kube-api-access-qd2jq\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:50.973915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.973886 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" Oct 30 04:58:50.974086 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.973941 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz" event={"ID":"b25ac0aa-86c0-4e22-aff5-490a57f95397","Type":"ContainerDied","Data":"9d7c2c7bd7ba7393b0299cccd0147a1f8a61c4339338abc01cf9981fbc0cf990"} Oct 30 04:58:50.974086 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.974002 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d7c2c7bd7ba7393b0299cccd0147a1f8a61c4339338abc01cf9981fbc0cf990" Oct 30 04:58:50.975558 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.975534 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" event={"ID":"030b0117-0d9a-4370-8a1c-40bd335543c7","Type":"ContainerDied","Data":"5b5dcb2d4409b1a808f84ea1efb4352feaf222d2d32833039e2e316fe6dff8bd"} Oct 30 04:58:50.975558 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.975557 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt" Oct 30 04:58:50.975707 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:50.975576 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b5dcb2d4409b1a808f84ea1efb4352feaf222d2d32833039e2e316fe6dff8bd" Oct 30 04:58:52.446564 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.446523 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8"] Oct 30 04:58:52.447122 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.447105 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b25ac0aa-86c0-4e22-aff5-490a57f95397" containerName="revoke-keys" Oct 30 04:58:52.447122 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.447120 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25ac0aa-86c0-4e22-aff5-490a57f95397" containerName="revoke-keys" Oct 30 04:58:52.447258 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.447158 6587 cpu_manager.go:414] "containerMap: removing container" podUID="030b0117-0d9a-4370-8a1c-40bd335543c7" containerName="revoke-keys" Oct 30 04:58:52.447258 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.447164 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="030b0117-0d9a-4370-8a1c-40bd335543c7" containerName="revoke-keys" Oct 30 04:58:52.447258 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.447251 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25ac0aa-86c0-4e22-aff5-490a57f95397" containerName="revoke-keys" Oct 30 04:58:52.447416 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.447264 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="030b0117-0d9a-4370-8a1c-40bd335543c7" containerName="revoke-keys" Oct 30 04:58:52.450489 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.450468 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" Oct 30 04:58:52.466436 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.466413 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:58:52.469128 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.469106 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8"] Oct 30 04:58:52.480417 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.478804 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 04:58:52.480417 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.478905 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:58:52.480417 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.479792 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:58:52.532875 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.532838 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mllqg\" (UniqueName: \"kubernetes.io/projected/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-kube-api-access-mllqg\") pod \"maas-api-revoke-sub-bd1f5fb4-hk2l8\" (UID: \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" Oct 30 04:58:52.533036 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.532894 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd1f5fb4-hk2l8\" (UID: \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" Oct 30 04:58:52.533085 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.533065 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd1f5fb4-hk2l8\" (UID: \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" Oct 30 04:58:52.634585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.634544 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mllqg\" (UniqueName: \"kubernetes.io/projected/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-kube-api-access-mllqg\") pod \"maas-api-revoke-sub-bd1f5fb4-hk2l8\" (UID: \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" Oct 30 04:58:52.634781 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.634595 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd1f5fb4-hk2l8\" (UID: \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" Oct 30 04:58:52.634781 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.634703 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd1f5fb4-hk2l8\" (UID: \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" Oct 30 04:58:52.635309 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.635283 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd1f5fb4-hk2l8\" (UID: \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" Oct 30 04:58:52.637049 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.637026 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd1f5fb4-hk2l8\" (UID: \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" Oct 30 04:58:52.654369 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.654341 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mllqg\" (UniqueName: \"kubernetes.io/projected/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-kube-api-access-mllqg\") pod \"maas-api-revoke-sub-bd1f5fb4-hk2l8\" (UID: \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" Oct 30 04:58:52.768047 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:52.767965 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" Oct 30 04:58:53.062179 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:58:53.062146 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c331d5_051b_4a00_ae7f_e9ad577dcaeb.slice/crio-442da3b03924f58e55b4ad6cf1029dabe7175cb59c2bf484d48c8b96ce9e47d4 WatchSource:0}: Error finding container 442da3b03924f58e55b4ad6cf1029dabe7175cb59c2bf484d48c8b96ce9e47d4: Status 404 returned error can't find the container with id 442da3b03924f58e55b4ad6cf1029dabe7175cb59c2bf484d48c8b96ce9e47d4 Oct 30 04:58:53.069530 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:53.069501 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8"] Oct 30 04:58:53.995562 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:53.995528 6587 generic.go:356] "Generic (PLEG): container finished" podID="c0c331d5-051b-4a00-ae7f-e9ad577dcaeb" containerID="b0fcc11eb89db88c02280cd0abcc315f3428494d129a8f1da0b4280d66797b80" exitCode=0 Oct 30 04:58:53.995902 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:53.995611 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" event={"ID":"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb","Type":"ContainerDied","Data":"b0fcc11eb89db88c02280cd0abcc315f3428494d129a8f1da0b4280d66797b80"} Oct 30 04:58:53.995902 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:53.995663 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" event={"ID":"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb","Type":"ContainerStarted","Data":"442da3b03924f58e55b4ad6cf1029dabe7175cb59c2bf484d48c8b96ce9e47d4"} Oct 30 04:58:55.173181 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.173154 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" Oct 30 04:58:55.282131 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.282048 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-openshift-ca\") pod \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\" (UID: \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\") " Oct 30 04:58:55.282297 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.282144 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-kube-api-access-mllqg\" (UniqueName: \"kubernetes.io/projected/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-kube-api-access-mllqg\") pod \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\" (UID: \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\") " Oct 30 04:58:55.282297 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.282179 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-xks-ca\") pod \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\" (UID: \"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb\") " Oct 30 04:58:55.282472 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.282442 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-openshift-ca" pod "c0c331d5-051b-4a00-ae7f-e9ad577dcaeb" (UID: "c0c331d5-051b-4a00-ae7f-e9ad577dcaeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:58:55.284215 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.284193 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-xks-ca" pod "c0c331d5-051b-4a00-ae7f-e9ad577dcaeb" (UID: "c0c331d5-051b-4a00-ae7f-e9ad577dcaeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:58:55.284327 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.284272 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-kube-api-access-mllqg" pod "c0c331d5-051b-4a00-ae7f-e9ad577dcaeb" (UID: "c0c331d5-051b-4a00-ae7f-e9ad577dcaeb"). InnerVolumeSpecName "kube-api-access-mllqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:58:55.383663 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.383630 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mllqg\" (UniqueName: \"kubernetes.io/projected/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-kube-api-access-mllqg\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:55.383663 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.383657 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:55.383663 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.383668 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:55.913298 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.913210 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz"] Oct 30 04:58:55.913818 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.913801 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c0c331d5-051b-4a00-ae7f-e9ad577dcaeb" containerName="revoke-keys" Oct 30 04:58:55.913893 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.913818 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c331d5-051b-4a00-ae7f-e9ad577dcaeb" containerName="revoke-keys" Oct 30 04:58:55.913963 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.913953 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c331d5-051b-4a00-ae7f-e9ad577dcaeb" containerName="revoke-keys" Oct 30 04:58:55.917677 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.917654 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" Oct 30 04:58:55.936083 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.936049 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz"] Oct 30 04:58:55.990750 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.990711 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015cf7e4-6pcrz\" (UID: \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" Oct 30 04:58:55.990750 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:58:55.990716 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:58:55.990940 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.990786 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jn85\" (UniqueName: \"kubernetes.io/projected/44d517d5-ed40-4bcf-9125-5a012c5e94ee-kube-api-access-9jn85\") pod \"maas-api-revoke-sub-015cf7e4-6pcrz\" (UID: \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" Oct 30 04:58:55.990940 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:55.990818 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015cf7e4-6pcrz\" (UID: \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" Oct 30 04:58:56.010448 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.010416 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" event={"ID":"c0c331d5-051b-4a00-ae7f-e9ad577dcaeb","Type":"ContainerDied","Data":"442da3b03924f58e55b4ad6cf1029dabe7175cb59c2bf484d48c8b96ce9e47d4"} Oct 30 04:58:56.010448 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.010445 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8" Oct 30 04:58:56.010633 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.010470 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="442da3b03924f58e55b4ad6cf1029dabe7175cb59c2bf484d48c8b96ce9e47d4" Oct 30 04:58:56.091827 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.091792 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015cf7e4-6pcrz\" (UID: \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" Oct 30 04:58:56.092060 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.092040 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015cf7e4-6pcrz\" (UID: \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" Oct 30 04:58:56.092120 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.092083 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jn85\" (UniqueName: \"kubernetes.io/projected/44d517d5-ed40-4bcf-9125-5a012c5e94ee-kube-api-access-9jn85\") pod \"maas-api-revoke-sub-015cf7e4-6pcrz\" (UID: \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" Oct 30 04:58:56.092620 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.092600 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015cf7e4-6pcrz\" (UID: \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" Oct 30 04:58:56.094411 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.094391 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015cf7e4-6pcrz\" (UID: \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" Oct 30 04:58:56.111773 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.111750 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jn85\" (UniqueName: \"kubernetes.io/projected/44d517d5-ed40-4bcf-9125-5a012c5e94ee-kube-api-access-9jn85\") pod \"maas-api-revoke-sub-015cf7e4-6pcrz\" (UID: \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" Oct 30 04:58:56.241639 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.241545 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" Oct 30 04:58:56.542482 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:58:56.542434 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d517d5_ed40_4bcf_9125_5a012c5e94ee.slice/crio-483987546ce44e31b8aa38e433c937b17ff260ae403d2fc7f5c3f60619975355 WatchSource:0}: Error finding container 483987546ce44e31b8aa38e433c937b17ff260ae403d2fc7f5c3f60619975355: Status 404 returned error can't find the container with id 483987546ce44e31b8aa38e433c937b17ff260ae403d2fc7f5c3f60619975355 Oct 30 04:58:56.543201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.543174 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz"] Oct 30 04:58:56.586234 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.586210 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6"] Oct 30 04:58:56.591688 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.591669 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" Oct 30 04:58:56.608021 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.607995 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6"] Oct 30 04:58:56.704897 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.704864 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe807f85-54hq6\" (UID: \"0e78fe40-2571-4741-a15a-739200d2d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" Oct 30 04:58:56.705082 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.705064 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hfdp\" (UniqueName: \"kubernetes.io/projected/0e78fe40-2571-4741-a15a-739200d2d0fd-kube-api-access-2hfdp\") pod \"maas-api-revoke-sub-fe807f85-54hq6\" (UID: \"0e78fe40-2571-4741-a15a-739200d2d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" Oct 30 04:58:56.705204 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.705186 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe807f85-54hq6\" (UID: \"0e78fe40-2571-4741-a15a-739200d2d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" Oct 30 04:58:56.806523 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.806499 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe807f85-54hq6\" (UID: \"0e78fe40-2571-4741-a15a-739200d2d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" Oct 30 04:58:56.806663 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.806538 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe807f85-54hq6\" (UID: \"0e78fe40-2571-4741-a15a-739200d2d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" Oct 30 04:58:56.806704 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.806682 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hfdp\" (UniqueName: \"kubernetes.io/projected/0e78fe40-2571-4741-a15a-739200d2d0fd-kube-api-access-2hfdp\") pod \"maas-api-revoke-sub-fe807f85-54hq6\" (UID: \"0e78fe40-2571-4741-a15a-739200d2d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" Oct 30 04:58:56.807181 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.807156 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe807f85-54hq6\" (UID: \"0e78fe40-2571-4741-a15a-739200d2d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" Oct 30 04:58:56.808789 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.808771 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe807f85-54hq6\" (UID: \"0e78fe40-2571-4741-a15a-739200d2d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" Oct 30 04:58:56.826227 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.826199 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hfdp\" (UniqueName: \"kubernetes.io/projected/0e78fe40-2571-4741-a15a-739200d2d0fd-kube-api-access-2hfdp\") pod \"maas-api-revoke-sub-fe807f85-54hq6\" (UID: \"0e78fe40-2571-4741-a15a-739200d2d0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" Oct 30 04:58:56.904405 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:56.904381 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" Oct 30 04:58:57.029420 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:57.029328 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" event={"ID":"44d517d5-ed40-4bcf-9125-5a012c5e94ee","Type":"ContainerStarted","Data":"249de0e1fb087df68f590e3129d2c06e10c6036638e8bbf50f1d8a67559aa74f"} Oct 30 04:58:57.029605 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:57.029482 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" event={"ID":"44d517d5-ed40-4bcf-9125-5a012c5e94ee","Type":"ContainerStarted","Data":"483987546ce44e31b8aa38e433c937b17ff260ae403d2fc7f5c3f60619975355"} Oct 30 04:58:57.074092 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:57.073985 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" podStartSLOduration=2.07396925 podStartE2EDuration="2.07396925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:57.072819312 +0000 UTC m=+861.600586098" watchObservedRunningTime="2026-10-30 04:58:57.07396925 +0000 UTC m=+861.601736039" Oct 30 04:58:57.204379 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:57.204340 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6"] Oct 30 04:58:57.206498 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:58:57.206465 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e78fe40_2571_4741_a15a_739200d2d0fd.slice/crio-93c0dae795a63c3d1e30d74288f7e850381cb1d10e33e0c2e668cd79f9754f24 WatchSource:0}: Error finding container 93c0dae795a63c3d1e30d74288f7e850381cb1d10e33e0c2e668cd79f9754f24: Status 404 returned error can't find the container with id 93c0dae795a63c3d1e30d74288f7e850381cb1d10e33e0c2e668cd79f9754f24 Oct 30 04:58:58.051382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:58.050882 6587 generic.go:356] "Generic (PLEG): container finished" podID="44d517d5-ed40-4bcf-9125-5a012c5e94ee" containerID="249de0e1fb087df68f590e3129d2c06e10c6036638e8bbf50f1d8a67559aa74f" exitCode=0 Oct 30 04:58:58.051382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:58.050979 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" event={"ID":"44d517d5-ed40-4bcf-9125-5a012c5e94ee","Type":"ContainerDied","Data":"249de0e1fb087df68f590e3129d2c06e10c6036638e8bbf50f1d8a67559aa74f"} Oct 30 04:58:58.055379 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:58.053664 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" event={"ID":"0e78fe40-2571-4741-a15a-739200d2d0fd","Type":"ContainerStarted","Data":"0bcb3675be5597cf9309a0494ae66c3acd442a3670e619cb0c38ac9824a84206"} Oct 30 04:58:58.055379 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:58.053719 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" event={"ID":"0e78fe40-2571-4741-a15a-739200d2d0fd","Type":"ContainerStarted","Data":"93c0dae795a63c3d1e30d74288f7e850381cb1d10e33e0c2e668cd79f9754f24"} Oct 30 04:58:58.160628 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:58.157056 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" podStartSLOduration=2.157035586 podStartE2EDuration="2.157035586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:58.153700996 +0000 UTC m=+862.681467783" watchObservedRunningTime="2026-10-30 04:58:58.157035586 +0000 UTC m=+862.684802375" Oct 30 04:58:59.064691 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:59.064660 6587 generic.go:356] "Generic (PLEG): container finished" podID="0e78fe40-2571-4741-a15a-739200d2d0fd" containerID="0bcb3675be5597cf9309a0494ae66c3acd442a3670e619cb0c38ac9824a84206" exitCode=0 Oct 30 04:58:59.065201 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:59.065089 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" event={"ID":"0e78fe40-2571-4741-a15a-739200d2d0fd","Type":"ContainerDied","Data":"0bcb3675be5597cf9309a0494ae66c3acd442a3670e619cb0c38ac9824a84206"} Oct 30 04:58:59.255160 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:59.255124 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" Oct 30 04:58:59.369317 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:59.369021 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d517d5-ed40-4bcf-9125-5a012c5e94ee-kube-api-access-9jn85\" (UniqueName: \"kubernetes.io/projected/44d517d5-ed40-4bcf-9125-5a012c5e94ee-kube-api-access-9jn85\") pod \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\" (UID: \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\") " Oct 30 04:58:59.369317 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:59.369108 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-openshift-ca\") pod \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\" (UID: \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\") " Oct 30 04:58:59.369317 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:59.369176 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-xks-ca\") pod \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\" (UID: \"44d517d5-ed40-4bcf-9125-5a012c5e94ee\") " Oct 30 04:58:59.370587 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:59.370557 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-openshift-ca" pod "44d517d5-ed40-4bcf-9125-5a012c5e94ee" (UID: "44d517d5-ed40-4bcf-9125-5a012c5e94ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:58:59.372302 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:59.372235 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-xks-ca" pod "44d517d5-ed40-4bcf-9125-5a012c5e94ee" (UID: "44d517d5-ed40-4bcf-9125-5a012c5e94ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:58:59.374600 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:59.374568 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d517d5-ed40-4bcf-9125-5a012c5e94ee-kube-api-access-9jn85" pod "44d517d5-ed40-4bcf-9125-5a012c5e94ee" (UID: "44d517d5-ed40-4bcf-9125-5a012c5e94ee"). InnerVolumeSpecName "kube-api-access-9jn85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:58:59.470782 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:59.470749 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:59.470782 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:59.470776 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jn85\" (UniqueName: \"kubernetes.io/projected/44d517d5-ed40-4bcf-9125-5a012c5e94ee-kube-api-access-9jn85\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:58:59.470782 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:58:59.470787 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d517d5-ed40-4bcf-9125-5a012c5e94ee-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:59:00.093055 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:00.093022 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" Oct 30 04:59:00.093557 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:00.093477 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz" event={"ID":"44d517d5-ed40-4bcf-9125-5a012c5e94ee","Type":"ContainerDied","Data":"483987546ce44e31b8aa38e433c937b17ff260ae403d2fc7f5c3f60619975355"} Oct 30 04:59:00.093557 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:00.093531 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="483987546ce44e31b8aa38e433c937b17ff260ae403d2fc7f5c3f60619975355" Oct 30 04:59:00.510202 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:00.510170 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" Oct 30 04:59:00.702136 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:00.702093 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-openshift-ca\") pod \"0e78fe40-2571-4741-a15a-739200d2d0fd\" (UID: \"0e78fe40-2571-4741-a15a-739200d2d0fd\") " Oct 30 04:59:00.702322 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:00.702261 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e78fe40-2571-4741-a15a-739200d2d0fd-kube-api-access-2hfdp\" (UniqueName: \"kubernetes.io/projected/0e78fe40-2571-4741-a15a-739200d2d0fd-kube-api-access-2hfdp\") pod \"0e78fe40-2571-4741-a15a-739200d2d0fd\" (UID: \"0e78fe40-2571-4741-a15a-739200d2d0fd\") " Oct 30 04:59:00.702322 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:00.702309 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-xks-ca\") pod \"0e78fe40-2571-4741-a15a-739200d2d0fd\" (UID: \"0e78fe40-2571-4741-a15a-739200d2d0fd\") " Oct 30 04:59:00.702777 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:00.702738 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-openshift-ca" pod "0e78fe40-2571-4741-a15a-739200d2d0fd" (UID: "0e78fe40-2571-4741-a15a-739200d2d0fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:59:00.705767 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:00.705737 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-xks-ca" pod "0e78fe40-2571-4741-a15a-739200d2d0fd" (UID: "0e78fe40-2571-4741-a15a-739200d2d0fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:59:00.705914 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:00.705845 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e78fe40-2571-4741-a15a-739200d2d0fd-kube-api-access-2hfdp" pod "0e78fe40-2571-4741-a15a-739200d2d0fd" (UID: "0e78fe40-2571-4741-a15a-739200d2d0fd"). InnerVolumeSpecName "kube-api-access-2hfdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:59:00.803941 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:00.803909 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hfdp\" (UniqueName: \"kubernetes.io/projected/0e78fe40-2571-4741-a15a-739200d2d0fd-kube-api-access-2hfdp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:59:00.803941 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:00.803938 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:59:00.804138 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:00.803949 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e78fe40-2571-4741-a15a-739200d2d0fd-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:59:01.100821 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:01.100743 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" Oct 30 04:59:01.100821 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:01.100741 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6" event={"ID":"0e78fe40-2571-4741-a15a-739200d2d0fd","Type":"ContainerDied","Data":"93c0dae795a63c3d1e30d74288f7e850381cb1d10e33e0c2e668cd79f9754f24"} Oct 30 04:59:01.100821 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:01.100800 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93c0dae795a63c3d1e30d74288f7e850381cb1d10e33e0c2e668cd79f9754f24" Oct 30 04:59:02.007147 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.007111 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc"] Oct 30 04:59:02.007741 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.007722 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0e78fe40-2571-4741-a15a-739200d2d0fd" containerName="revoke-keys" Oct 30 04:59:02.007848 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.007742 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e78fe40-2571-4741-a15a-739200d2d0fd" containerName="revoke-keys" Oct 30 04:59:02.007848 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.007804 6587 cpu_manager.go:414] "containerMap: removing container" podUID="44d517d5-ed40-4bcf-9125-5a012c5e94ee" containerName="revoke-keys" Oct 30 04:59:02.007848 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.007813 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d517d5-ed40-4bcf-9125-5a012c5e94ee" containerName="revoke-keys" Oct 30 04:59:02.008004 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.007948 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d517d5-ed40-4bcf-9125-5a012c5e94ee" containerName="revoke-keys" Oct 30 04:59:02.008004 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.007968 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e78fe40-2571-4741-a15a-739200d2d0fd" containerName="revoke-keys" Oct 30 04:59:02.011655 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.011634 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" Oct 30 04:59:02.044382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.040713 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:59:02.044382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.040880 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:59:02.044382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.040972 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 04:59:02.044382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.041185 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:59:02.046644 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.046616 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc"] Oct 30 04:59:02.117619 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.117581 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5fth\" (UniqueName: \"kubernetes.io/projected/d3ae2139-e029-4928-a061-9ab4fb7fde04-kube-api-access-l5fth\") pod \"maas-api-revoke-sub-409d4ef0-zzptc\" (UID: \"d3ae2139-e029-4928-a061-9ab4fb7fde04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" Oct 30 04:59:02.118109 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.117649 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-409d4ef0-zzptc\" (UID: \"d3ae2139-e029-4928-a061-9ab4fb7fde04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" Oct 30 04:59:02.118109 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.117760 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-409d4ef0-zzptc\" (UID: \"d3ae2139-e029-4928-a061-9ab4fb7fde04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" Oct 30 04:59:02.218649 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.218610 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-409d4ef0-zzptc\" (UID: \"d3ae2139-e029-4928-a061-9ab4fb7fde04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" Oct 30 04:59:02.218856 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.218768 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5fth\" (UniqueName: \"kubernetes.io/projected/d3ae2139-e029-4928-a061-9ab4fb7fde04-kube-api-access-l5fth\") pod \"maas-api-revoke-sub-409d4ef0-zzptc\" (UID: \"d3ae2139-e029-4928-a061-9ab4fb7fde04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" Oct 30 04:59:02.218856 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.218835 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-409d4ef0-zzptc\" (UID: \"d3ae2139-e029-4928-a061-9ab4fb7fde04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" Oct 30 04:59:02.219468 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.219444 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-409d4ef0-zzptc\" (UID: \"d3ae2139-e029-4928-a061-9ab4fb7fde04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" Oct 30 04:59:02.221123 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.221090 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-409d4ef0-zzptc\" (UID: \"d3ae2139-e029-4928-a061-9ab4fb7fde04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" Oct 30 04:59:02.239045 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.239012 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5fth\" (UniqueName: \"kubernetes.io/projected/d3ae2139-e029-4928-a061-9ab4fb7fde04-kube-api-access-l5fth\") pod \"maas-api-revoke-sub-409d4ef0-zzptc\" (UID: \"d3ae2139-e029-4928-a061-9ab4fb7fde04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" Oct 30 04:59:02.354606 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.354574 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" Oct 30 04:59:02.651448 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:59:02.651421 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ae2139_e029_4928_a061_9ab4fb7fde04.slice/crio-c87776bd02ffc08d39d1aa36b5e73767fe554c711c55834d6b0b60743dd6c684 WatchSource:0}: Error finding container c87776bd02ffc08d39d1aa36b5e73767fe554c711c55834d6b0b60743dd6c684: Status 404 returned error can't find the container with id c87776bd02ffc08d39d1aa36b5e73767fe554c711c55834d6b0b60743dd6c684 Oct 30 04:59:02.651544 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:02.651518 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc"] Oct 30 04:59:03.115003 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.114963 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" event={"ID":"d3ae2139-e029-4928-a061-9ab4fb7fde04","Type":"ContainerStarted","Data":"f5fb5f644c43d6f48f92d2465edd6ccbc61db9cb9fab36df89ccd7173476d9f2"} Oct 30 04:59:03.115199 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.115027 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" event={"ID":"d3ae2139-e029-4928-a061-9ab4fb7fde04","Type":"ContainerStarted","Data":"c87776bd02ffc08d39d1aa36b5e73767fe554c711c55834d6b0b60743dd6c684"} Oct 30 04:59:03.158306 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.158258 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" podStartSLOduration=2.158244386 podStartE2EDuration="2.158244386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:03.155630526 +0000 UTC m=+867.683397304" watchObservedRunningTime="2026-10-30 04:59:03.158244386 +0000 UTC m=+867.686011209" Oct 30 04:59:03.692741 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.692709 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc"] Oct 30 04:59:03.696752 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.696734 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" Oct 30 04:59:03.716789 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.716758 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc"] Oct 30 04:59:03.736309 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.736268 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt5n7\" (UniqueName: \"kubernetes.io/projected/4d02f424-d083-4820-90d5-abf6a9764688-kube-api-access-zt5n7\") pod \"maas-api-revoke-sub-186d6b66-kz9tc\" (UID: \"4d02f424-d083-4820-90d5-abf6a9764688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" Oct 30 04:59:03.736554 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.736318 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-186d6b66-kz9tc\" (UID: \"4d02f424-d083-4820-90d5-abf6a9764688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" Oct 30 04:59:03.736554 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.736474 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-xks-ca\") pod \"maas-api-revoke-sub-186d6b66-kz9tc\" (UID: \"4d02f424-d083-4820-90d5-abf6a9764688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" Oct 30 04:59:03.837507 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.837468 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-xks-ca\") pod \"maas-api-revoke-sub-186d6b66-kz9tc\" (UID: \"4d02f424-d083-4820-90d5-abf6a9764688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" Oct 30 04:59:03.837694 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.837599 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt5n7\" (UniqueName: \"kubernetes.io/projected/4d02f424-d083-4820-90d5-abf6a9764688-kube-api-access-zt5n7\") pod \"maas-api-revoke-sub-186d6b66-kz9tc\" (UID: \"4d02f424-d083-4820-90d5-abf6a9764688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" Oct 30 04:59:03.837694 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.837630 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-186d6b66-kz9tc\" (UID: \"4d02f424-d083-4820-90d5-abf6a9764688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" Oct 30 04:59:03.838406 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.838310 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-186d6b66-kz9tc\" (UID: \"4d02f424-d083-4820-90d5-abf6a9764688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" Oct 30 04:59:03.840071 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.840047 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-xks-ca\") pod \"maas-api-revoke-sub-186d6b66-kz9tc\" (UID: \"4d02f424-d083-4820-90d5-abf6a9764688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" Oct 30 04:59:03.857439 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:03.857411 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt5n7\" (UniqueName: \"kubernetes.io/projected/4d02f424-d083-4820-90d5-abf6a9764688-kube-api-access-zt5n7\") pod \"maas-api-revoke-sub-186d6b66-kz9tc\" (UID: \"4d02f424-d083-4820-90d5-abf6a9764688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" Oct 30 04:59:04.011575 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:04.011541 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" Oct 30 04:59:04.126721 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:04.126635 6587 generic.go:356] "Generic (PLEG): container finished" podID="d3ae2139-e029-4928-a061-9ab4fb7fde04" containerID="f5fb5f644c43d6f48f92d2465edd6ccbc61db9cb9fab36df89ccd7173476d9f2" exitCode=0 Oct 30 04:59:04.126877 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:04.126790 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" event={"ID":"d3ae2139-e029-4928-a061-9ab4fb7fde04","Type":"ContainerDied","Data":"f5fb5f644c43d6f48f92d2465edd6ccbc61db9cb9fab36df89ccd7173476d9f2"} Oct 30 04:59:04.313743 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:59:04.313713 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d02f424_d083_4820_90d5_abf6a9764688.slice/crio-6f6582144152c838b9a290e0553becc0e1fbd87f9cf4509e913745c77b248fce WatchSource:0}: Error finding container 6f6582144152c838b9a290e0553becc0e1fbd87f9cf4509e913745c77b248fce: Status 404 returned error can't find the container with id 6f6582144152c838b9a290e0553becc0e1fbd87f9cf4509e913745c77b248fce Oct 30 04:59:04.314664 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:04.314638 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc"] Oct 30 04:59:05.135290 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:05.135249 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" event={"ID":"4d02f424-d083-4820-90d5-abf6a9764688","Type":"ContainerStarted","Data":"56844ffbd10c7b51be4e0b7d28385ebdc57fd6d16af540c35ba247362c096bcc"} Oct 30 04:59:05.135487 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:05.135323 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" event={"ID":"4d02f424-d083-4820-90d5-abf6a9764688","Type":"ContainerStarted","Data":"6f6582144152c838b9a290e0553becc0e1fbd87f9cf4509e913745c77b248fce"} Oct 30 04:59:05.178332 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:05.178286 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" podStartSLOduration=2.178273039 podStartE2EDuration="2.178273039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:05.177346687 +0000 UTC m=+869.705113473" watchObservedRunningTime="2026-10-30 04:59:05.178273039 +0000 UTC m=+869.706039862" Oct 30 04:59:05.303972 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:05.303938 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" Oct 30 04:59:05.357825 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:05.356961 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ae2139-e029-4928-a061-9ab4fb7fde04-kube-api-access-l5fth\" (UniqueName: \"kubernetes.io/projected/d3ae2139-e029-4928-a061-9ab4fb7fde04-kube-api-access-l5fth\") pod \"d3ae2139-e029-4928-a061-9ab4fb7fde04\" (UID: \"d3ae2139-e029-4928-a061-9ab4fb7fde04\") " Oct 30 04:59:05.357825 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:05.357047 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-xks-ca\") pod \"d3ae2139-e029-4928-a061-9ab4fb7fde04\" (UID: \"d3ae2139-e029-4928-a061-9ab4fb7fde04\") " Oct 30 04:59:05.357825 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:05.357118 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-openshift-ca\") pod \"d3ae2139-e029-4928-a061-9ab4fb7fde04\" (UID: \"d3ae2139-e029-4928-a061-9ab4fb7fde04\") " Oct 30 04:59:05.358904 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:05.358589 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-openshift-ca" pod "d3ae2139-e029-4928-a061-9ab4fb7fde04" (UID: "d3ae2139-e029-4928-a061-9ab4fb7fde04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:59:05.387008 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:05.386930 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-xks-ca" pod "d3ae2139-e029-4928-a061-9ab4fb7fde04" (UID: "d3ae2139-e029-4928-a061-9ab4fb7fde04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:59:05.387562 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:05.387528 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ae2139-e029-4928-a061-9ab4fb7fde04-kube-api-access-l5fth" pod "d3ae2139-e029-4928-a061-9ab4fb7fde04" (UID: "d3ae2139-e029-4928-a061-9ab4fb7fde04"). InnerVolumeSpecName "kube-api-access-l5fth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:59:05.458554 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:05.458521 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:59:05.458554 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:05.458549 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae2139-e029-4928-a061-9ab4fb7fde04-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:59:05.458554 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:05.458560 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5fth\" (UniqueName: \"kubernetes.io/projected/d3ae2139-e029-4928-a061-9ab4fb7fde04-kube-api-access-l5fth\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:59:06.142884 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.142851 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" Oct 30 04:59:06.142884 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.142874 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc" event={"ID":"d3ae2139-e029-4928-a061-9ab4fb7fde04","Type":"ContainerDied","Data":"c87776bd02ffc08d39d1aa36b5e73767fe554c711c55834d6b0b60743dd6c684"} Oct 30 04:59:06.143103 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.142927 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c87776bd02ffc08d39d1aa36b5e73767fe554c711c55834d6b0b60743dd6c684" Oct 30 04:59:06.144839 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.144812 6587 generic.go:356] "Generic (PLEG): container finished" podID="4d02f424-d083-4820-90d5-abf6a9764688" containerID="56844ffbd10c7b51be4e0b7d28385ebdc57fd6d16af540c35ba247362c096bcc" exitCode=0 Oct 30 04:59:06.144974 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.144881 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" event={"ID":"4d02f424-d083-4820-90d5-abf6a9764688","Type":"ContainerDied","Data":"56844ffbd10c7b51be4e0b7d28385ebdc57fd6d16af540c35ba247362c096bcc"} Oct 30 04:59:06.296489 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.296453 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd"] Oct 30 04:59:06.297037 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.297024 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d3ae2139-e029-4928-a061-9ab4fb7fde04" containerName="revoke-keys" Oct 30 04:59:06.297094 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.297038 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ae2139-e029-4928-a061-9ab4fb7fde04" containerName="revoke-keys" Oct 30 04:59:06.297149 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.297140 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ae2139-e029-4928-a061-9ab4fb7fde04" containerName="revoke-keys" Oct 30 04:59:06.300642 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.300627 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" Oct 30 04:59:06.319584 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.319562 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd"] Oct 30 04:59:06.369102 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.369066 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e9458c9-s9bgd\" (UID: \"9c720829-65a8-4900-8b86-09ddb4c1608d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" Oct 30 04:59:06.369553 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.369243 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e9458c9-s9bgd\" (UID: \"9c720829-65a8-4900-8b86-09ddb4c1608d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" Oct 30 04:59:06.369553 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.369273 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46948\" (UniqueName: \"kubernetes.io/projected/9c720829-65a8-4900-8b86-09ddb4c1608d-kube-api-access-46948\") pod \"maas-api-revoke-sub-5e9458c9-s9bgd\" (UID: \"9c720829-65a8-4900-8b86-09ddb4c1608d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" Oct 30 04:59:06.470330 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.470251 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e9458c9-s9bgd\" (UID: \"9c720829-65a8-4900-8b86-09ddb4c1608d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" Oct 30 04:59:06.470513 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.470381 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e9458c9-s9bgd\" (UID: \"9c720829-65a8-4900-8b86-09ddb4c1608d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" Oct 30 04:59:06.470513 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.470403 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46948\" (UniqueName: \"kubernetes.io/projected/9c720829-65a8-4900-8b86-09ddb4c1608d-kube-api-access-46948\") pod \"maas-api-revoke-sub-5e9458c9-s9bgd\" (UID: \"9c720829-65a8-4900-8b86-09ddb4c1608d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" Oct 30 04:59:06.470963 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.470936 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e9458c9-s9bgd\" (UID: \"9c720829-65a8-4900-8b86-09ddb4c1608d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" Oct 30 04:59:06.472681 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.472656 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e9458c9-s9bgd\" (UID: \"9c720829-65a8-4900-8b86-09ddb4c1608d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" Oct 30 04:59:06.490415 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.490383 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46948\" (UniqueName: \"kubernetes.io/projected/9c720829-65a8-4900-8b86-09ddb4c1608d-kube-api-access-46948\") pod \"maas-api-revoke-sub-5e9458c9-s9bgd\" (UID: \"9c720829-65a8-4900-8b86-09ddb4c1608d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" Oct 30 04:59:06.617306 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.617267 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" Oct 30 04:59:06.916723 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:59:06.916685 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c720829_65a8_4900_8b86_09ddb4c1608d.slice/crio-2828ee3d70d8194cc7bd396ee3f4525a66e125f63265e5d4ebc2a729cec3144e WatchSource:0}: Error finding container 2828ee3d70d8194cc7bd396ee3f4525a66e125f63265e5d4ebc2a729cec3144e: Status 404 returned error can't find the container with id 2828ee3d70d8194cc7bd396ee3f4525a66e125f63265e5d4ebc2a729cec3144e Oct 30 04:59:06.917068 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:06.917047 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd"] Oct 30 04:59:07.144312 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.144274 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54"] Oct 30 04:59:07.157167 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.151679 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.163693 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.163431 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" event={"ID":"9c720829-65a8-4900-8b86-09ddb4c1608d","Type":"ContainerStarted","Data":"2828ee3d70d8194cc7bd396ee3f4525a66e125f63265e5d4ebc2a729cec3144e"} Oct 30 04:59:07.194408 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.180566 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54"] Oct 30 04:59:07.194408 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.181205 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.194408 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.181242 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8358c921-c2fe-4b5f-8374-c466080e8631-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.194408 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.181265 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.194408 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.181297 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8358c921-c2fe-4b5f-8374-c466080e8631-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.194408 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.181333 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.194408 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.181383 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbfcs\" (UniqueName: \"kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-kube-api-access-qbfcs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.194408 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.181459 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.194408 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.181473 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.194408 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.181560 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.283461 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.283419 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.283600 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.283466 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.283600 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.283583 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.283794 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.283622 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.283794 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.283656 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8358c921-c2fe-4b5f-8374-c466080e8631-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.283794 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.283684 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.283794 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.283711 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8358c921-c2fe-4b5f-8374-c466080e8631-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.283794 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.283768 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.284054 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.283812 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbfcs\" (UniqueName: \"kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-kube-api-access-qbfcs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.284054 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.283825 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.284174 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.284148 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.284978 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.284950 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8358c921-c2fe-4b5f-8374-c466080e8631-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.285074 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.285007 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.285168 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.285135 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.286104 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.286083 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.287113 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.287093 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8358c921-c2fe-4b5f-8374-c466080e8631-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.304700 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.304671 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbfcs\" (UniqueName: \"kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-kube-api-access-qbfcs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.305916 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.305893 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.339094 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.339072 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" Oct 30 04:59:07.385550 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.384822 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d02f424-d083-4820-90d5-abf6a9764688-kube-api-access-zt5n7\" (UniqueName: \"kubernetes.io/projected/4d02f424-d083-4820-90d5-abf6a9764688-kube-api-access-zt5n7\") pod \"4d02f424-d083-4820-90d5-abf6a9764688\" (UID: \"4d02f424-d083-4820-90d5-abf6a9764688\") " Oct 30 04:59:07.385550 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.384954 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-openshift-ca\") pod \"4d02f424-d083-4820-90d5-abf6a9764688\" (UID: \"4d02f424-d083-4820-90d5-abf6a9764688\") " Oct 30 04:59:07.385550 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.385054 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-xks-ca\") pod \"4d02f424-d083-4820-90d5-abf6a9764688\" (UID: \"4d02f424-d083-4820-90d5-abf6a9764688\") " Oct 30 04:59:07.388138 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.388064 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-openshift-ca" pod "4d02f424-d083-4820-90d5-abf6a9764688" (UID: "4d02f424-d083-4820-90d5-abf6a9764688"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:59:07.388779 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.388761 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:59:07.391931 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.391906 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-xks-ca" pod "4d02f424-d083-4820-90d5-abf6a9764688" (UID: "4d02f424-d083-4820-90d5-abf6a9764688"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:59:07.392006 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.391983 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d02f424-d083-4820-90d5-abf6a9764688-kube-api-access-zt5n7" pod "4d02f424-d083-4820-90d5-abf6a9764688" (UID: "4d02f424-d083-4820-90d5-abf6a9764688"). InnerVolumeSpecName "kube-api-access-zt5n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:59:07.489659 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.489632 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt5n7\" (UniqueName: \"kubernetes.io/projected/4d02f424-d083-4820-90d5-abf6a9764688-kube-api-access-zt5n7\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:59:07.489659 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.489659 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d02f424-d083-4820-90d5-abf6a9764688-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:59:07.490253 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.490221 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:07.798825 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.798800 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54"] Oct 30 04:59:07.799389 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:59:07.799332 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8358c921_c2fe_4b5f_8374_c466080e8631.slice/crio-40da0bac71973a17f06e6414e55c9ff5d0095c3ad48784e87190727d41925690 WatchSource:0}: Error finding container 40da0bac71973a17f06e6414e55c9ff5d0095c3ad48784e87190727d41925690: Status 404 returned error can't find the container with id 40da0bac71973a17f06e6414e55c9ff5d0095c3ad48784e87190727d41925690 Oct 30 04:59:07.802005 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.801967 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:07.802083 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.802058 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:07.802127 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:07.802111 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:08.174222 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:08.174186 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" event={"ID":"8358c921-c2fe-4b5f-8374-c466080e8631","Type":"ContainerStarted","Data":"3f3b9ec08203c24a468e95b3925ba0cdee74abc6b5642890d29c76afe90ea365"} Oct 30 04:59:08.174418 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:08.174243 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" event={"ID":"8358c921-c2fe-4b5f-8374-c466080e8631","Type":"ContainerStarted","Data":"40da0bac71973a17f06e6414e55c9ff5d0095c3ad48784e87190727d41925690"} Oct 30 04:59:08.176652 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:08.176625 6587 generic.go:356] "Generic (PLEG): container finished" podID="9c720829-65a8-4900-8b86-09ddb4c1608d" containerID="10dbd51cdf070de7a0ffad4877d8890f295213ef0718043099d8c996ff725eed" exitCode=0 Oct 30 04:59:08.176783 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:08.176712 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" event={"ID":"9c720829-65a8-4900-8b86-09ddb4c1608d","Type":"ContainerDied","Data":"10dbd51cdf070de7a0ffad4877d8890f295213ef0718043099d8c996ff725eed"} Oct 30 04:59:08.178230 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:08.178210 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" Oct 30 04:59:08.178388 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:08.178238 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc" event={"ID":"4d02f424-d083-4820-90d5-abf6a9764688","Type":"ContainerDied","Data":"6f6582144152c838b9a290e0553becc0e1fbd87f9cf4509e913745c77b248fce"} Oct 30 04:59:08.178388 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:08.178272 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f6582144152c838b9a290e0553becc0e1fbd87f9cf4509e913745c77b248fce" Oct 30 04:59:08.226856 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:08.226803 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" podStartSLOduration=1.226788303 podStartE2EDuration="1.226788303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:08.217329248 +0000 UTC m=+872.745096036" watchObservedRunningTime="2026-10-30 04:59:08.226788303 +0000 UTC m=+872.754555150" Oct 30 04:59:08.491096 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:08.491068 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:08.492532 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:08.492511 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 04:59:08.492585 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:08.492559 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" podUID="8358c921-c2fe-4b5f-8374-c466080e8631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 04:59:09.383170 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:09.383143 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" Oct 30 04:59:09.414212 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:09.414186 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-xks-ca\") pod \"9c720829-65a8-4900-8b86-09ddb4c1608d\" (UID: \"9c720829-65a8-4900-8b86-09ddb4c1608d\") " Oct 30 04:59:09.414388 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:09.414290 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-openshift-ca\") pod \"9c720829-65a8-4900-8b86-09ddb4c1608d\" (UID: \"9c720829-65a8-4900-8b86-09ddb4c1608d\") " Oct 30 04:59:09.414463 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:09.414445 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c720829-65a8-4900-8b86-09ddb4c1608d-kube-api-access-46948\" (UniqueName: \"kubernetes.io/projected/9c720829-65a8-4900-8b86-09ddb4c1608d-kube-api-access-46948\") pod \"9c720829-65a8-4900-8b86-09ddb4c1608d\" (UID: \"9c720829-65a8-4900-8b86-09ddb4c1608d\") " Oct 30 04:59:09.414716 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:09.414695 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-openshift-ca" pod "9c720829-65a8-4900-8b86-09ddb4c1608d" (UID: "9c720829-65a8-4900-8b86-09ddb4c1608d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:59:09.414935 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:09.414911 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:59:09.458414 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:09.456547 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-xks-ca" pod "9c720829-65a8-4900-8b86-09ddb4c1608d" (UID: "9c720829-65a8-4900-8b86-09ddb4c1608d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:59:09.459328 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:09.459292 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c720829-65a8-4900-8b86-09ddb4c1608d-kube-api-access-46948" pod "9c720829-65a8-4900-8b86-09ddb4c1608d" (UID: "9c720829-65a8-4900-8b86-09ddb4c1608d"). InnerVolumeSpecName "kube-api-access-46948". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:59:09.491347 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:09.491321 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 04:59:09.491775 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:09.491401 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" podUID="8358c921-c2fe-4b5f-8374-c466080e8631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 04:59:09.520557 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:09.516127 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c720829-65a8-4900-8b86-09ddb4c1608d-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:59:09.520557 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:09.516174 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46948\" (UniqueName: \"kubernetes.io/projected/9c720829-65a8-4900-8b86-09ddb4c1608d-kube-api-access-46948\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 04:59:10.194910 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:10.194876 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" event={"ID":"9c720829-65a8-4900-8b86-09ddb4c1608d","Type":"ContainerDied","Data":"2828ee3d70d8194cc7bd396ee3f4525a66e125f63265e5d4ebc2a729cec3144e"} Oct 30 04:59:10.194910 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:10.194907 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd" Oct 30 04:59:10.195128 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:10.194951 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2828ee3d70d8194cc7bd396ee3f4525a66e125f63265e5d4ebc2a729cec3144e" Oct 30 04:59:10.491188 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:10.491097 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 04:59:10.491188 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:10.491162 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" podUID="8358c921-c2fe-4b5f-8374-c466080e8631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 04:59:11.490742 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:11.490707 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 04:59:11.491114 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:11.490771 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" podUID="8358c921-c2fe-4b5f-8374-c466080e8631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 04:59:12.491238 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:12.491205 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 04:59:12.491674 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:12.491282 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" podUID="8358c921-c2fe-4b5f-8374-c466080e8631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 04:59:13.490825 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:13.490787 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 04:59:13.491037 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:13.490862 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" podUID="8358c921-c2fe-4b5f-8374-c466080e8631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 04:59:14.495288 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:14.495251 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 04:59:14.495930 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:14.495881 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" podUID="8358c921-c2fe-4b5f-8374-c466080e8631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 04:59:16.504378 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:16.503480 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:59:16.504378 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:16.503570 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" podUID="8358c921-c2fe-4b5f-8374-c466080e8631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:16.504942 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:16.504772 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:16.506155 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:16.506136 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:16.506992 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:16.506970 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 04:59:21.361130 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.361096 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn"] Oct 30 04:59:21.361884 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.361866 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4d02f424-d083-4820-90d5-abf6a9764688" containerName="revoke-keys" Oct 30 04:59:21.361934 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.361885 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d02f424-d083-4820-90d5-abf6a9764688" containerName="revoke-keys" Oct 30 04:59:21.361934 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.361924 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9c720829-65a8-4900-8b86-09ddb4c1608d" containerName="revoke-keys" Oct 30 04:59:21.361934 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.361929 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c720829-65a8-4900-8b86-09ddb4c1608d" containerName="revoke-keys" Oct 30 04:59:21.362076 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.362065 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d02f424-d083-4820-90d5-abf6a9764688" containerName="revoke-keys" Oct 30 04:59:21.362114 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.362089 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c720829-65a8-4900-8b86-09ddb4c1608d" containerName="revoke-keys" Oct 30 04:59:21.365469 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.365454 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.401589 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.401556 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn"] Oct 30 04:59:21.443008 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.442970 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363a5464-b736-4f6c-800c-38b13778716c-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.443171 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.443083 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.443171 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.443164 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.443288 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.443220 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.443337 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.443317 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.443397 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.443353 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.443498 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.443477 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.443552 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.443527 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqb7\" (UniqueName: \"kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-kube-api-access-nlqb7\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.443611 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.443587 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363a5464-b736-4f6c-800c-38b13778716c-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.545042 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.545004 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.545240 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.545108 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.545463 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.545440 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqb7\" (UniqueName: \"kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-kube-api-access-nlqb7\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.545532 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.545489 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363a5464-b736-4f6c-800c-38b13778716c-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.545532 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.545510 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.545653 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.545558 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.545653 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.545568 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363a5464-b736-4f6c-800c-38b13778716c-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.545753 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.545662 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.545753 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.545730 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.545857 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.545792 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.545935 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.545915 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.546130 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.546102 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.546283 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.546141 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.546283 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.546173 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363a5464-b736-4f6c-800c-38b13778716c-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.548038 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.548016 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.548167 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.548149 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz"] Oct 30 04:59:21.548495 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.548473 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363a5464-b736-4f6c-800c-38b13778716c-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.560833 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.560799 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.572786 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.572756 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz"] Oct 30 04:59:21.573759 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.573644 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqb7\" (UniqueName: \"kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-kube-api-access-nlqb7\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.574969 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.574923 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.647689 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.647603 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.647689 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.647668 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.647915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.647790 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.647915 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.647870 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.648041 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.647951 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.648142 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.648122 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe7c6a91-845f-4333-83fa-2c23d0b94128-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.648197 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.648164 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.648242 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.648227 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.648293 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.648267 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z6sq\" (UniqueName: \"kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-kube-api-access-9z6sq\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.679365 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.679330 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:21.750090 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.750059 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe7c6a91-845f-4333-83fa-2c23d0b94128-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.750233 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.750108 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.750233 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.750138 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.750233 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.750187 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z6sq\" (UniqueName: \"kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-kube-api-access-9z6sq\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.750746 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.750243 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.750746 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.750278 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.750746 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.750341 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.750746 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.750425 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.750746 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.750474 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.751006 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.750901 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe7c6a91-845f-4333-83fa-2c23d0b94128-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.755331 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.755040 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.755331 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.755277 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.755533 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.755469 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.755604 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.755547 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.755983 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.755961 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.757098 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.757077 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.780440 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.777038 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z6sq\" (UniqueName: \"kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-kube-api-access-9z6sq\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.780440 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.780022 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.914957 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.914858 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:21.998779 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:59:21.998726 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363a5464_b736_4f6c_800c_38b13778716c.slice/crio-8faec0360bfeba207cf9ec7cdd02cf607e62aae0ec822a14a9b70d9c1e3be07d WatchSource:0}: Error finding container 8faec0360bfeba207cf9ec7cdd02cf607e62aae0ec822a14a9b70d9c1e3be07d: Status 404 returned error can't find the container with id 8faec0360bfeba207cf9ec7cdd02cf607e62aae0ec822a14a9b70d9c1e3be07d Oct 30 04:59:21.999279 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:21.999246 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn"] Oct 30 04:59:22.001648 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.001614 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:22.001755 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.001679 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:22.001755 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.001707 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:22.214493 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:59:22.214462 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7c6a91_845f_4333_83fa_2c23d0b94128.slice/crio-09e6fd07d522acda4410332dd4a7036bb3dd0b1e577818e581431f72d2c03af5 WatchSource:0}: Error finding container 09e6fd07d522acda4410332dd4a7036bb3dd0b1e577818e581431f72d2c03af5: Status 404 returned error can't find the container with id 09e6fd07d522acda4410332dd4a7036bb3dd0b1e577818e581431f72d2c03af5 Oct 30 04:59:22.216004 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.215978 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz"] Oct 30 04:59:22.217592 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.217556 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:22.217672 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.217641 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:22.217709 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.217686 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:22.360054 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.360014 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" event={"ID":"363a5464-b736-4f6c-800c-38b13778716c","Type":"ContainerStarted","Data":"07eca600f75f9e641e2a95b9105e55bc88852c65779bf795b05125c5c8898a42"} Oct 30 04:59:22.360252 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.360078 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" event={"ID":"363a5464-b736-4f6c-800c-38b13778716c","Type":"ContainerStarted","Data":"8faec0360bfeba207cf9ec7cdd02cf607e62aae0ec822a14a9b70d9c1e3be07d"} Oct 30 04:59:22.362552 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.362514 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" event={"ID":"fe7c6a91-845f-4333-83fa-2c23d0b94128","Type":"ContainerStarted","Data":"f3cfa2ac17ecb8b7d9489f0412304cf0ce43f8bd14def17c71ea991232b27c3a"} Oct 30 04:59:22.363016 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.362575 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" event={"ID":"fe7c6a91-845f-4333-83fa-2c23d0b94128","Type":"ContainerStarted","Data":"09e6fd07d522acda4410332dd4a7036bb3dd0b1e577818e581431f72d2c03af5"} Oct 30 04:59:22.407168 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.407107 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" podStartSLOduration=1.407086174 podStartE2EDuration="1.407086174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:22.402463961 +0000 UTC m=+886.930230746" watchObservedRunningTime="2026-10-30 04:59:22.407086174 +0000 UTC m=+886.934852961" Oct 30 04:59:22.449381 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.448791 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" podStartSLOduration=1.448770589 podStartE2EDuration="1.448770589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:22.445799337 +0000 UTC m=+886.973566123" watchObservedRunningTime="2026-10-30 04:59:22.448770589 +0000 UTC m=+886.976537376" Oct 30 04:59:22.679931 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.679896 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:22.929390 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:22.929322 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:23.681266 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:23.681232 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 04:59:23.681882 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:23.681851 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" podUID="363a5464-b736-4f6c-800c-38b13778716c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 04:59:23.921078 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:23.921039 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 04:59:23.921407 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:23.921346 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" podUID="fe7c6a91-845f-4333-83fa-2c23d0b94128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 04:59:24.679820 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:24.679779 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 04:59:24.680122 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:24.680094 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" podUID="363a5464-b736-4f6c-800c-38b13778716c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 04:59:24.921395 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:24.915593 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 04:59:24.921395 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:24.915651 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" podUID="fe7c6a91-845f-4333-83fa-2c23d0b94128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 04:59:25.682391 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:25.682181 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 04:59:25.682391 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:25.682247 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" podUID="363a5464-b736-4f6c-800c-38b13778716c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 04:59:25.917773 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:25.917738 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 04:59:25.918005 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:25.917979 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" podUID="fe7c6a91-845f-4333-83fa-2c23d0b94128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 04:59:26.006536 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:59:26.006453 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:59:26.493179 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.493146 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt"] Oct 30 04:59:26.497666 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.497642 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.508001 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.507973 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx"] Oct 30 04:59:26.512176 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.512157 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588278 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588332 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588427 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588470 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588518 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588545 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588609 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhckd\" (UniqueName: \"kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-kube-api-access-lhckd\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588674 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2lwp\" (UniqueName: \"kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-kube-api-access-s2lwp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588720 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588752 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588785 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588843 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588887 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588948 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.588972 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.589009 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.589046 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.589396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.589090 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.598322 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.598300 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt"] Oct 30 04:59:26.601785 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.601760 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx"] Oct 30 04:59:26.680643 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.680611 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 04:59:26.681056 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.681027 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" podUID="363a5464-b736-4f6c-800c-38b13778716c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 04:59:26.690339 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.690309 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.690501 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.690376 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.690501 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.690405 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.690501 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.690469 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.690664 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.690510 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.690664 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.690581 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.690780 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.690666 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.690780 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.690707 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.690780 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.690736 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.690780 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.690737 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.690780 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.690759 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.691041 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.690928 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.691041 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.690980 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.691041 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691006 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.691041 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691011 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.691228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691047 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.691228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691059 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.691228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691084 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.691228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691095 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.691228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691105 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.691228 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691211 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.691579 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691271 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.691579 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691297 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.691579 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691384 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhckd\" (UniqueName: \"kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-kube-api-access-lhckd\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.691579 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691443 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lwp\" (UniqueName: \"kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-kube-api-access-s2lwp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.691579 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691454 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.691579 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.691271 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.692321 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.692298 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.693685 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.693664 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.693782 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.693729 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.694013 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.693991 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.694413 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.694394 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.710654 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.710626 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.711888 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.711871 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2lwp\" (UniqueName: \"kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-kube-api-access-s2lwp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.712120 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.712096 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhckd\" (UniqueName: \"kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-kube-api-access-lhckd\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.712579 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.712559 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.814981 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.814950 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:26.903694 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.903555 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:26.938697 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.919742 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 04:59:26.938697 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:26.919798 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" podUID="fe7c6a91-845f-4333-83fa-2c23d0b94128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 04:59:27.291206 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:27.290108 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt"] Oct 30 04:59:27.294848 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:59:27.294304 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53c02d6_a2cb_4d27_b746_10f4a67978b7.slice/crio-444d242516eab07563fedd267443ac73d77dfcbc351093a8bbe944816ddb6483 WatchSource:0}: Error finding container 444d242516eab07563fedd267443ac73d77dfcbc351093a8bbe944816ddb6483: Status 404 returned error can't find the container with id 444d242516eab07563fedd267443ac73d77dfcbc351093a8bbe944816ddb6483 Oct 30 04:59:27.296821 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:27.296789 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:27.296901 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:27.296857 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:27.296901 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:27.296887 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:27.394462 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:27.394026 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx"] Oct 30 04:59:27.408624 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:59:27.408596 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc74df1_27eb_433d_9e9f_0ad481d00df1.slice/crio-977f880a4d585e5c8b8edbb7f610f2b0d4ab0cae560838bfa4a70196a4ff5842 WatchSource:0}: Error finding container 977f880a4d585e5c8b8edbb7f610f2b0d4ab0cae560838bfa4a70196a4ff5842: Status 404 returned error can't find the container with id 977f880a4d585e5c8b8edbb7f610f2b0d4ab0cae560838bfa4a70196a4ff5842 Oct 30 04:59:27.411876 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:27.411513 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:27.411876 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:27.411580 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:27.411876 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:27.411608 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:27.414296 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:27.414138 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" event={"ID":"e53c02d6-a2cb-4d27-b746-10f4a67978b7","Type":"ContainerStarted","Data":"444d242516eab07563fedd267443ac73d77dfcbc351093a8bbe944816ddb6483"} Oct 30 04:59:27.686192 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:27.685695 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 04:59:27.686192 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:27.685750 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" podUID="363a5464-b736-4f6c-800c-38b13778716c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 04:59:27.920383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:27.915471 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 04:59:27.920383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:27.915531 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" podUID="fe7c6a91-845f-4333-83fa-2c23d0b94128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 04:59:28.429042 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.428996 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" event={"ID":"e53c02d6-a2cb-4d27-b746-10f4a67978b7","Type":"ContainerStarted","Data":"3c78bf483706a8240b1864e3a3405aa0cd547f44b1954e4842d5f597d7baff59"} Oct 30 04:59:28.435392 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.432707 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" event={"ID":"7fc74df1-27eb-433d-9e9f-0ad481d00df1","Type":"ContainerStarted","Data":"4fee2c5ea10ba025fec580594655be2a8a3c76b349e98db7edb7c7df27c06d2f"} Oct 30 04:59:28.435392 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.432767 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" event={"ID":"7fc74df1-27eb-433d-9e9f-0ad481d00df1","Type":"ContainerStarted","Data":"977f880a4d585e5c8b8edbb7f610f2b0d4ab0cae560838bfa4a70196a4ff5842"} Oct 30 04:59:28.503552 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.503488 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" podStartSLOduration=2.503469374 podStartE2EDuration="2.503469374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:28.48633027 +0000 UTC m=+893.014097058" watchObservedRunningTime="2026-10-30 04:59:28.503469374 +0000 UTC m=+893.031236160" Oct 30 04:59:28.542467 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.541619 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" podStartSLOduration=2.5415969179999998 podStartE2EDuration="2.541596918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:28.52922142 +0000 UTC m=+893.056988206" watchObservedRunningTime="2026-10-30 04:59:28.541596918 +0000 UTC m=+893.069363707" Oct 30 04:59:28.684977 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.683011 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 04:59:28.684977 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.683095 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" podUID="363a5464-b736-4f6c-800c-38b13778716c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 04:59:28.815346 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.815279 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:28.819810 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.819484 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 04:59:28.819810 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.819547 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 04:59:28.919394 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.918935 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:28.923388 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.920030 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 04:59:28.923388 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.920084 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 04:59:28.923388 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.920652 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 04:59:28.923388 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:28.920692 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" podUID="fe7c6a91-845f-4333-83fa-2c23d0b94128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 04:59:29.777381 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:29.777332 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:29.788379 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:29.786695 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:29.795377 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:29.794483 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 04:59:29.815286 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:29.815256 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 04:59:29.824577 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:29.824540 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 04:59:29.936381 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:29.931656 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 04:59:29.936381 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:29.931707 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 04:59:30.825382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:30.824746 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 04:59:30.825382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:30.824815 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 04:59:30.919382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:30.906651 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 04:59:30.919382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:30.906708 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 04:59:30.978520 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:30.978473 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:59:30.978804 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:30.978770 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" podUID="fe7c6a91-845f-4333-83fa-2c23d0b94128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:31.051405 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:31.051009 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 30 04:59:31.220439 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:31.216462 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:31.224377 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:31.221430 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:31.545438 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:31.541875 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:31.839784 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:31.839733 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 04:59:31.841260 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:31.841225 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 04:59:31.909383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:31.906726 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 04:59:31.909383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:31.906785 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 04:59:31.994383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:31.993517 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:59:31.994383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:31.993581 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" podUID="fe7c6a91-845f-4333-83fa-2c23d0b94128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:32.010382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:32.007683 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:32.010382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:32.009020 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:32.066383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:32.065604 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 04:59:32.245436 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:32.223069 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:32.245436 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:32.223797 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:59:32.267397 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:32.246933 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 04:59:32.333764 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:32.333046 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 04:59:32.819385 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:32.815743 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 04:59:32.819700 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:32.819667 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 04:59:32.905000 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:32.904964 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 04:59:32.905617 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:32.905574 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 04:59:33.815280 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:33.815241 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 04:59:33.815511 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:33.815331 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 04:59:33.904527 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:33.904493 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 04:59:33.904703 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:33.904547 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 04:59:34.815374 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:34.815331 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 04:59:34.815873 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:34.815418 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 04:59:34.904902 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:34.904868 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 04:59:34.905114 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:34.904925 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 04:59:35.725923 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:35.725890 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 04:59:35.728467 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:35.728443 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 04:59:35.816237 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:35.816202 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 04:59:35.816639 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:35.816280 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 04:59:35.904129 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:35.904089 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 04:59:35.904309 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:35.904156 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 04:59:36.624559 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.624516 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw"] Oct 30 04:59:36.632328 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.632302 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.647827 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.647797 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw"] Oct 30 04:59:36.655410 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.655314 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-83e83e\"/\"distinct-w6-83e83e-kserve-self-signed-certs\"" Oct 30 04:59:36.716277 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.716243 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-home\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.716497 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.716413 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9kk2\" (UniqueName: \"kubernetes.io/projected/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-kube-api-access-x9kk2\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.716497 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.716467 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-model-cache\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.716621 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.716514 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-dshm\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.716621 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.716542 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tls-certs\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.716621 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.716610 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tmp-dir\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.815945 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.815553 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 04:59:36.815945 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.815617 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 04:59:36.818330 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.818201 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-dshm\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.818330 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.818250 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tls-certs\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.818330 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.818319 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tmp-dir\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.818904 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.818454 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-home\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.818904 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.818619 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kk2\" (UniqueName: \"kubernetes.io/projected/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-kube-api-access-x9kk2\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.818904 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.818676 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-model-cache\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.828254 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.824579 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-model-cache\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.841997 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.832169 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tmp-dir\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.841997 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.841860 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-home\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.894632 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.894551 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tls-certs\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.896180 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.896152 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-dshm\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.901934 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.901068 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9kk2\" (UniqueName: \"kubernetes.io/projected/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-kube-api-access-x9kk2\") pod \"distinct-w6-83e83e-kserve-5b7f687bff-2vscw\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:36.934213 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.933584 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 04:59:36.934213 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.933644 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 04:59:36.963479 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:36.963446 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:37.278492 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:59:37.278465 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod068d2c70_454e_4ebd_ad63_4c17ac9fc1da.slice/crio-0a8f0cef2184904e8acee8382ddc6511511d2df68ceacb965b85340176c2b213 WatchSource:0}: Error finding container 0a8f0cef2184904e8acee8382ddc6511511d2df68ceacb965b85340176c2b213: Status 404 returned error can't find the container with id 0a8f0cef2184904e8acee8382ddc6511511d2df68ceacb965b85340176c2b213 Oct 30 04:59:37.280468 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:37.280442 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw"] Oct 30 04:59:37.917382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:37.917262 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" event={"ID":"068d2c70-454e-4ebd-ad63-4c17ac9fc1da","Type":"ContainerStarted","Data":"c780a96e3c9a8b8d6563d624e7a5c673861f4004f4330ffbdd4010d6b20d522b"} Oct 30 04:59:37.917382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:37.917322 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" event={"ID":"068d2c70-454e-4ebd-ad63-4c17ac9fc1da","Type":"ContainerStarted","Data":"0a8f0cef2184904e8acee8382ddc6511511d2df68ceacb965b85340176c2b213"} Oct 30 04:59:37.963384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:37.962446 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podStartSLOduration=1.9624287580000002 podStartE2EDuration="1.962428758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:37.957526498 +0000 UTC m=+902.485293284" watchObservedRunningTime="2026-10-30 04:59:37.962428758 +0000 UTC m=+902.490195545" Oct 30 04:59:37.964525 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:37.964482 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:38.172949 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:38.172862 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:38.173396 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:38.173343 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:38.174292 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:38.174265 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 04:59:38.237791 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:38.237758 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:38.238326 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:38.238296 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:38.239269 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:38.239241 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 04:59:46.964792 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:46.964695 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:46.977369 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:46.977342 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:47.011045 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:47.011016 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 04:59:52.000884 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.000851 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7"] Oct 30 04:59:52.005262 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.005240 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.055470 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.055434 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7"] Oct 30 04:59:52.077964 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.077933 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.078138 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.078079 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.078214 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.078176 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.078275 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.078262 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.078331 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.078306 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.078425 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.078333 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.078553 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.078426 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2vz2\" (UniqueName: \"kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-kube-api-access-c2vz2\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.078553 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.078540 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.078704 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.078585 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7600beab-bc24-4634-b008-6aebfb2c71c9-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.179456 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.179425 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.179613 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.179544 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.179613 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.179588 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.179706 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.179659 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.179755 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.179713 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.179867 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.179843 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.179934 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.179905 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2vz2\" (UniqueName: \"kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-kube-api-access-c2vz2\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.179989 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.179954 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.179989 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.179978 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7600beab-bc24-4634-b008-6aebfb2c71c9-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.180093 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.180033 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.180093 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.180033 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.180204 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.180091 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.180204 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.180119 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.180693 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.180674 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7600beab-bc24-4634-b008-6aebfb2c71c9-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.182167 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.182145 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.182458 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.182442 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.199528 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.199507 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.199750 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.199727 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2vz2\" (UniqueName: \"kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-kube-api-access-c2vz2\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.279382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.279277 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd"] Oct 30 04:59:52.284623 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.284601 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.307921 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.307888 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd"] Oct 30 04:59:52.337455 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.337426 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:52.382346 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.382317 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.382346 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.382352 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.382591 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.382446 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.382591 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.382490 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.382591 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.382529 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.382591 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.382578 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hplw9\" (UniqueName: \"kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-kube-api-access-hplw9\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.382827 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.382639 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8656d545-bb21-4db3-ade1-1e164a6c9744-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.382827 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.382724 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.382827 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.382741 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.485623 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.485585 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.485796 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.485634 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.485855 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.485797 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.485855 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.485829 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.485947 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.485860 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.485947 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.485893 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.485947 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.485941 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.486063 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.485971 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hplw9\" (UniqueName: \"kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-kube-api-access-hplw9\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.486063 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.486026 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8656d545-bb21-4db3-ade1-1e164a6c9744-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.486129 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.486102 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.486287 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.486261 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.487610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.486861 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.487610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.486987 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.487610 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.487062 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8656d545-bb21-4db3-ade1-1e164a6c9744-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.488743 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.488714 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.488892 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.488871 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.505538 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.505513 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.505745 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.505728 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hplw9\" (UniqueName: \"kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-kube-api-access-hplw9\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.602120 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.602094 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:52.671909 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.671253 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7"] Oct 30 04:59:52.672074 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:59:52.669082 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7600beab_bc24_4634_b008_6aebfb2c71c9.slice/crio-53561e28b961fd5a7a9655e6e9f3a20127e97038058e3a28ff15eb5c40f9bd86 WatchSource:0}: Error finding container 53561e28b961fd5a7a9655e6e9f3a20127e97038058e3a28ff15eb5c40f9bd86: Status 404 returned error can't find the container with id 53561e28b961fd5a7a9655e6e9f3a20127e97038058e3a28ff15eb5c40f9bd86 Oct 30 04:59:52.674525 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.674487 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:52.674632 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.674572 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:52.674632 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.674618 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:52.941170 ip-10-0-134-105 kubenswrapper[6587]: W1030 04:59:52.941139 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8656d545_bb21_4db3_ade1_1e164a6c9744.slice/crio-7433b3e828d3158116484c4c2f0135c2a8df1f83523b3980154b8a22333bb537 WatchSource:0}: Error finding container 7433b3e828d3158116484c4c2f0135c2a8df1f83523b3980154b8a22333bb537: Status 404 returned error can't find the container with id 7433b3e828d3158116484c4c2f0135c2a8df1f83523b3980154b8a22333bb537 Oct 30 04:59:52.941295 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.941265 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd"] Oct 30 04:59:52.944238 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.944209 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:52.944340 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.944271 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:52.944340 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:52.944301 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 04:59:53.066062 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:53.065948 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" event={"ID":"7600beab-bc24-4634-b008-6aebfb2c71c9","Type":"ContainerStarted","Data":"b98580e24f3b7bd6fde951a55af33cfc592eb321c320404347304a6d6c8d3dbc"} Oct 30 04:59:53.066062 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:53.066017 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" event={"ID":"7600beab-bc24-4634-b008-6aebfb2c71c9","Type":"ContainerStarted","Data":"53561e28b961fd5a7a9655e6e9f3a20127e97038058e3a28ff15eb5c40f9bd86"} Oct 30 04:59:53.069011 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:53.068980 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" event={"ID":"8656d545-bb21-4db3-ade1-1e164a6c9744","Type":"ContainerStarted","Data":"f3204866f7893e97f91ede12e20418c31b632b3a6b3fd6f74a411b124cbd9f33"} Oct 30 04:59:53.069165 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:53.069024 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" event={"ID":"8656d545-bb21-4db3-ade1-1e164a6c9744","Type":"ContainerStarted","Data":"7433b3e828d3158116484c4c2f0135c2a8df1f83523b3980154b8a22333bb537"} Oct 30 04:59:53.110181 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:53.109901 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" podStartSLOduration=2.109883886 podStartE2EDuration="2.109883886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:53.109120429 +0000 UTC m=+917.636887216" watchObservedRunningTime="2026-10-30 04:59:53.109883886 +0000 UTC m=+917.637650673" Oct 30 04:59:53.159282 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:53.158813 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" podStartSLOduration=1.158793388 podStartE2EDuration="1.158793388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:53.158015385 +0000 UTC m=+917.685782174" watchObservedRunningTime="2026-10-30 04:59:53.158793388 +0000 UTC m=+917.686560176" Oct 30 04:59:53.338535 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:53.338501 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 04:59:53.340348 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:53.340319 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 04:59:53.340504 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:53.340397 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 04:59:53.603242 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:53.603156 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 04:59:53.604970 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:53.604944 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 04:59:53.605091 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:53.604991 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" podUID="8656d545-bb21-4db3-ade1-1e164a6c9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 04:59:54.341797 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:54.341754 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 04:59:54.342407 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:54.342348 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 04:59:54.602952 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:54.602867 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 04:59:54.602952 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:54.602916 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" podUID="8656d545-bb21-4db3-ade1-1e164a6c9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 04:59:55.343382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:55.342788 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 04:59:55.343382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:55.342855 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 04:59:55.603502 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:55.603427 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 04:59:55.603707 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:55.603683 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" podUID="8656d545-bb21-4db3-ade1-1e164a6c9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 04:59:56.007439 ip-10-0-134-105 kubenswrapper[6587]: E1030 04:59:56.007339 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 04:59:56.343750 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:56.343712 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 04:59:56.344344 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:56.344311 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 04:59:56.603254 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:56.603000 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 04:59:56.603254 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:56.603050 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" podUID="8656d545-bb21-4db3-ade1-1e164a6c9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 04:59:57.354383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:57.347389 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 04:59:57.354383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:57.347460 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 04:59:57.616456 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:57.616320 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 04:59:57.616756 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:57.616698 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" podUID="8656d545-bb21-4db3-ade1-1e164a6c9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 04:59:58.182990 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:58.181984 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 04:59:58.256421 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:58.250459 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 04:59:58.361382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:58.350459 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 04:59:58.361382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:58.350522 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 04:59:58.513593 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:58.513467 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:58.604052 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:58.602899 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 04:59:58.604052 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:58.602979 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" podUID="8656d545-bb21-4db3-ade1-1e164a6c9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 04:59:59.004383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.003462 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:59.196384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.186048 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:59.196384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.186662 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 04:59:59.205384 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.203625 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:59.342098 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.342052 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:59.449382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.446523 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 04:59:59.449382 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.446588 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 04:59:59.541383 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.502515 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 04:59:59.545380 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.520528 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 04:59:59.567388 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.545955 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:59:59.603777 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.603688 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 04:59:59.604048 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.604019 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" podUID="8656d545-bb21-4db3-ade1-1e164a6c9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 04:59:59.604881 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.604858 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 04:59:59.623310 ip-10-0-134-105 kubenswrapper[6587]: I1030 04:59:59.623284 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:00:00.149473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.149435 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w"] Oct 30 05:00:00.154881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.154857 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" Oct 30 05:00:00.155087 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.155061 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6"] Oct 30 05:00:00.163886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.163856 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98"] Oct 30 05:00:00.166063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.166037 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" Oct 30 05:00:00.169882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.169860 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" Oct 30 05:00:00.172478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.172453 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w"] Oct 30 05:00:00.174815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.174576 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:00:00.177005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.176960 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5"] Oct 30 05:00:00.181343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.181325 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:00:00.185582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.185554 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:00:00.187606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.187588 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" Oct 30 05:00:00.213231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.210167 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm"] Oct 30 05:00:00.233478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.233441 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6"] Oct 30 05:00:00.233705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.233678 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5"] Oct 30 05:00:00.233839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.233827 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm"] Oct 30 05:00:00.234071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.234054 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" Oct 30 05:00:00.244400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.241974 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98"] Oct 30 05:00:00.311403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.311187 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8v5c\" (UniqueName: \"kubernetes.io/projected/ea87894e-d49d-4a93-b8d6-d17908cd68fd-kube-api-access-m8v5c\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w\" (UID: \"ea87894e-d49d-4a93-b8d6-d17908cd68fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" Oct 30 05:00:00.311403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.311250 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888940-jdkc6\" (UID: \"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" Oct 30 05:00:00.311403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.311300 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm\" (UID: \"1c5495cb-8dcb-4caf-85d6-8ca2f770077c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" Oct 30 05:00:00.311403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.311411 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw8zl\" (UniqueName: \"kubernetes.io/projected/0f942106-372c-4185-9a1f-79426c772a3c-kube-api-access-cw8zl\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5\" (UID: \"0f942106-372c-4185-9a1f-79426c772a3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" Oct 30 05:00:00.311675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.311498 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f942106-372c-4185-9a1f-79426c772a3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5\" (UID: \"0f942106-372c-4185-9a1f-79426c772a3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" Oct 30 05:00:00.311675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.311526 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj75n\" (UniqueName: \"kubernetes.io/projected/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-kube-api-access-fj75n\") pod \"maas-api-key-cleanup-29888940-jdkc6\" (UID: \"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" Oct 30 05:00:00.311675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.311610 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98\" (UID: \"e853a0fb-b9ed-413d-8fb0-96c76a43ba69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" Oct 30 05:00:00.311675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.311629 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gssr\" (UniqueName: \"kubernetes.io/projected/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-kube-api-access-6gssr\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98\" (UID: \"e853a0fb-b9ed-413d-8fb0-96c76a43ba69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" Oct 30 05:00:00.311796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.311675 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea87894e-d49d-4a93-b8d6-d17908cd68fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w\" (UID: \"ea87894e-d49d-4a93-b8d6-d17908cd68fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" Oct 30 05:00:00.311796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.311697 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52cgz\" (UniqueName: \"kubernetes.io/projected/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-kube-api-access-52cgz\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm\" (UID: \"1c5495cb-8dcb-4caf-85d6-8ca2f770077c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" Oct 30 05:00:00.338747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.338700 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:00:00.338926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.338768 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 05:00:00.413433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.413160 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8v5c\" (UniqueName: \"kubernetes.io/projected/ea87894e-d49d-4a93-b8d6-d17908cd68fd-kube-api-access-m8v5c\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w\" (UID: \"ea87894e-d49d-4a93-b8d6-d17908cd68fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" Oct 30 05:00:00.413433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.413229 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888940-jdkc6\" (UID: \"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" Oct 30 05:00:00.413433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.413294 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm\" (UID: \"1c5495cb-8dcb-4caf-85d6-8ca2f770077c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" Oct 30 05:00:00.413433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.413379 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8zl\" (UniqueName: \"kubernetes.io/projected/0f942106-372c-4185-9a1f-79426c772a3c-kube-api-access-cw8zl\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5\" (UID: \"0f942106-372c-4185-9a1f-79426c772a3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" Oct 30 05:00:00.413771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.413471 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f942106-372c-4185-9a1f-79426c772a3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5\" (UID: \"0f942106-372c-4185-9a1f-79426c772a3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" Oct 30 05:00:00.413771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.413502 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj75n\" (UniqueName: \"kubernetes.io/projected/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-kube-api-access-fj75n\") pod \"maas-api-key-cleanup-29888940-jdkc6\" (UID: \"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" Oct 30 05:00:00.413771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.413585 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98\" (UID: \"e853a0fb-b9ed-413d-8fb0-96c76a43ba69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" Oct 30 05:00:00.413771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.413610 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gssr\" (UniqueName: \"kubernetes.io/projected/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-kube-api-access-6gssr\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98\" (UID: \"e853a0fb-b9ed-413d-8fb0-96c76a43ba69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" Oct 30 05:00:00.413771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.413691 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea87894e-d49d-4a93-b8d6-d17908cd68fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w\" (UID: \"ea87894e-d49d-4a93-b8d6-d17908cd68fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" Oct 30 05:00:00.413771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.413716 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52cgz\" (UniqueName: \"kubernetes.io/projected/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-kube-api-access-52cgz\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm\" (UID: \"1c5495cb-8dcb-4caf-85d6-8ca2f770077c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" Oct 30 05:00:00.415383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.415211 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888940-jdkc6\" (UID: \"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" Oct 30 05:00:00.415957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.415927 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm\" (UID: \"1c5495cb-8dcb-4caf-85d6-8ca2f770077c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" Oct 30 05:00:00.416868 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.416835 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea87894e-d49d-4a93-b8d6-d17908cd68fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w\" (UID: \"ea87894e-d49d-4a93-b8d6-d17908cd68fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" Oct 30 05:00:00.417039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.417014 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98\" (UID: \"e853a0fb-b9ed-413d-8fb0-96c76a43ba69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" Oct 30 05:00:00.417517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.417487 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f942106-372c-4185-9a1f-79426c772a3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5\" (UID: \"0f942106-372c-4185-9a1f-79426c772a3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" Oct 30 05:00:00.435536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.435475 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8v5c\" (UniqueName: \"kubernetes.io/projected/ea87894e-d49d-4a93-b8d6-d17908cd68fd-kube-api-access-m8v5c\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w\" (UID: \"ea87894e-d49d-4a93-b8d6-d17908cd68fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" Oct 30 05:00:00.436761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.436738 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj75n\" (UniqueName: \"kubernetes.io/projected/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-kube-api-access-fj75n\") pod \"maas-api-key-cleanup-29888940-jdkc6\" (UID: \"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" Oct 30 05:00:00.438666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.438636 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52cgz\" (UniqueName: \"kubernetes.io/projected/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-kube-api-access-52cgz\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm\" (UID: \"1c5495cb-8dcb-4caf-85d6-8ca2f770077c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" Oct 30 05:00:00.440575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.440555 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8zl\" (UniqueName: \"kubernetes.io/projected/0f942106-372c-4185-9a1f-79426c772a3c-kube-api-access-cw8zl\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5\" (UID: \"0f942106-372c-4185-9a1f-79426c772a3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" Oct 30 05:00:00.451598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.451573 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gssr\" (UniqueName: \"kubernetes.io/projected/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-kube-api-access-6gssr\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98\" (UID: \"e853a0fb-b9ed-413d-8fb0-96c76a43ba69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" Oct 30 05:00:00.490317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.489048 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" Oct 30 05:00:00.500947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.500922 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" Oct 30 05:00:00.539599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.539574 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" Oct 30 05:00:00.600090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.599841 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" Oct 30 05:00:00.604201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.603567 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" Oct 30 05:00:00.863007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.862962 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 05:00:00.872233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.865307 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 05:00:00.872233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:00.867986 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 05:00:01.111891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:01.110194 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6"] Oct 30 05:00:01.322638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:01.322202 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" event={"ID":"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d","Type":"ContainerStarted","Data":"85fa9964b0d26c4e225ff12904e19991f857475023f1da36f35c6215c88723c9"} Oct 30 05:00:01.338410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:01.338376 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w"] Oct 30 05:00:01.399257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:01.399230 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98"] Oct 30 05:00:01.416405 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:00:01.416371 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea87894e_d49d_4a93_b8d6_d17908cd68fd.slice/crio-25dd7889b6fa14d0528b04e5c83f1d547abaebc5b622054f9ec1dad1f836ff06 WatchSource:0}: Error finding container 25dd7889b6fa14d0528b04e5c83f1d547abaebc5b622054f9ec1dad1f836ff06: Status 404 returned error can't find the container with id 25dd7889b6fa14d0528b04e5c83f1d547abaebc5b622054f9ec1dad1f836ff06 Oct 30 05:00:01.418215 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:00:01.418170 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode853a0fb_b9ed_413d_8fb0_96c76a43ba69.slice/crio-aeb1a6736f10b91da78321e6fe73d72e0de1da08655476c2d77787b7adfe1e11 WatchSource:0}: Error finding container aeb1a6736f10b91da78321e6fe73d72e0de1da08655476c2d77787b7adfe1e11: Status 404 returned error can't find the container with id aeb1a6736f10b91da78321e6fe73d72e0de1da08655476c2d77787b7adfe1e11 Oct 30 05:00:01.475496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:01.475466 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5"] Oct 30 05:00:01.482761 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:00:01.482734 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f942106_372c_4185_9a1f_79426c772a3c.slice/crio-7fbfeb39ea7c8342b658b9374bc6da48168a1a937875b36974387d6d3c931f88 WatchSource:0}: Error finding container 7fbfeb39ea7c8342b658b9374bc6da48168a1a937875b36974387d6d3c931f88: Status 404 returned error can't find the container with id 7fbfeb39ea7c8342b658b9374bc6da48168a1a937875b36974387d6d3c931f88 Oct 30 05:00:01.522643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:01.522607 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm"] Oct 30 05:00:01.526961 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:00:01.526929 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5495cb_8dcb_4caf_85d6_8ca2f770077c.slice/crio-47d11fa3c4b24907af12f644f01dd14a378a1c2e059735607ff244e1ba2fe192 WatchSource:0}: Error finding container 47d11fa3c4b24907af12f644f01dd14a378a1c2e059735607ff244e1ba2fe192: Status 404 returned error can't find the container with id 47d11fa3c4b24907af12f644f01dd14a378a1c2e059735607ff244e1ba2fe192 Oct 30 05:00:02.340510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.339297 6587 generic.go:356] "Generic (PLEG): container finished" podID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" containerID="9811ce184839fcdedde2f646d6a6218f51ce6c99e1a0997538382a93663fde01" exitCode=60 Oct 30 05:00:02.340510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.339410 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" event={"ID":"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d","Type":"ContainerDied","Data":"9811ce184839fcdedde2f646d6a6218f51ce6c99e1a0997538382a93663fde01"} Oct 30 05:00:02.340510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.339842 6587 scope.go:122] "RemoveContainer" containerID="9811ce184839fcdedde2f646d6a6218f51ce6c99e1a0997538382a93663fde01" Oct 30 05:00:02.345036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.345007 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" event={"ID":"ea87894e-d49d-4a93-b8d6-d17908cd68fd","Type":"ContainerStarted","Data":"4425363a66625cca59e39a5185c31aa7b373e9af3d6cbeadee0d2f3efeebedef"} Oct 30 05:00:02.345274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.345255 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" event={"ID":"ea87894e-d49d-4a93-b8d6-d17908cd68fd","Type":"ContainerStarted","Data":"25dd7889b6fa14d0528b04e5c83f1d547abaebc5b622054f9ec1dad1f836ff06"} Oct 30 05:00:02.350153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.350122 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" event={"ID":"1c5495cb-8dcb-4caf-85d6-8ca2f770077c","Type":"ContainerStarted","Data":"3585f5b1bb700019d772f76d64df8c49babb971095d9a638191c8d80d088bab0"} Oct 30 05:00:02.350280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.350178 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" event={"ID":"1c5495cb-8dcb-4caf-85d6-8ca2f770077c","Type":"ContainerStarted","Data":"47d11fa3c4b24907af12f644f01dd14a378a1c2e059735607ff244e1ba2fe192"} Oct 30 05:00:02.356244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.356216 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" event={"ID":"0f942106-372c-4185-9a1f-79426c772a3c","Type":"ContainerStarted","Data":"1531927c4b4568cf3699c13757077f278e99bbef709bd314dce329fe17fb075d"} Oct 30 05:00:02.356459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.356267 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" event={"ID":"0f942106-372c-4185-9a1f-79426c772a3c","Type":"ContainerStarted","Data":"7fbfeb39ea7c8342b658b9374bc6da48168a1a937875b36974387d6d3c931f88"} Oct 30 05:00:02.362972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.362842 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" event={"ID":"e853a0fb-b9ed-413d-8fb0-96c76a43ba69","Type":"ContainerStarted","Data":"d73cd93ed3b2b6f9702a4eff4ec629faa7efd45127932a827ede7e6327510224"} Oct 30 05:00:02.362972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.362891 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" event={"ID":"e853a0fb-b9ed-413d-8fb0-96c76a43ba69","Type":"ContainerStarted","Data":"aeb1a6736f10b91da78321e6fe73d72e0de1da08655476c2d77787b7adfe1e11"} Oct 30 05:00:02.383542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.383513 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:00:02.383720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.383571 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:02.434534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.434441 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" podStartSLOduration=2.434418549 podStartE2EDuration="2.434418549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:02.430009081 +0000 UTC m=+926.957775867" watchObservedRunningTime="2026-10-30 05:00:02.434418549 +0000 UTC m=+926.962185336" Oct 30 05:00:02.480545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.480423 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" podStartSLOduration=2.480404341 podStartE2EDuration="2.480404341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:02.473135351 +0000 UTC m=+927.000902137" watchObservedRunningTime="2026-10-30 05:00:02.480404341 +0000 UTC m=+927.008171128" Oct 30 05:00:02.597937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.597877 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" podStartSLOduration=2.597857835 podStartE2EDuration="2.597857835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:02.596879516 +0000 UTC m=+927.124646302" watchObservedRunningTime="2026-10-30 05:00:02.597857835 +0000 UTC m=+927.125624622" Oct 30 05:00:02.599245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.599207 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" podStartSLOduration=2.599194716 podStartE2EDuration="2.599194716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:02.532776959 +0000 UTC m=+927.060543750" watchObservedRunningTime="2026-10-30 05:00:02.599194716 +0000 UTC m=+927.126961522" Oct 30 05:00:02.651183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.651073 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 05:00:02.651752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.651721 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 05:00:02.653450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:02.653426 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 05:00:03.371584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.371548 6587 generic.go:356] "Generic (PLEG): container finished" podID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" containerID="e6efbcfdcfe2ce88ab4cd0426e0a8f6cc73565b424d8967fb2093e0748b6cf0e" exitCode=60 Oct 30 05:00:03.371840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.371628 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" event={"ID":"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d","Type":"ContainerDied","Data":"e6efbcfdcfe2ce88ab4cd0426e0a8f6cc73565b424d8967fb2093e0748b6cf0e"} Oct 30 05:00:03.371840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.371680 6587 scope.go:122] "RemoveContainer" containerID="9811ce184839fcdedde2f646d6a6218f51ce6c99e1a0997538382a93663fde01" Oct 30 05:00:03.371973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.371953 6587 scope.go:122] "RemoveContainer" containerID="e6efbcfdcfe2ce88ab4cd0426e0a8f6cc73565b424d8967fb2093e0748b6cf0e" Oct 30 05:00:03.372324 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:03.372297 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888940-jdkc6_odh-ai-gateway-infra(2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" podUID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" Oct 30 05:00:03.373316 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.373293 6587 generic.go:356] "Generic (PLEG): container finished" podID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" containerID="4425363a66625cca59e39a5185c31aa7b373e9af3d6cbeadee0d2f3efeebedef" exitCode=60 Oct 30 05:00:03.373435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.373395 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" event={"ID":"ea87894e-d49d-4a93-b8d6-d17908cd68fd","Type":"ContainerDied","Data":"4425363a66625cca59e39a5185c31aa7b373e9af3d6cbeadee0d2f3efeebedef"} Oct 30 05:00:03.373643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.373628 6587 scope.go:122] "RemoveContainer" containerID="4425363a66625cca59e39a5185c31aa7b373e9af3d6cbeadee0d2f3efeebedef" Oct 30 05:00:03.375123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.375104 6587 generic.go:356] "Generic (PLEG): container finished" podID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" containerID="3585f5b1bb700019d772f76d64df8c49babb971095d9a638191c8d80d088bab0" exitCode=60 Oct 30 05:00:03.375184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.375134 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" event={"ID":"1c5495cb-8dcb-4caf-85d6-8ca2f770077c","Type":"ContainerDied","Data":"3585f5b1bb700019d772f76d64df8c49babb971095d9a638191c8d80d088bab0"} Oct 30 05:00:03.375521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.375500 6587 scope.go:122] "RemoveContainer" containerID="3585f5b1bb700019d772f76d64df8c49babb971095d9a638191c8d80d088bab0" Oct 30 05:00:03.378228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.378090 6587 generic.go:356] "Generic (PLEG): container finished" podID="0f942106-372c-4185-9a1f-79426c772a3c" containerID="1531927c4b4568cf3699c13757077f278e99bbef709bd314dce329fe17fb075d" exitCode=60 Oct 30 05:00:03.378627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.378350 6587 scope.go:122] "RemoveContainer" containerID="1531927c4b4568cf3699c13757077f278e99bbef709bd314dce329fe17fb075d" Oct 30 05:00:03.378627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.378206 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" event={"ID":"0f942106-372c-4185-9a1f-79426c772a3c","Type":"ContainerDied","Data":"1531927c4b4568cf3699c13757077f278e99bbef709bd314dce329fe17fb075d"} Oct 30 05:00:03.380209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.380186 6587 generic.go:356] "Generic (PLEG): container finished" podID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" containerID="d73cd93ed3b2b6f9702a4eff4ec629faa7efd45127932a827ede7e6327510224" exitCode=60 Oct 30 05:00:03.380301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.380274 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" event={"ID":"e853a0fb-b9ed-413d-8fb0-96c76a43ba69","Type":"ContainerDied","Data":"d73cd93ed3b2b6f9702a4eff4ec629faa7efd45127932a827ede7e6327510224"} Oct 30 05:00:03.380528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:03.380492 6587 scope.go:122] "RemoveContainer" containerID="d73cd93ed3b2b6f9702a4eff4ec629faa7efd45127932a827ede7e6327510224" Oct 30 05:00:04.389076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:04.389039 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" event={"ID":"0f942106-372c-4185-9a1f-79426c772a3c","Type":"ContainerStarted","Data":"931487f6a33c190e03fcd4caf5311d47acac1d6ec0c623e198a3eee570586242"} Oct 30 05:00:04.390967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:04.390939 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" event={"ID":"e853a0fb-b9ed-413d-8fb0-96c76a43ba69","Type":"ContainerStarted","Data":"ae32795acb6a969c82e6694053093997d09870062411887877cddc501aeb7129"} Oct 30 05:00:04.392707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:04.392689 6587 scope.go:122] "RemoveContainer" containerID="e6efbcfdcfe2ce88ab4cd0426e0a8f6cc73565b424d8967fb2093e0748b6cf0e" Oct 30 05:00:04.392995 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:04.392973 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888940-jdkc6_odh-ai-gateway-infra(2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" podUID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" Oct 30 05:00:04.394411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:04.394384 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" event={"ID":"ea87894e-d49d-4a93-b8d6-d17908cd68fd","Type":"ContainerStarted","Data":"942a11ca3079e14ae6b02c1f5646223ecb898e58b9d171531dccb35917d9b4ef"} Oct 30 05:00:04.396388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:04.396342 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" event={"ID":"1c5495cb-8dcb-4caf-85d6-8ca2f770077c","Type":"ContainerStarted","Data":"651774454c552e6e8266b686602ed7f4442d9a6aaec19f930f597af2090f0355"} Oct 30 05:00:05.404592 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.404553 6587 generic.go:356] "Generic (PLEG): container finished" podID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" containerID="942a11ca3079e14ae6b02c1f5646223ecb898e58b9d171531dccb35917d9b4ef" exitCode=60 Oct 30 05:00:05.405029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.404637 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" event={"ID":"ea87894e-d49d-4a93-b8d6-d17908cd68fd","Type":"ContainerDied","Data":"942a11ca3079e14ae6b02c1f5646223ecb898e58b9d171531dccb35917d9b4ef"} Oct 30 05:00:05.405029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.404698 6587 scope.go:122] "RemoveContainer" containerID="4425363a66625cca59e39a5185c31aa7b373e9af3d6cbeadee0d2f3efeebedef" Oct 30 05:00:05.405029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.405003 6587 scope.go:122] "RemoveContainer" containerID="942a11ca3079e14ae6b02c1f5646223ecb898e58b9d171531dccb35917d9b4ef" Oct 30 05:00:05.405344 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:05.405324 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w_odh-ai-gateway-infra(ea87894e-d49d-4a93-b8d6-d17908cd68fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" podUID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" Oct 30 05:00:05.406771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.406749 6587 generic.go:356] "Generic (PLEG): container finished" podID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" containerID="651774454c552e6e8266b686602ed7f4442d9a6aaec19f930f597af2090f0355" exitCode=60 Oct 30 05:00:05.407246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.406774 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" event={"ID":"1c5495cb-8dcb-4caf-85d6-8ca2f770077c","Type":"ContainerDied","Data":"651774454c552e6e8266b686602ed7f4442d9a6aaec19f930f597af2090f0355"} Oct 30 05:00:05.407246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.407111 6587 scope.go:122] "RemoveContainer" containerID="651774454c552e6e8266b686602ed7f4442d9a6aaec19f930f597af2090f0355" Oct 30 05:00:05.407483 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:05.407455 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm_odh-ai-gateway-infra(1c5495cb-8dcb-4caf-85d6-8ca2f770077c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" podUID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" Oct 30 05:00:05.408803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.408779 6587 generic.go:356] "Generic (PLEG): container finished" podID="0f942106-372c-4185-9a1f-79426c772a3c" containerID="931487f6a33c190e03fcd4caf5311d47acac1d6ec0c623e198a3eee570586242" exitCode=60 Oct 30 05:00:05.408879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.408854 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" event={"ID":"0f942106-372c-4185-9a1f-79426c772a3c","Type":"ContainerDied","Data":"931487f6a33c190e03fcd4caf5311d47acac1d6ec0c623e198a3eee570586242"} Oct 30 05:00:05.409089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.409075 6587 scope.go:122] "RemoveContainer" containerID="931487f6a33c190e03fcd4caf5311d47acac1d6ec0c623e198a3eee570586242" Oct 30 05:00:05.409395 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:05.409370 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5_odh-ai-gateway-infra(0f942106-372c-4185-9a1f-79426c772a3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" podUID="0f942106-372c-4185-9a1f-79426c772a3c" Oct 30 05:00:05.410746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.410723 6587 generic.go:356] "Generic (PLEG): container finished" podID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" containerID="ae32795acb6a969c82e6694053093997d09870062411887877cddc501aeb7129" exitCode=60 Oct 30 05:00:05.410835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.410767 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" event={"ID":"e853a0fb-b9ed-413d-8fb0-96c76a43ba69","Type":"ContainerDied","Data":"ae32795acb6a969c82e6694053093997d09870062411887877cddc501aeb7129"} Oct 30 05:00:05.411047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.411032 6587 scope.go:122] "RemoveContainer" containerID="ae32795acb6a969c82e6694053093997d09870062411887877cddc501aeb7129" Oct 30 05:00:05.411338 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:05.411318 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98_odh-ai-gateway-infra(e853a0fb-b9ed-413d-8fb0-96c76a43ba69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" podUID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" Oct 30 05:00:05.418006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.417986 6587 scope.go:122] "RemoveContainer" containerID="3585f5b1bb700019d772f76d64df8c49babb971095d9a638191c8d80d088bab0" Oct 30 05:00:05.432300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.432280 6587 scope.go:122] "RemoveContainer" containerID="1531927c4b4568cf3699c13757077f278e99bbef709bd314dce329fe17fb075d" Oct 30 05:00:05.446412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:05.446390 6587 scope.go:122] "RemoveContainer" containerID="d73cd93ed3b2b6f9702a4eff4ec629faa7efd45127932a827ede7e6327510224" Oct 30 05:00:06.420929 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:06.420900 6587 scope.go:122] "RemoveContainer" containerID="651774454c552e6e8266b686602ed7f4442d9a6aaec19f930f597af2090f0355" Oct 30 05:00:06.421456 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:06.421304 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm_odh-ai-gateway-infra(1c5495cb-8dcb-4caf-85d6-8ca2f770077c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" podUID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" Oct 30 05:00:06.423076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:06.423051 6587 scope.go:122] "RemoveContainer" containerID="931487f6a33c190e03fcd4caf5311d47acac1d6ec0c623e198a3eee570586242" Oct 30 05:00:06.423445 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:06.423416 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5_odh-ai-gateway-infra(0f942106-372c-4185-9a1f-79426c772a3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" podUID="0f942106-372c-4185-9a1f-79426c772a3c" Oct 30 05:00:06.424986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:06.424966 6587 scope.go:122] "RemoveContainer" containerID="ae32795acb6a969c82e6694053093997d09870062411887877cddc501aeb7129" Oct 30 05:00:06.425276 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:06.425255 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98_odh-ai-gateway-infra(e853a0fb-b9ed-413d-8fb0-96c76a43ba69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" podUID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" Oct 30 05:00:06.426792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:06.426771 6587 scope.go:122] "RemoveContainer" containerID="942a11ca3079e14ae6b02c1f5646223ecb898e58b9d171531dccb35917d9b4ef" Oct 30 05:00:06.427086 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:06.427064 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w_odh-ai-gateway-infra(ea87894e-d49d-4a93-b8d6-d17908cd68fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" podUID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" Oct 30 05:00:07.038312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.038278 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g"] Oct 30 05:00:07.044179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.044153 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.084772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.084733 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g"] Oct 30 05:00:07.102130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.102095 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf37513-1094-4148-9858-2e716307c4a0-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.102294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.102153 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.102393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.102374 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.102480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.102436 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf37513-1094-4148-9858-2e716307c4a0-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.102533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.102484 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.102533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.102510 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.102642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.102553 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.102698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.102678 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.102757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.102738 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqdfh\" (UniqueName: \"kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-kube-api-access-qqdfh\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.203896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.203861 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.203896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.203901 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf37513-1094-4148-9858-2e716307c4a0-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.204124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.203942 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.204124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.203969 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.204124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.203988 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.204124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.204044 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.204124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.204080 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqdfh\" (UniqueName: \"kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-kube-api-access-qqdfh\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.204397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.204153 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf37513-1094-4148-9858-2e716307c4a0-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.204397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.204180 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.204397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.204316 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.204567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.204456 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.204567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.204517 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.204758 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.204736 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf37513-1094-4148-9858-2e716307c4a0-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.204821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.204772 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.206433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.206409 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.206676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.206657 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf37513-1094-4148-9858-2e716307c4a0-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.224301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.224270 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.224474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.224349 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqdfh\" (UniqueName: \"kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-kube-api-access-qqdfh\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.361107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.361072 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:07.671763 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:00:07.671732 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf37513_1094_4148_9858_2e716307c4a0.slice/crio-90f093f59d32de154c17bc7147023abd68e7570d1b6032ef1cf8e87a108cb705 WatchSource:0}: Error finding container 90f093f59d32de154c17bc7147023abd68e7570d1b6032ef1cf8e87a108cb705: Status 404 returned error can't find the container with id 90f093f59d32de154c17bc7147023abd68e7570d1b6032ef1cf8e87a108cb705 Oct 30 05:00:07.673125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.673094 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g"] Oct 30 05:00:07.674971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.674940 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:07.675048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.675001 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:07.675048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:07.675034 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:08.457191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:08.457150 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" event={"ID":"6cf37513-1094-4148-9858-2e716307c4a0","Type":"ContainerStarted","Data":"02d9688b3cb6da5ae290fac045ca19b5433f5e8f99b39fd33b29373cdea73ec2"} Oct 30 05:00:08.457416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:08.457220 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" event={"ID":"6cf37513-1094-4148-9858-2e716307c4a0","Type":"ContainerStarted","Data":"90f093f59d32de154c17bc7147023abd68e7570d1b6032ef1cf8e87a108cb705"} Oct 30 05:00:08.547910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:08.547842 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" podStartSLOduration=1.547819591 podStartE2EDuration="1.547819591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:08.515008335 +0000 UTC m=+933.042775121" watchObservedRunningTime="2026-10-30 05:00:08.547819591 +0000 UTC m=+933.075586465" Oct 30 05:00:09.361787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:09.361754 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:09.363620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:09.363594 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 05:00:09.363738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:09.363643 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" podUID="6cf37513-1094-4148-9858-2e716307c4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 05:00:10.363956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:10.362777 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 05:00:10.363956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:10.362842 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" podUID="6cf37513-1094-4148-9858-2e716307c4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 05:00:11.362250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:11.362211 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 05:00:11.362457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:11.362288 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" podUID="6cf37513-1094-4148-9858-2e716307c4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 05:00:12.285801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.285767 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv"] Oct 30 05:00:12.291690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.291668 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" Oct 30 05:00:12.312396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.311304 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv"] Oct 30 05:00:12.320544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.320518 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:00:12.366782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.366341 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 05:00:12.366782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.366430 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" podUID="6cf37513-1094-4148-9858-2e716307c4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 05:00:12.404233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.404196 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ac2012-plddv\" (UID: \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" Oct 30 05:00:12.404393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.404248 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ac2012-plddv\" (UID: \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" Oct 30 05:00:12.404393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.404316 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4x4x\" (UniqueName: \"kubernetes.io/projected/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-kube-api-access-s4x4x\") pod \"maas-api-revoke-sub-e0ac2012-plddv\" (UID: \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" Oct 30 05:00:12.505718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.505684 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ac2012-plddv\" (UID: \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" Oct 30 05:00:12.505718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.505719 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4x4x\" (UniqueName: \"kubernetes.io/projected/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-kube-api-access-s4x4x\") pod \"maas-api-revoke-sub-e0ac2012-plddv\" (UID: \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" Oct 30 05:00:12.506005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.505987 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ac2012-plddv\" (UID: \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" Oct 30 05:00:12.506615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.506591 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ac2012-plddv\" (UID: \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" Oct 30 05:00:12.508078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.508054 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ac2012-plddv\" (UID: \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" Oct 30 05:00:12.553233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.553197 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4x4x\" (UniqueName: \"kubernetes.io/projected/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-kube-api-access-s4x4x\") pod \"maas-api-revoke-sub-e0ac2012-plddv\" (UID: \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" Oct 30 05:00:12.606420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.606388 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" Oct 30 05:00:12.955652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:12.955612 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv"] Oct 30 05:00:13.363278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:13.363251 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 05:00:13.363682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:13.363303 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" podUID="6cf37513-1094-4148-9858-2e716307c4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 05:00:13.497319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:13.497274 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" event={"ID":"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07","Type":"ContainerStarted","Data":"864b2d147928af8ce8508ab4b30947966c1f773207226f433c035647ba07eee2"} Oct 30 05:00:13.497521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:13.497337 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" event={"ID":"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07","Type":"ContainerStarted","Data":"1b094cd9c216447a03576d057affcfab23e310dbe7def27d663912a044f344fc"} Oct 30 05:00:13.555090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:13.555022 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" podStartSLOduration=1.555000191 podStartE2EDuration="1.555000191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:13.553955959 +0000 UTC m=+938.081722758" watchObservedRunningTime="2026-10-30 05:00:13.555000191 +0000 UTC m=+938.082766977" Oct 30 05:00:14.361816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:14.361777 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 05:00:14.361995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:14.361843 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" podUID="6cf37513-1094-4148-9858-2e716307c4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 05:00:14.508022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:14.507988 6587 generic.go:356] "Generic (PLEG): container finished" podID="79d39cd0-5c20-4f5c-acc3-6a63bebfcc07" containerID="864b2d147928af8ce8508ab4b30947966c1f773207226f433c035647ba07eee2" exitCode=0 Oct 30 05:00:14.508453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:14.508060 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" event={"ID":"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07","Type":"ContainerDied","Data":"864b2d147928af8ce8508ab4b30947966c1f773207226f433c035647ba07eee2"} Oct 30 05:00:14.574599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:14.574569 6587 scope.go:122] "RemoveContainer" containerID="e6efbcfdcfe2ce88ab4cd0426e0a8f6cc73565b424d8967fb2093e0748b6cf0e" Oct 30 05:00:15.366028 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.365997 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:15.366476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.366462 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:15.367154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.367128 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:00:15.518968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.518888 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" event={"ID":"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d","Type":"ContainerStarted","Data":"4bbeb21ee6dc4887a3cdc014889993e0f38db865611daf1c181bdaae8910f3ac"} Oct 30 05:00:15.560181 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.560112 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" podStartSLOduration=15.560091544 podStartE2EDuration="15.560091544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:15.558248319 +0000 UTC m=+940.086015106" watchObservedRunningTime="2026-10-30 05:00:15.560091544 +0000 UTC m=+940.087858330" Oct 30 05:00:15.690427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.690399 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" Oct 30 05:00:15.783139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.783060 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-openshift-ca\") pod \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\" (UID: \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\") " Oct 30 05:00:15.783341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.783218 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-kube-api-access-s4x4x\" (UniqueName: \"kubernetes.io/projected/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-kube-api-access-s4x4x\") pod \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\" (UID: \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\") " Oct 30 05:00:15.783437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.783332 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-xks-ca\") pod \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\" (UID: \"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07\") " Oct 30 05:00:15.783491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.783462 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-openshift-ca" pod "79d39cd0-5c20-4f5c-acc3-6a63bebfcc07" (UID: "79d39cd0-5c20-4f5c-acc3-6a63bebfcc07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:15.784052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.784026 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:15.785435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.785408 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-kube-api-access-s4x4x" pod "79d39cd0-5c20-4f5c-acc3-6a63bebfcc07" (UID: "79d39cd0-5c20-4f5c-acc3-6a63bebfcc07"). InnerVolumeSpecName "kube-api-access-s4x4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:15.785724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.785566 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-xks-ca" pod "79d39cd0-5c20-4f5c-acc3-6a63bebfcc07" (UID: "79d39cd0-5c20-4f5c-acc3-6a63bebfcc07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:00:15.885792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.885752 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4x4x\" (UniqueName: \"kubernetes.io/projected/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-kube-api-access-s4x4x\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:15.885792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:15.885792 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:16.529760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:16.529729 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" Oct 30 05:00:16.529760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:16.529748 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv" event={"ID":"79d39cd0-5c20-4f5c-acc3-6a63bebfcc07","Type":"ContainerDied","Data":"1b094cd9c216447a03576d057affcfab23e310dbe7def27d663912a044f344fc"} Oct 30 05:00:16.530245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:16.529800 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b094cd9c216447a03576d057affcfab23e310dbe7def27d663912a044f344fc" Oct 30 05:00:16.531964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:16.531932 6587 generic.go:356] "Generic (PLEG): container finished" podID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" containerID="4bbeb21ee6dc4887a3cdc014889993e0f38db865611daf1c181bdaae8910f3ac" exitCode=60 Oct 30 05:00:16.532111 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:16.531960 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" event={"ID":"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d","Type":"ContainerDied","Data":"4bbeb21ee6dc4887a3cdc014889993e0f38db865611daf1c181bdaae8910f3ac"} Oct 30 05:00:16.532111 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:16.532017 6587 scope.go:122] "RemoveContainer" containerID="e6efbcfdcfe2ce88ab4cd0426e0a8f6cc73565b424d8967fb2093e0748b6cf0e" Oct 30 05:00:16.532923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:16.532896 6587 scope.go:122] "RemoveContainer" containerID="4bbeb21ee6dc4887a3cdc014889993e0f38db865611daf1c181bdaae8910f3ac" Oct 30 05:00:16.533275 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:16.533244 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888940-jdkc6_odh-ai-gateway-infra(2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" podUID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" Oct 30 05:00:16.569550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:16.569523 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6"] Oct 30 05:00:16.573710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:16.573688 6587 scope.go:122] "RemoveContainer" containerID="931487f6a33c190e03fcd4caf5311d47acac1d6ec0c623e198a3eee570586242" Oct 30 05:00:16.578247 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:16.578218 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d39cd0_5c20_4f5c_acc3_6a63bebfcc07.slice/crio-1b094cd9c216447a03576d057affcfab23e310dbe7def27d663912a044f344fc\": RecentStats: unable to find data in memory cache]" Oct 30 05:00:17.544350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:17.544323 6587 generic.go:356] "Generic (PLEG): container finished" podID="0f942106-372c-4185-9a1f-79426c772a3c" containerID="db65fcad7b72ca1c84fa5a002d2adf37a6ef0ee6ccd0a4db95dbe3bd213c6ba3" exitCode=60 Oct 30 05:00:17.544686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:17.544410 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" event={"ID":"0f942106-372c-4185-9a1f-79426c772a3c","Type":"ContainerDied","Data":"db65fcad7b72ca1c84fa5a002d2adf37a6ef0ee6ccd0a4db95dbe3bd213c6ba3"} Oct 30 05:00:17.544686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:17.544513 6587 scope.go:122] "RemoveContainer" containerID="931487f6a33c190e03fcd4caf5311d47acac1d6ec0c623e198a3eee570586242" Oct 30 05:00:17.545027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:17.545007 6587 scope.go:122] "RemoveContainer" containerID="db65fcad7b72ca1c84fa5a002d2adf37a6ef0ee6ccd0a4db95dbe3bd213c6ba3" Oct 30 05:00:17.545439 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:17.545409 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5_odh-ai-gateway-infra(0f942106-372c-4185-9a1f-79426c772a3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" podUID="0f942106-372c-4185-9a1f-79426c772a3c" Oct 30 05:00:17.574207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:17.574183 6587 scope.go:122] "RemoveContainer" containerID="ae32795acb6a969c82e6694053093997d09870062411887877cddc501aeb7129" Oct 30 05:00:17.576829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:17.574511 6587 scope.go:122] "RemoveContainer" containerID="942a11ca3079e14ae6b02c1f5646223ecb898e58b9d171531dccb35917d9b4ef" Oct 30 05:00:17.787696 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:17.787670 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" Oct 30 05:00:17.926754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:17.926666 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-maas-api-service-ca\") pod \"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d\" (UID: \"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d\") " Oct 30 05:00:17.926754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:17.926745 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-kube-api-access-fj75n\" (UniqueName: \"kubernetes.io/projected/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-kube-api-access-fj75n\") pod \"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d\" (UID: \"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d\") " Oct 30 05:00:17.927032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:17.927009 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-maas-api-service-ca" pod "2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" (UID: "2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:17.927386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:17.927368 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:18.015864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.015793 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-kube-api-access-fj75n" pod "2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" (UID: "2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d"). InnerVolumeSpecName "kube-api-access-fj75n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:18.028608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.028585 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj75n\" (UniqueName: \"kubernetes.io/projected/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d-kube-api-access-fj75n\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:18.553382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.553331 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" Oct 30 05:00:18.553382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.553331 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6" event={"ID":"2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d","Type":"ContainerDied","Data":"85fa9964b0d26c4e225ff12904e19991f857475023f1da36f35c6215c88723c9"} Oct 30 05:00:18.553854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.553433 6587 scope.go:122] "RemoveContainer" containerID="4bbeb21ee6dc4887a3cdc014889993e0f38db865611daf1c181bdaae8910f3ac" Oct 30 05:00:18.555775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.555752 6587 generic.go:356] "Generic (PLEG): container finished" podID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" containerID="845b0a4b29ad78d9cefd5a20d76c4a0cbc0cf8da1ec5bd6f4766813c4083779d" exitCode=60 Oct 30 05:00:18.555959 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.555830 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" event={"ID":"ea87894e-d49d-4a93-b8d6-d17908cd68fd","Type":"ContainerDied","Data":"845b0a4b29ad78d9cefd5a20d76c4a0cbc0cf8da1ec5bd6f4766813c4083779d"} Oct 30 05:00:18.556300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.556275 6587 scope.go:122] "RemoveContainer" containerID="845b0a4b29ad78d9cefd5a20d76c4a0cbc0cf8da1ec5bd6f4766813c4083779d" Oct 30 05:00:18.556702 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:18.556674 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w_odh-ai-gateway-infra(ea87894e-d49d-4a93-b8d6-d17908cd68fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" podUID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" Oct 30 05:00:18.561130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.561110 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" event={"ID":"e853a0fb-b9ed-413d-8fb0-96c76a43ba69","Type":"ContainerStarted","Data":"761975eb30db42bbee863fd9a28cfeb2bff0e110688daefd9568cf240cfee490"} Oct 30 05:00:18.574117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.574097 6587 scope.go:122] "RemoveContainer" containerID="651774454c552e6e8266b686602ed7f4442d9a6aaec19f930f597af2090f0355" Oct 30 05:00:18.596732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.596661 6587 scope.go:122] "RemoveContainer" containerID="942a11ca3079e14ae6b02c1f5646223ecb898e58b9d171531dccb35917d9b4ef" Oct 30 05:00:18.633839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.633812 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5"] Oct 30 05:00:18.689301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.689219 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6"] Oct 30 05:00:18.695002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.694977 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888940-jdkc6"] Oct 30 05:00:18.840247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.840222 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" Oct 30 05:00:18.943845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.943770 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f942106-372c-4185-9a1f-79426c772a3c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f942106-372c-4185-9a1f-79426c772a3c-maas-api-service-ca\") pod \"0f942106-372c-4185-9a1f-79426c772a3c\" (UID: \"0f942106-372c-4185-9a1f-79426c772a3c\") " Oct 30 05:00:18.944006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.943988 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f942106-372c-4185-9a1f-79426c772a3c-kube-api-access-cw8zl\" (UniqueName: \"kubernetes.io/projected/0f942106-372c-4185-9a1f-79426c772a3c-kube-api-access-cw8zl\") pod \"0f942106-372c-4185-9a1f-79426c772a3c\" (UID: \"0f942106-372c-4185-9a1f-79426c772a3c\") " Oct 30 05:00:18.944163 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.944138 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f942106-372c-4185-9a1f-79426c772a3c-maas-api-service-ca" pod "0f942106-372c-4185-9a1f-79426c772a3c" (UID: "0f942106-372c-4185-9a1f-79426c772a3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:18.944451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.944431 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f942106-372c-4185-9a1f-79426c772a3c-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:18.946254 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:18.946229 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f942106-372c-4185-9a1f-79426c772a3c-kube-api-access-cw8zl" pod "0f942106-372c-4185-9a1f-79426c772a3c" (UID: "0f942106-372c-4185-9a1f-79426c772a3c"). InnerVolumeSpecName "kube-api-access-cw8zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:19.046223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.046195 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw8zl\" (UniqueName: \"kubernetes.io/projected/0f942106-372c-4185-9a1f-79426c772a3c-kube-api-access-cw8zl\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:19.579390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.579328 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" Oct 30 05:00:19.581407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.581348 6587 generic.go:356] "Generic (PLEG): container finished" podID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" containerID="761975eb30db42bbee863fd9a28cfeb2bff0e110688daefd9568cf240cfee490" exitCode=60 Oct 30 05:00:19.583991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.583966 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" path="/var/lib/kubelet/pods/2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d/volumes" Oct 30 05:00:19.584567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.584543 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" event={"ID":"1c5495cb-8dcb-4caf-85d6-8ca2f770077c","Type":"ContainerStarted","Data":"a0ca7cf8af15a7c53169206e1d36b5848f814f79637b951f4e14133b7457ae61"} Oct 30 05:00:19.584648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.584584 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5" event={"ID":"0f942106-372c-4185-9a1f-79426c772a3c","Type":"ContainerDied","Data":"7fbfeb39ea7c8342b658b9374bc6da48168a1a937875b36974387d6d3c931f88"} Oct 30 05:00:19.584648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.584606 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" event={"ID":"e853a0fb-b9ed-413d-8fb0-96c76a43ba69","Type":"ContainerDied","Data":"761975eb30db42bbee863fd9a28cfeb2bff0e110688daefd9568cf240cfee490"} Oct 30 05:00:19.584726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.584675 6587 scope.go:122] "RemoveContainer" containerID="db65fcad7b72ca1c84fa5a002d2adf37a6ef0ee6ccd0a4db95dbe3bd213c6ba3" Oct 30 05:00:19.584953 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.584935 6587 scope.go:122] "RemoveContainer" containerID="761975eb30db42bbee863fd9a28cfeb2bff0e110688daefd9568cf240cfee490" Oct 30 05:00:19.585241 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:19.585219 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98_odh-ai-gateway-infra(e853a0fb-b9ed-413d-8fb0-96c76a43ba69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" podUID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" Oct 30 05:00:19.597563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.597546 6587 scope.go:122] "RemoveContainer" containerID="ae32795acb6a969c82e6694053093997d09870062411887877cddc501aeb7129" Oct 30 05:00:19.622328 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.622299 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w"] Oct 30 05:00:19.634830 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.634807 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5"] Oct 30 05:00:19.646916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.646889 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888940-kbsg5"] Oct 30 05:00:19.663815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.663785 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98"] Oct 30 05:00:19.806079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.806055 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" Oct 30 05:00:19.957443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.957414 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea87894e-d49d-4a93-b8d6-d17908cd68fd-kube-api-access-m8v5c\" (UniqueName: \"kubernetes.io/projected/ea87894e-d49d-4a93-b8d6-d17908cd68fd-kube-api-access-m8v5c\") pod \"ea87894e-d49d-4a93-b8d6-d17908cd68fd\" (UID: \"ea87894e-d49d-4a93-b8d6-d17908cd68fd\") " Oct 30 05:00:19.957609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.957581 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea87894e-d49d-4a93-b8d6-d17908cd68fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea87894e-d49d-4a93-b8d6-d17908cd68fd-maas-api-service-ca\") pod \"ea87894e-d49d-4a93-b8d6-d17908cd68fd\" (UID: \"ea87894e-d49d-4a93-b8d6-d17908cd68fd\") " Oct 30 05:00:19.958013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.957979 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea87894e-d49d-4a93-b8d6-d17908cd68fd-maas-api-service-ca" pod "ea87894e-d49d-4a93-b8d6-d17908cd68fd" (UID: "ea87894e-d49d-4a93-b8d6-d17908cd68fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:19.958252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.958233 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea87894e-d49d-4a93-b8d6-d17908cd68fd-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:19.959643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:19.959617 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea87894e-d49d-4a93-b8d6-d17908cd68fd-kube-api-access-m8v5c" pod "ea87894e-d49d-4a93-b8d6-d17908cd68fd" (UID: "ea87894e-d49d-4a93-b8d6-d17908cd68fd"). InnerVolumeSpecName "kube-api-access-m8v5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:20.059672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.059645 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8v5c\" (UniqueName: \"kubernetes.io/projected/ea87894e-d49d-4a93-b8d6-d17908cd68fd-kube-api-access-m8v5c\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:20.594429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.594391 6587 generic.go:356] "Generic (PLEG): container finished" podID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" containerID="a0ca7cf8af15a7c53169206e1d36b5848f814f79637b951f4e14133b7457ae61" exitCode=60 Oct 30 05:00:20.594925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.594467 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" event={"ID":"1c5495cb-8dcb-4caf-85d6-8ca2f770077c","Type":"ContainerDied","Data":"a0ca7cf8af15a7c53169206e1d36b5848f814f79637b951f4e14133b7457ae61"} Oct 30 05:00:20.594925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.594539 6587 scope.go:122] "RemoveContainer" containerID="651774454c552e6e8266b686602ed7f4442d9a6aaec19f930f597af2090f0355" Oct 30 05:00:20.595049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.594953 6587 scope.go:122] "RemoveContainer" containerID="a0ca7cf8af15a7c53169206e1d36b5848f814f79637b951f4e14133b7457ae61" Oct 30 05:00:20.595335 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:20.595307 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm_odh-ai-gateway-infra(1c5495cb-8dcb-4caf-85d6-8ca2f770077c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" podUID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" Oct 30 05:00:20.600453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.600343 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" event={"ID":"ea87894e-d49d-4a93-b8d6-d17908cd68fd","Type":"ContainerDied","Data":"25dd7889b6fa14d0528b04e5c83f1d547abaebc5b622054f9ec1dad1f836ff06"} Oct 30 05:00:20.600554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.600424 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w" Oct 30 05:00:20.610256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.610233 6587 scope.go:122] "RemoveContainer" containerID="845b0a4b29ad78d9cefd5a20d76c4a0cbc0cf8da1ec5bd6f4766813c4083779d" Oct 30 05:00:20.686390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.681081 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w"] Oct 30 05:00:20.690853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.686544 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888940-5fp7w"] Oct 30 05:00:20.698391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.697560 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm"] Oct 30 05:00:20.806483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.806456 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" Oct 30 05:00:20.971517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.971438 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-kube-api-access-6gssr\" (UniqueName: \"kubernetes.io/projected/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-kube-api-access-6gssr\") pod \"e853a0fb-b9ed-413d-8fb0-96c76a43ba69\" (UID: \"e853a0fb-b9ed-413d-8fb0-96c76a43ba69\") " Oct 30 05:00:20.971657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.971559 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-maas-api-service-ca\") pod \"e853a0fb-b9ed-413d-8fb0-96c76a43ba69\" (UID: \"e853a0fb-b9ed-413d-8fb0-96c76a43ba69\") " Oct 30 05:00:20.971916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.971893 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-maas-api-service-ca" pod "e853a0fb-b9ed-413d-8fb0-96c76a43ba69" (UID: "e853a0fb-b9ed-413d-8fb0-96c76a43ba69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:20.971997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.971983 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:20.973648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:20.973622 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-kube-api-access-6gssr" pod "e853a0fb-b9ed-413d-8fb0-96c76a43ba69" (UID: "e853a0fb-b9ed-413d-8fb0-96c76a43ba69"). InnerVolumeSpecName "kube-api-access-6gssr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:21.073220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.073189 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gssr\" (UniqueName: \"kubernetes.io/projected/e853a0fb-b9ed-413d-8fb0-96c76a43ba69-kube-api-access-6gssr\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.589738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.589702 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f942106-372c-4185-9a1f-79426c772a3c" path="/var/lib/kubelet/pods/0f942106-372c-4185-9a1f-79426c772a3c/volumes" Oct 30 05:00:21.590137 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.590122 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" path="/var/lib/kubelet/pods/ea87894e-d49d-4a93-b8d6-d17908cd68fd/volumes" Oct 30 05:00:21.610143 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.610109 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" event={"ID":"e853a0fb-b9ed-413d-8fb0-96c76a43ba69","Type":"ContainerDied","Data":"aeb1a6736f10b91da78321e6fe73d72e0de1da08655476c2d77787b7adfe1e11"} Oct 30 05:00:21.610534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.610124 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98" Oct 30 05:00:21.610534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.610181 6587 scope.go:122] "RemoveContainer" containerID="761975eb30db42bbee863fd9a28cfeb2bff0e110688daefd9568cf240cfee490" Oct 30 05:00:21.696217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.696087 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98"] Oct 30 05:00:21.706312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.706278 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888940-4wt98"] Oct 30 05:00:21.728640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.728341 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg"] Oct 30 05:00:21.729950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.729924 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0f942106-372c-4185-9a1f-79426c772a3c" containerName="cleanup" Oct 30 05:00:21.730168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730149 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f942106-372c-4185-9a1f-79426c772a3c" containerName="cleanup" Oct 30 05:00:21.730306 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730292 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" containerName="cleanup" Oct 30 05:00:21.730496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730478 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" containerName="cleanup" Oct 30 05:00:21.730578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730504 6587 cpu_manager.go:414] "containerMap: removing container" podUID="79d39cd0-5c20-4f5c-acc3-6a63bebfcc07" containerName="revoke-keys" Oct 30 05:00:21.730578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730513 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d39cd0-5c20-4f5c-acc3-6a63bebfcc07" containerName="revoke-keys" Oct 30 05:00:21.730578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730528 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" containerName="cleanup" Oct 30 05:00:21.730578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730535 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" containerName="cleanup" Oct 30 05:00:21.730578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730561 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" containerName="cleanup" Oct 30 05:00:21.730578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730569 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730585 6587 cpu_manager.go:414] "containerMap: removing container" podUID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730592 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730617 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730624 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730641 6587 cpu_manager.go:414] "containerMap: removing container" podUID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730648 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730666 6587 cpu_manager.go:414] "containerMap: removing container" podUID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730673 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730695 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730703 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730721 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0f942106-372c-4185-9a1f-79426c772a3c" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730729 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f942106-372c-4185-9a1f-79426c772a3c" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730877 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730891 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730903 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730914 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730926 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f942106-372c-4185-9a1f-79426c772a3c" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730937 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f942106-372c-4185-9a1f-79426c772a3c" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730946 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730955 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d39cd0-5c20-4f5c-acc3-6a63bebfcc07" containerName="revoke-keys" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730962 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" containerName="cleanup" Oct 30 05:00:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.730971 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" containerName="cleanup" Oct 30 05:00:21.735224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.735203 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.773570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.773547 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg"] Oct 30 05:00:21.805937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.805898 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" Oct 30 05:00:21.890831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.890761 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.890960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.890902 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df01352d-3e53-4bcc-a30a-a2b291f95879-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.891012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.890953 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.891012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.890986 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.891152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.891130 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.891217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.891202 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.891268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.891259 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.891407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.891352 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrbws\" (UniqueName: \"kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-kube-api-access-hrbws\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.891442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.891422 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.992752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.992714 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-kube-api-access-52cgz\" (UniqueName: \"kubernetes.io/projected/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-kube-api-access-52cgz\") pod \"1c5495cb-8dcb-4caf-85d6-8ca2f770077c\" (UID: \"1c5495cb-8dcb-4caf-85d6-8ca2f770077c\") " Oct 30 05:00:21.992917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.992900 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-maas-api-service-ca\") pod \"1c5495cb-8dcb-4caf-85d6-8ca2f770077c\" (UID: \"1c5495cb-8dcb-4caf-85d6-8ca2f770077c\") " Oct 30 05:00:21.993172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.993153 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.993214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.993196 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrbws\" (UniqueName: \"kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-kube-api-access-hrbws\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.993258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.993222 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.993305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.993254 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-maas-api-service-ca" pod "1c5495cb-8dcb-4caf-85d6-8ca2f770077c" (UID: "1c5495cb-8dcb-4caf-85d6-8ca2f770077c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:21.993343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.993327 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.993565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.993542 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df01352d-3e53-4bcc-a30a-a2b291f95879-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.993713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.993581 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.993713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.993611 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.993713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.993628 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.993713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.993677 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.993927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.993751 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.993927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.993872 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.993927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.993910 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.994163 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.994141 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.994258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.994237 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df01352d-3e53-4bcc-a30a-a2b291f95879-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.994305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.994262 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.995150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.995131 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-kube-api-access-52cgz" pod "1c5495cb-8dcb-4caf-85d6-8ca2f770077c" (UID: "1c5495cb-8dcb-4caf-85d6-8ca2f770077c"). InnerVolumeSpecName "kube-api-access-52cgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:21.995956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.995932 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:21.996154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:21.996138 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:22.013498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.013477 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:22.013626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.013556 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrbws\" (UniqueName: \"kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-kube-api-access-hrbws\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:22.076911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.076879 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:22.095309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.095284 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52cgz\" (UniqueName: \"kubernetes.io/projected/1c5495cb-8dcb-4caf-85d6-8ca2f770077c-kube-api-access-52cgz\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:22.392170 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:00:22.392138 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf01352d_3e53_4bcc_a30a_a2b291f95879.slice/crio-cbad4387470471468f9c14f8e38df78dc881b201f5fb0a4ade3da00512e6b5cf WatchSource:0}: Error finding container cbad4387470471468f9c14f8e38df78dc881b201f5fb0a4ade3da00512e6b5cf: Status 404 returned error can't find the container with id cbad4387470471468f9c14f8e38df78dc881b201f5fb0a4ade3da00512e6b5cf Oct 30 05:00:22.393920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.393898 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg"] Oct 30 05:00:22.394802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.394747 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:22.394861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.394808 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:22.394861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.394835 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:22.621237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.621203 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" Oct 30 05:00:22.621761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.621243 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm" event={"ID":"1c5495cb-8dcb-4caf-85d6-8ca2f770077c","Type":"ContainerDied","Data":"47d11fa3c4b24907af12f644f01dd14a378a1c2e059735607ff244e1ba2fe192"} Oct 30 05:00:22.621761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.621322 6587 scope.go:122] "RemoveContainer" containerID="a0ca7cf8af15a7c53169206e1d36b5848f814f79637b951f4e14133b7457ae61" Oct 30 05:00:22.628970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.628103 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" event={"ID":"df01352d-3e53-4bcc-a30a-a2b291f95879","Type":"ContainerStarted","Data":"4871bec13135c78c32505a0154a573254bf2af3dd8aa91ebcb1442fb03ad403c"} Oct 30 05:00:22.628970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.628163 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" event={"ID":"df01352d-3e53-4bcc-a30a-a2b291f95879","Type":"ContainerStarted","Data":"cbad4387470471468f9c14f8e38df78dc881b201f5fb0a4ade3da00512e6b5cf"} Oct 30 05:00:22.675922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.675843 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm"] Oct 30 05:00:22.682771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.682746 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888940-jvhbm"] Oct 30 05:00:22.715799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.715766 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86"] Oct 30 05:00:22.716491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716469 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" containerName="cleanup" Oct 30 05:00:22.716491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716488 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" containerName="cleanup" Oct 30 05:00:22.716661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716508 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" containerName="cleanup" Oct 30 05:00:22.716661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716513 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7b9eaa-d1a2-457f-9622-c8b3a7f2e19d" containerName="cleanup" Oct 30 05:00:22.716661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716522 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" containerName="cleanup" Oct 30 05:00:22.716661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716527 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" containerName="cleanup" Oct 30 05:00:22.716661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716540 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0f942106-372c-4185-9a1f-79426c772a3c" containerName="cleanup" Oct 30 05:00:22.716661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716545 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f942106-372c-4185-9a1f-79426c772a3c" containerName="cleanup" Oct 30 05:00:22.716661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716639 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" containerName="cleanup" Oct 30 05:00:22.716661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716652 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" containerName="cleanup" Oct 30 05:00:22.716661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716660 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea87894e-d49d-4a93-b8d6-d17908cd68fd" containerName="cleanup" Oct 30 05:00:22.716661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716667 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" containerName="cleanup" Oct 30 05:00:22.717097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716673 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" containerName="cleanup" Oct 30 05:00:22.717097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.716682 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f942106-372c-4185-9a1f-79426c772a3c" containerName="cleanup" Oct 30 05:00:22.721550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.721516 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" Oct 30 05:00:22.726454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.726297 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" podStartSLOduration=1.726279954 podStartE2EDuration="1.726279954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:22.720622522 +0000 UTC m=+947.248389310" watchObservedRunningTime="2026-10-30 05:00:22.726279954 +0000 UTC m=+947.254046741" Oct 30 05:00:22.738900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.738872 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86"] Oct 30 05:00:22.759738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.759709 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:00:22.760088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.760069 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:00:22.760258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.760245 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:00:22.760459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.760443 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:00:22.804098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.804069 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77ab3227-v4f86\" (UID: \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" Oct 30 05:00:22.804230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.804146 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77ab3227-v4f86\" (UID: \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" Oct 30 05:00:22.804400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.804379 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfvnm\" (UniqueName: \"kubernetes.io/projected/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-kube-api-access-mfvnm\") pod \"maas-api-revoke-sub-77ab3227-v4f86\" (UID: \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" Oct 30 05:00:22.906087 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.906048 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77ab3227-v4f86\" (UID: \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" Oct 30 05:00:22.906243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.906146 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77ab3227-v4f86\" (UID: \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" Oct 30 05:00:22.906307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.906288 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfvnm\" (UniqueName: \"kubernetes.io/projected/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-kube-api-access-mfvnm\") pod \"maas-api-revoke-sub-77ab3227-v4f86\" (UID: \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" Oct 30 05:00:22.906863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.906838 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77ab3227-v4f86\" (UID: \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" Oct 30 05:00:22.908891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.908866 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77ab3227-v4f86\" (UID: \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" Oct 30 05:00:22.937539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:22.937444 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfvnm\" (UniqueName: \"kubernetes.io/projected/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-kube-api-access-mfvnm\") pod \"maas-api-revoke-sub-77ab3227-v4f86\" (UID: \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" Oct 30 05:00:23.064127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:23.063845 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" Oct 30 05:00:23.077258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:23.077078 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:23.079076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:23.079048 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:00:23.079160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:23.079110 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" podUID="df01352d-3e53-4bcc-a30a-a2b291f95879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:00:23.408604 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:00:23.408560 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e9ddc8_8e34_4c3f_be96_ec71ebbd0a10.slice/crio-f31aa872dcc56e2715fde448e2822885d59a36b437770b2a0db5a61e3479960e WatchSource:0}: Error finding container f31aa872dcc56e2715fde448e2822885d59a36b437770b2a0db5a61e3479960e: Status 404 returned error can't find the container with id f31aa872dcc56e2715fde448e2822885d59a36b437770b2a0db5a61e3479960e Oct 30 05:00:23.410111 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:23.410082 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86"] Oct 30 05:00:23.592258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:23.592221 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" path="/var/lib/kubelet/pods/1c5495cb-8dcb-4caf-85d6-8ca2f770077c/volumes" Oct 30 05:00:23.592916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:23.592891 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e853a0fb-b9ed-413d-8fb0-96c76a43ba69" path="/var/lib/kubelet/pods/e853a0fb-b9ed-413d-8fb0-96c76a43ba69/volumes" Oct 30 05:00:23.661408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:23.661300 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" event={"ID":"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10","Type":"ContainerStarted","Data":"f31aa872dcc56e2715fde448e2822885d59a36b437770b2a0db5a61e3479960e"} Oct 30 05:00:23.981832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:23.981747 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq"] Oct 30 05:00:23.982456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:23.982427 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" containerName="cleanup" Oct 30 05:00:23.982456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:23.982449 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5495cb-8dcb-4caf-85d6-8ca2f770077c" containerName="cleanup" Oct 30 05:00:23.986403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:23.986385 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" Oct 30 05:00:24.006291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:24.006259 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq"] Oct 30 05:00:24.053410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:24.052809 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84f21d02-5m8tq\" (UID: \"4e462373-e3db-49a2-87b7-d860a46f0e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" Oct 30 05:00:24.053410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:24.052917 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84f21d02-5m8tq\" (UID: \"4e462373-e3db-49a2-87b7-d860a46f0e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" Oct 30 05:00:24.053410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:24.053042 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slkqk\" (UniqueName: \"kubernetes.io/projected/4e462373-e3db-49a2-87b7-d860a46f0e5c-kube-api-access-slkqk\") pod \"maas-api-revoke-sub-84f21d02-5m8tq\" (UID: \"4e462373-e3db-49a2-87b7-d860a46f0e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" Oct 30 05:00:24.077476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:24.077346 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:00:24.077476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:24.077411 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" podUID="df01352d-3e53-4bcc-a30a-a2b291f95879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:00:24.154287 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:24.154245 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84f21d02-5m8tq\" (UID: \"4e462373-e3db-49a2-87b7-d860a46f0e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" Oct 30 05:00:24.154493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:24.154348 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slkqk\" (UniqueName: \"kubernetes.io/projected/4e462373-e3db-49a2-87b7-d860a46f0e5c-kube-api-access-slkqk\") pod \"maas-api-revoke-sub-84f21d02-5m8tq\" (UID: \"4e462373-e3db-49a2-87b7-d860a46f0e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" Oct 30 05:00:24.154561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:24.154503 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84f21d02-5m8tq\" (UID: \"4e462373-e3db-49a2-87b7-d860a46f0e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" Oct 30 05:00:24.155527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:24.155484 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84f21d02-5m8tq\" (UID: \"4e462373-e3db-49a2-87b7-d860a46f0e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" Oct 30 05:00:24.163195 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:24.163054 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84f21d02-5m8tq\" (UID: \"4e462373-e3db-49a2-87b7-d860a46f0e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" Oct 30 05:00:24.176253 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:24.176233 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slkqk\" (UniqueName: \"kubernetes.io/projected/4e462373-e3db-49a2-87b7-d860a46f0e5c-kube-api-access-slkqk\") pod \"maas-api-revoke-sub-84f21d02-5m8tq\" (UID: \"4e462373-e3db-49a2-87b7-d860a46f0e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" Oct 30 05:00:24.300521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:24.300491 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" Oct 30 05:00:25.079383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:25.073497 6587 generic.go:356] "Generic (PLEG): container finished" podID="07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10" containerID="8ed6a2ab49b7ad6c39859815ddb219ec141064d67baedbf9273a75315f2047a8" exitCode=0 Oct 30 05:00:25.096185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:25.096014 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" event={"ID":"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10","Type":"ContainerDied","Data":"8ed6a2ab49b7ad6c39859815ddb219ec141064d67baedbf9273a75315f2047a8"} Oct 30 05:00:25.154618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:25.126482 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:00:25.154618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:25.126540 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" podUID="df01352d-3e53-4bcc-a30a-a2b291f95879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:00:25.404438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:25.404029 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq"] Oct 30 05:00:25.413381 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:00:25.407698 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e462373_e3db_49a2_87b7_d860a46f0e5c.slice/crio-eda8a22f718ae7baba544152f67755183930a9fd5fe35297ce86b522c98ce5d7 WatchSource:0}: Error finding container eda8a22f718ae7baba544152f67755183930a9fd5fe35297ce86b522c98ce5d7: Status 404 returned error can't find the container with id eda8a22f718ae7baba544152f67755183930a9fd5fe35297ce86b522c98ce5d7 Oct 30 05:00:25.622407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:25.621094 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:25.647392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:25.633551 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:25.940590 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:25.908485 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr"] Oct 30 05:00:25.974300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:25.974186 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr"] Oct 30 05:00:25.974846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:25.974823 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.006118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.006096 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b21507\"/\"kube-root-ca.crt\"" Oct 30 05:00:26.007580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.006617 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b21507\"/\"default-dockercfg-9drgk\"" Oct 30 05:00:26.007878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.007220 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b21507\"/\"openshift-service-ca.crt\"" Oct 30 05:00:26.008010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.007814 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b21507\"/\"distinct-2-w0-b21507-kserve-self-signed-certs\"" Oct 30 05:00:26.102186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.093235 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-tmp-dir\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.102186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.093404 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-model-cache\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.102186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.093438 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3758b02b-300e-4659-a74e-f96bc8967080-tls-certs\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.102186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.099294 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:00:26.102186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.099342 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" podUID="df01352d-3e53-4bcc-a30a-a2b291f95879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:00:26.102186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.093482 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-dshm\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.102186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.101909 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpx9c\" (UniqueName: \"kubernetes.io/projected/3758b02b-300e-4659-a74e-f96bc8967080-kube-api-access-tpx9c\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.102186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.101944 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-home\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.164389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.160627 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:26.216410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.207273 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-dshm\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.216410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.207399 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpx9c\" (UniqueName: \"kubernetes.io/projected/3758b02b-300e-4659-a74e-f96bc8967080-kube-api-access-tpx9c\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.216410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.207435 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-home\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.216410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.207560 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-tmp-dir\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.216410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.207726 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-model-cache\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.216410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.207763 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3758b02b-300e-4659-a74e-f96bc8967080-tls-certs\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.216410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.208710 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-home\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.216410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.214010 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-tmp-dir\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.216410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.214267 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-model-cache\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.239423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.229462 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:00:26.259454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.254447 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:26.262765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.262663 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3758b02b-300e-4659-a74e-f96bc8967080-tls-certs\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.267748 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.267687 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-dshm\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.296390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.295416 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpx9c\" (UniqueName: \"kubernetes.io/projected/3758b02b-300e-4659-a74e-f96bc8967080-kube-api-access-tpx9c\") pod \"distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.369398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.343800 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:00:26.399415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.399373 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" event={"ID":"4e462373-e3db-49a2-87b7-d860a46f0e5c","Type":"ContainerStarted","Data":"eda8a22f718ae7baba544152f67755183930a9fd5fe35297ce86b522c98ce5d7"} Oct 30 05:00:26.439867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.439833 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:26.503393 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:26.502566 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:00:26.628379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.622447 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:26.628379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.624394 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:00:26.643783 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.638526 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:26.643783 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.638934 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:00:26.849703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:26.830662 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:00:27.035511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:27.035477 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:00:27.084406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:27.083822 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:00:27.084406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:27.083887 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" podUID="df01352d-3e53-4bcc-a30a-a2b291f95879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:00:27.179484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:27.179236 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:27.199381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:27.187591 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:00:27.334353 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:27.332216 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:00:27.509338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:27.369577 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:00:27.509338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:27.378465 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:27.509338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:27.429095 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:00:27.509338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:27.484347 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:00:27.923767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:27.923727 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:00:28.002394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:27.981227 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:28.032228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.032084 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:00:28.193457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.143233 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" event={"ID":"4e462373-e3db-49a2-87b7-d860a46f0e5c","Type":"ContainerStarted","Data":"87ff3071793e2aa212a68a0ff5bd597b9dacf14cfe843f55c56dcafe5d3b5bda"} Oct 30 05:00:28.193457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.175649 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:00:28.193457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.175720 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" podUID="df01352d-3e53-4bcc-a30a-a2b291f95879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:00:28.193457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.183690 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:28.294793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.285503 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:00:28.414831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.414800 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:00:28.473619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.451588 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" Oct 30 05:00:28.634530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.630694 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-xks-ca\") pod \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\" (UID: \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\") " Oct 30 05:00:28.649399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.635074 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-kube-api-access-mfvnm\" (UniqueName: \"kubernetes.io/projected/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-kube-api-access-mfvnm\") pod \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\" (UID: \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\") " Oct 30 05:00:28.649399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.645531 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-openshift-ca\") pod \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\" (UID: \"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10\") " Oct 30 05:00:28.667444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.665720 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:00:28.668089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.668060 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-xks-ca" pod "07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10" (UID: "07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:00:28.668732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.668709 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-kube-api-access-mfvnm" pod "07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10" (UID: "07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10"). InnerVolumeSpecName "kube-api-access-mfvnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:28.776430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.709061 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-openshift-ca" pod "07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10" (UID: "07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:28.798835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.792085 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:28.798835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.792120 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfvnm\" (UniqueName: \"kubernetes.io/projected/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-kube-api-access-mfvnm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:28.798835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.792137 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:28.892197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.892132 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr"] Oct 30 05:00:28.947130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.926590 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:28.947130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.927541 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:00:28.970321 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:28.967942 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:00:29.093388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.085463 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:00:29.093388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.085522 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" podUID="df01352d-3e53-4bcc-a30a-a2b291f95879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:00:29.355938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.355850 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" event={"ID":"3758b02b-300e-4659-a74e-f96bc8967080","Type":"ContainerStarted","Data":"f6ecff9b44d52190920c76af403207954a191d9ba26b94abe0f1c7de3d8a03b2"} Oct 30 05:00:29.358915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.358882 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" event={"ID":"07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10","Type":"ContainerDied","Data":"f31aa872dcc56e2715fde448e2822885d59a36b437770b2a0db5a61e3479960e"} Oct 30 05:00:29.359062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.358939 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f31aa872dcc56e2715fde448e2822885d59a36b437770b2a0db5a61e3479960e" Oct 30 05:00:29.359170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.359154 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86" Oct 30 05:00:29.361179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.361151 6587 generic.go:356] "Generic (PLEG): container finished" podID="4e462373-e3db-49a2-87b7-d860a46f0e5c" containerID="87ff3071793e2aa212a68a0ff5bd597b9dacf14cfe843f55c56dcafe5d3b5bda" exitCode=0 Oct 30 05:00:29.362141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.362113 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" event={"ID":"4e462373-e3db-49a2-87b7-d860a46f0e5c","Type":"ContainerDied","Data":"87ff3071793e2aa212a68a0ff5bd597b9dacf14cfe843f55c56dcafe5d3b5bda"} Oct 30 05:00:29.700501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.700470 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" Oct 30 05:00:29.798191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.798161 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e462373-e3db-49a2-87b7-d860a46f0e5c-kube-api-access-slkqk\" (UniqueName: \"kubernetes.io/projected/4e462373-e3db-49a2-87b7-d860a46f0e5c-kube-api-access-slkqk\") pod \"4e462373-e3db-49a2-87b7-d860a46f0e5c\" (UID: \"4e462373-e3db-49a2-87b7-d860a46f0e5c\") " Oct 30 05:00:29.798385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.798233 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-xks-ca\") pod \"4e462373-e3db-49a2-87b7-d860a46f0e5c\" (UID: \"4e462373-e3db-49a2-87b7-d860a46f0e5c\") " Oct 30 05:00:29.798385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.798279 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-openshift-ca\") pod \"4e462373-e3db-49a2-87b7-d860a46f0e5c\" (UID: \"4e462373-e3db-49a2-87b7-d860a46f0e5c\") " Oct 30 05:00:29.798689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.798659 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-openshift-ca" pod "4e462373-e3db-49a2-87b7-d860a46f0e5c" (UID: "4e462373-e3db-49a2-87b7-d860a46f0e5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:29.798834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.798787 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:29.800391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.800340 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-xks-ca" pod "4e462373-e3db-49a2-87b7-d860a46f0e5c" (UID: "4e462373-e3db-49a2-87b7-d860a46f0e5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:00:29.800467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.800350 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e462373-e3db-49a2-87b7-d860a46f0e5c-kube-api-access-slkqk" pod "4e462373-e3db-49a2-87b7-d860a46f0e5c" (UID: "4e462373-e3db-49a2-87b7-d860a46f0e5c"). InnerVolumeSpecName "kube-api-access-slkqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:29.900347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.900301 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slkqk\" (UniqueName: \"kubernetes.io/projected/4e462373-e3db-49a2-87b7-d860a46f0e5c-kube-api-access-slkqk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:29.900347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:29.900337 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e462373-e3db-49a2-87b7-d860a46f0e5c-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:30.077995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:30.077964 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:00:30.078184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:30.078022 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" podUID="df01352d-3e53-4bcc-a30a-a2b291f95879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:00:30.372292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:30.372195 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" Oct 30 05:00:30.372292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:30.372210 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq" event={"ID":"4e462373-e3db-49a2-87b7-d860a46f0e5c","Type":"ContainerDied","Data":"eda8a22f718ae7baba544152f67755183930a9fd5fe35297ce86b522c98ce5d7"} Oct 30 05:00:30.372823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:30.372292 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eda8a22f718ae7baba544152f67755183930a9fd5fe35297ce86b522c98ce5d7" Oct 30 05:00:30.374807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:30.374782 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" event={"ID":"3758b02b-300e-4659-a74e-f96bc8967080","Type":"ContainerStarted","Data":"b386800447a4fb9b835031d7305f652ffd41e2d7d12c4d4cbaa642dc8304df50"} Oct 30 05:00:30.416086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:30.416026 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podStartSLOduration=5.416007915 podStartE2EDuration="5.416007915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:30.414327024 +0000 UTC m=+954.942093811" watchObservedRunningTime="2026-10-30 05:00:30.416007915 +0000 UTC m=+954.943774697" Oct 30 05:00:30.440346 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:30.440315 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:31.496626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:31.496590 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:31.497088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:31.497064 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:31.497950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:31.497931 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:00:32.508424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.508386 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk"] Oct 30 05:00:32.509203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.509180 6587 cpu_manager.go:414] "containerMap: removing container" podUID="07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10" containerName="revoke-keys" Oct 30 05:00:32.509203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.509204 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10" containerName="revoke-keys" Oct 30 05:00:32.509406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.509278 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4e462373-e3db-49a2-87b7-d860a46f0e5c" containerName="revoke-keys" Oct 30 05:00:32.509406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.509295 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e462373-e3db-49a2-87b7-d860a46f0e5c" containerName="revoke-keys" Oct 30 05:00:32.509502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.509489 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e462373-e3db-49a2-87b7-d860a46f0e5c" containerName="revoke-keys" Oct 30 05:00:32.509547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.509507 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10" containerName="revoke-keys" Oct 30 05:00:32.513426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.513405 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" Oct 30 05:00:32.544821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.544784 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:00:32.545015 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.544997 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:00:32.549385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.545700 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:00:32.553537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.553510 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk"] Oct 30 05:00:32.558254 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.558224 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:00:32.649214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.649163 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d17404e8-twwnk\" (UID: \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" Oct 30 05:00:32.649647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.649621 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkd44\" (UniqueName: \"kubernetes.io/projected/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-kube-api-access-jkd44\") pod \"maas-api-revoke-sub-d17404e8-twwnk\" (UID: \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" Oct 30 05:00:32.649856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.649839 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d17404e8-twwnk\" (UID: \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" Oct 30 05:00:32.751201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.751146 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d17404e8-twwnk\" (UID: \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" Oct 30 05:00:32.751421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.751313 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d17404e8-twwnk\" (UID: \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" Oct 30 05:00:32.752251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.752225 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d17404e8-twwnk\" (UID: \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" Oct 30 05:00:32.752599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.752572 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkd44\" (UniqueName: \"kubernetes.io/projected/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-kube-api-access-jkd44\") pod \"maas-api-revoke-sub-d17404e8-twwnk\" (UID: \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" Oct 30 05:00:32.765272 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.765160 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d17404e8-twwnk\" (UID: \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" Oct 30 05:00:32.773609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.773578 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkd44\" (UniqueName: \"kubernetes.io/projected/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-kube-api-access-jkd44\") pod \"maas-api-revoke-sub-d17404e8-twwnk\" (UID: \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" Oct 30 05:00:32.858595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:32.858544 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" Oct 30 05:00:33.167091 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:00:33.167050 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b7e47a_91c4_4e73_a1b0_02d5353b3a62.slice/crio-8f7faee5c75633ebe3d4f381f54f3aa9f016d233c2e8d2809c962dc3dafc8482 WatchSource:0}: Error finding container 8f7faee5c75633ebe3d4f381f54f3aa9f016d233c2e8d2809c962dc3dafc8482: Status 404 returned error can't find the container with id 8f7faee5c75633ebe3d4f381f54f3aa9f016d233c2e8d2809c962dc3dafc8482 Oct 30 05:00:33.169252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:33.169227 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk"] Oct 30 05:00:33.402345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:33.402308 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" event={"ID":"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62","Type":"ContainerStarted","Data":"8f7faee5c75633ebe3d4f381f54f3aa9f016d233c2e8d2809c962dc3dafc8482"} Oct 30 05:00:34.410814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:34.410778 6587 generic.go:356] "Generic (PLEG): container finished" podID="b8b7e47a-91c4-4e73-a1b0-02d5353b3a62" containerID="f3693471d94de13538f393febf244e26f74c6262f00e1104a449e6cb1adc70bb" exitCode=0 Oct 30 05:00:34.411270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:34.410859 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" event={"ID":"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62","Type":"ContainerDied","Data":"f3693471d94de13538f393febf244e26f74c6262f00e1104a449e6cb1adc70bb"} Oct 30 05:00:34.862129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:34.862094 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4"] Oct 30 05:00:34.867370 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:34.867337 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" Oct 30 05:00:34.882418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:34.882392 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4"] Oct 30 05:00:34.980663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:34.980627 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-404168a7-297x4\" (UID: \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" Oct 30 05:00:34.980839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:34.980778 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-404168a7-297x4\" (UID: \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" Oct 30 05:00:34.980955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:34.980932 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw6h4\" (UniqueName: \"kubernetes.io/projected/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-kube-api-access-qw6h4\") pod \"maas-api-revoke-sub-404168a7-297x4\" (UID: \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" Oct 30 05:00:35.081964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.081931 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw6h4\" (UniqueName: \"kubernetes.io/projected/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-kube-api-access-qw6h4\") pod \"maas-api-revoke-sub-404168a7-297x4\" (UID: \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" Oct 30 05:00:35.082158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.082050 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-404168a7-297x4\" (UID: \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" Oct 30 05:00:35.082158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.082137 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-404168a7-297x4\" (UID: \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" Oct 30 05:00:35.082801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.082773 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-404168a7-297x4\" (UID: \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" Oct 30 05:00:35.084434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.084410 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-404168a7-297x4\" (UID: \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" Oct 30 05:00:35.102375 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.102342 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw6h4\" (UniqueName: \"kubernetes.io/projected/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-kube-api-access-qw6h4\") pod \"maas-api-revoke-sub-404168a7-297x4\" (UID: \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" Oct 30 05:00:35.181983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.181906 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" Oct 30 05:00:35.485931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.485898 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4"] Oct 30 05:00:35.584968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.584935 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" Oct 30 05:00:35.690476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.690442 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-xks-ca\") pod \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\" (UID: \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\") " Oct 30 05:00:35.690672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.690555 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-openshift-ca\") pod \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\" (UID: \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\") " Oct 30 05:00:35.690743 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.690692 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-kube-api-access-jkd44\" (UniqueName: \"kubernetes.io/projected/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-kube-api-access-jkd44\") pod \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\" (UID: \"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62\") " Oct 30 05:00:35.690898 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.690873 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-openshift-ca" pod "b8b7e47a-91c4-4e73-a1b0-02d5353b3a62" (UID: "b8b7e47a-91c4-4e73-a1b0-02d5353b3a62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:35.691471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.691443 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:35.692634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.692605 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-xks-ca" pod "b8b7e47a-91c4-4e73-a1b0-02d5353b3a62" (UID: "b8b7e47a-91c4-4e73-a1b0-02d5353b3a62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:00:35.692888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.692865 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-kube-api-access-jkd44" pod "b8b7e47a-91c4-4e73-a1b0-02d5353b3a62" (UID: "b8b7e47a-91c4-4e73-a1b0-02d5353b3a62"). InnerVolumeSpecName "kube-api-access-jkd44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:35.793408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.793376 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkd44\" (UniqueName: \"kubernetes.io/projected/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-kube-api-access-jkd44\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:35.793408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:35.793411 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:36.430733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:36.430651 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" Oct 30 05:00:36.430733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:36.430676 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk" event={"ID":"b8b7e47a-91c4-4e73-a1b0-02d5353b3a62","Type":"ContainerDied","Data":"8f7faee5c75633ebe3d4f381f54f3aa9f016d233c2e8d2809c962dc3dafc8482"} Oct 30 05:00:36.430733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:36.430727 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f7faee5c75633ebe3d4f381f54f3aa9f016d233c2e8d2809c962dc3dafc8482" Oct 30 05:00:36.433075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:36.433040 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" event={"ID":"e6e16f83-b311-4ae9-8aba-d87af6fd0eca","Type":"ContainerStarted","Data":"b55844fa44b0385bedcf08ed92b5f920d169e74a4eeea405eee43d05f949fa26"} Oct 30 05:00:36.433248 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:36.433099 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" event={"ID":"e6e16f83-b311-4ae9-8aba-d87af6fd0eca","Type":"ContainerStarted","Data":"ce5c5d061a2c676f95d3fb924a83a918b724fa7b4a98dd8ba96e38902bbe6600"} Oct 30 05:00:36.440494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:36.440469 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:36.454042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:36.454018 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:36.530149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:36.530088 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" podStartSLOduration=2.530069632 podStartE2EDuration="2.530069632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:36.485657923 +0000 UTC m=+961.013424709" watchObservedRunningTime="2026-10-30 05:00:36.530069632 +0000 UTC m=+961.057836419" Oct 30 05:00:37.444046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:37.444013 6587 generic.go:356] "Generic (PLEG): container finished" podID="e6e16f83-b311-4ae9-8aba-d87af6fd0eca" containerID="b55844fa44b0385bedcf08ed92b5f920d169e74a4eeea405eee43d05f949fa26" exitCode=0 Oct 30 05:00:37.444260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:37.444106 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" event={"ID":"e6e16f83-b311-4ae9-8aba-d87af6fd0eca","Type":"ContainerDied","Data":"b55844fa44b0385bedcf08ed92b5f920d169e74a4eeea405eee43d05f949fa26"} Oct 30 05:00:37.452688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:37.452664 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:38.618767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:38.618736 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" Oct 30 05:00:38.750680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:38.750586 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-xks-ca\") pod \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\" (UID: \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\") " Oct 30 05:00:38.750896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:38.750872 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-openshift-ca\") pod \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\" (UID: \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\") " Oct 30 05:00:38.750976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:38.750946 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-kube-api-access-qw6h4\" (UniqueName: \"kubernetes.io/projected/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-kube-api-access-qw6h4\") pod \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\" (UID: \"e6e16f83-b311-4ae9-8aba-d87af6fd0eca\") " Oct 30 05:00:38.751294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:38.751259 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-openshift-ca" pod "e6e16f83-b311-4ae9-8aba-d87af6fd0eca" (UID: "e6e16f83-b311-4ae9-8aba-d87af6fd0eca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:38.751629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:38.751590 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:38.752931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:38.752903 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-xks-ca" pod "e6e16f83-b311-4ae9-8aba-d87af6fd0eca" (UID: "e6e16f83-b311-4ae9-8aba-d87af6fd0eca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:00:38.753634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:38.753600 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-kube-api-access-qw6h4" pod "e6e16f83-b311-4ae9-8aba-d87af6fd0eca" (UID: "e6e16f83-b311-4ae9-8aba-d87af6fd0eca"). InnerVolumeSpecName "kube-api-access-qw6h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:38.852678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:38.852632 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw6h4\" (UniqueName: \"kubernetes.io/projected/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-kube-api-access-qw6h4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:38.852678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:38.852676 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e16f83-b311-4ae9-8aba-d87af6fd0eca-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:39.461790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:39.461751 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" Oct 30 05:00:39.461790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:39.461778 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4" event={"ID":"e6e16f83-b311-4ae9-8aba-d87af6fd0eca","Type":"ContainerDied","Data":"ce5c5d061a2c676f95d3fb924a83a918b724fa7b4a98dd8ba96e38902bbe6600"} Oct 30 05:00:39.461998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:39.461869 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce5c5d061a2c676f95d3fb924a83a918b724fa7b4a98dd8ba96e38902bbe6600" Oct 30 05:00:41.805817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:41.805764 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:42.030523 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:42.030453 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:00:42.199438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:42.198611 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:42.201158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:42.199686 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:42.297390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:42.295450 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:42.427390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:42.426188 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:00:42.455393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:42.454528 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:00:42.827384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:42.824424 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:00:42.827384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:42.825266 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:00:43.052402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.034490 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:43.077463 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.053606 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:00:43.253995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.200678 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:43.253995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.214906 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:00:43.348928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.256773 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:43.348928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.285252 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:00:43.348928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.308457 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:00:43.348928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.310470 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:00:43.525689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.495224 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:00:43.525689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.508193 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:43.525689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.508250 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:00:43.525689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.508283 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:00:43.594539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.594506 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:00:43.699910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.698433 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:00:43.700547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.700507 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:00:43.701390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:43.701185 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:00:48.765043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.765008 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs"] Oct 30 05:00:48.765680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.765664 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b8b7e47a-91c4-4e73-a1b0-02d5353b3a62" containerName="revoke-keys" Oct 30 05:00:48.765724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.765682 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b7e47a-91c4-4e73-a1b0-02d5353b3a62" containerName="revoke-keys" Oct 30 05:00:48.765724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.765704 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e6e16f83-b311-4ae9-8aba-d87af6fd0eca" containerName="revoke-keys" Oct 30 05:00:48.765724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.765709 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e16f83-b311-4ae9-8aba-d87af6fd0eca" containerName="revoke-keys" Oct 30 05:00:48.765845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.765835 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b7e47a-91c4-4e73-a1b0-02d5353b3a62" containerName="revoke-keys" Oct 30 05:00:48.765879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.765847 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e16f83-b311-4ae9-8aba-d87af6fd0eca" containerName="revoke-keys" Oct 30 05:00:48.770625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.770606 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" Oct 30 05:00:48.788796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.788772 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs"] Oct 30 05:00:48.797644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.797620 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:00:48.797813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.797790 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:00:48.797981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.797736 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:00:48.799636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.799613 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:00:48.897755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.897722 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc0405e0-lk6gs\" (UID: \"145a9be0-0ea7-48ec-aab8-c186cb5af459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" Oct 30 05:00:48.897931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.897786 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxlt\" (UniqueName: \"kubernetes.io/projected/145a9be0-0ea7-48ec-aab8-c186cb5af459-kube-api-access-hdxlt\") pod \"maas-api-revoke-sub-bc0405e0-lk6gs\" (UID: \"145a9be0-0ea7-48ec-aab8-c186cb5af459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" Oct 30 05:00:48.897970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.897948 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc0405e0-lk6gs\" (UID: \"145a9be0-0ea7-48ec-aab8-c186cb5af459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" Oct 30 05:00:48.999061 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.999025 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc0405e0-lk6gs\" (UID: \"145a9be0-0ea7-48ec-aab8-c186cb5af459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" Oct 30 05:00:48.999229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.999203 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc0405e0-lk6gs\" (UID: \"145a9be0-0ea7-48ec-aab8-c186cb5af459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" Oct 30 05:00:48.999272 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.999259 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxlt\" (UniqueName: \"kubernetes.io/projected/145a9be0-0ea7-48ec-aab8-c186cb5af459-kube-api-access-hdxlt\") pod \"maas-api-revoke-sub-bc0405e0-lk6gs\" (UID: \"145a9be0-0ea7-48ec-aab8-c186cb5af459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" Oct 30 05:00:48.999676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:48.999653 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc0405e0-lk6gs\" (UID: \"145a9be0-0ea7-48ec-aab8-c186cb5af459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" Oct 30 05:00:49.001587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:49.001565 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc0405e0-lk6gs\" (UID: \"145a9be0-0ea7-48ec-aab8-c186cb5af459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" Oct 30 05:00:49.019206 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:49.019149 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxlt\" (UniqueName: \"kubernetes.io/projected/145a9be0-0ea7-48ec-aab8-c186cb5af459-kube-api-access-hdxlt\") pod \"maas-api-revoke-sub-bc0405e0-lk6gs\" (UID: \"145a9be0-0ea7-48ec-aab8-c186cb5af459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" Oct 30 05:00:49.084419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:49.084395 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" Oct 30 05:00:49.381504 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:00:49.381471 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145a9be0_0ea7_48ec_aab8_c186cb5af459.slice/crio-6609e16281c3c078a4d42988478854a6a7d63d8474600934fedd38c67690a8b0 WatchSource:0}: Error finding container 6609e16281c3c078a4d42988478854a6a7d63d8474600934fedd38c67690a8b0: Status 404 returned error can't find the container with id 6609e16281c3c078a4d42988478854a6a7d63d8474600934fedd38c67690a8b0 Oct 30 05:00:49.385075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:49.385047 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs"] Oct 30 05:00:49.663007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:49.662966 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" event={"ID":"145a9be0-0ea7-48ec-aab8-c186cb5af459","Type":"ContainerStarted","Data":"2d4b9f0ade9b8bc113b153ae371a36f9976a8b62c0c8dcbca1d9c2cd125fadfa"} Oct 30 05:00:49.663157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:49.663039 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" event={"ID":"145a9be0-0ea7-48ec-aab8-c186cb5af459","Type":"ContainerStarted","Data":"6609e16281c3c078a4d42988478854a6a7d63d8474600934fedd38c67690a8b0"} Oct 30 05:00:49.706897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:49.706837 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" podStartSLOduration=1.706818808 podStartE2EDuration="1.706818808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:49.701954021 +0000 UTC m=+974.229720809" watchObservedRunningTime="2026-10-30 05:00:49.706818808 +0000 UTC m=+974.234585593" Oct 30 05:00:50.671352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:50.671313 6587 generic.go:356] "Generic (PLEG): container finished" podID="145a9be0-0ea7-48ec-aab8-c186cb5af459" containerID="2d4b9f0ade9b8bc113b153ae371a36f9976a8b62c0c8dcbca1d9c2cd125fadfa" exitCode=0 Oct 30 05:00:50.671772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:50.671411 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" event={"ID":"145a9be0-0ea7-48ec-aab8-c186cb5af459","Type":"ContainerDied","Data":"2d4b9f0ade9b8bc113b153ae371a36f9976a8b62c0c8dcbca1d9c2cd125fadfa"} Oct 30 05:00:50.840185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:50.840153 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh"] Oct 30 05:00:50.844661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:50.844642 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" Oct 30 05:00:50.862599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:50.862576 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh"] Oct 30 05:00:51.027984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.027895 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b84178-86zsh\" (UID: \"e58f88cf-af86-49b9-8e2e-fa14124410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" Oct 30 05:00:51.028124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.028094 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b84178-86zsh\" (UID: \"e58f88cf-af86-49b9-8e2e-fa14124410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" Oct 30 05:00:51.028174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.028141 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8wx5\" (UniqueName: \"kubernetes.io/projected/e58f88cf-af86-49b9-8e2e-fa14124410a9-kube-api-access-d8wx5\") pod \"maas-api-revoke-sub-37b84178-86zsh\" (UID: \"e58f88cf-af86-49b9-8e2e-fa14124410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" Oct 30 05:00:51.129037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.128999 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b84178-86zsh\" (UID: \"e58f88cf-af86-49b9-8e2e-fa14124410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" Oct 30 05:00:51.129220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.129119 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b84178-86zsh\" (UID: \"e58f88cf-af86-49b9-8e2e-fa14124410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" Oct 30 05:00:51.129220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.129153 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8wx5\" (UniqueName: \"kubernetes.io/projected/e58f88cf-af86-49b9-8e2e-fa14124410a9-kube-api-access-d8wx5\") pod \"maas-api-revoke-sub-37b84178-86zsh\" (UID: \"e58f88cf-af86-49b9-8e2e-fa14124410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" Oct 30 05:00:51.129852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.129827 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b84178-86zsh\" (UID: \"e58f88cf-af86-49b9-8e2e-fa14124410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" Oct 30 05:00:51.131510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.131488 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b84178-86zsh\" (UID: \"e58f88cf-af86-49b9-8e2e-fa14124410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" Oct 30 05:00:51.150123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.150085 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8wx5\" (UniqueName: \"kubernetes.io/projected/e58f88cf-af86-49b9-8e2e-fa14124410a9-kube-api-access-d8wx5\") pod \"maas-api-revoke-sub-37b84178-86zsh\" (UID: \"e58f88cf-af86-49b9-8e2e-fa14124410a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" Oct 30 05:00:51.158665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.158642 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" Oct 30 05:00:51.458987 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:00:51.458943 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58f88cf_af86_49b9_8e2e_fa14124410a9.slice/crio-28163ca6c87320dac282ab45dabe79620665715eb3519a3a95b8f99efe2e93b5 WatchSource:0}: Error finding container 28163ca6c87320dac282ab45dabe79620665715eb3519a3a95b8f99efe2e93b5: Status 404 returned error can't find the container with id 28163ca6c87320dac282ab45dabe79620665715eb3519a3a95b8f99efe2e93b5 Oct 30 05:00:51.460483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.460456 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh"] Oct 30 05:00:51.681400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.681332 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" event={"ID":"e58f88cf-af86-49b9-8e2e-fa14124410a9","Type":"ContainerStarted","Data":"28163ca6c87320dac282ab45dabe79620665715eb3519a3a95b8f99efe2e93b5"} Oct 30 05:00:51.874841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.874816 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" Oct 30 05:00:51.943113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.943077 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-xks-ca\") pod \"145a9be0-0ea7-48ec-aab8-c186cb5af459\" (UID: \"145a9be0-0ea7-48ec-aab8-c186cb5af459\") " Oct 30 05:00:51.943282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.943191 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-openshift-ca\") pod \"145a9be0-0ea7-48ec-aab8-c186cb5af459\" (UID: \"145a9be0-0ea7-48ec-aab8-c186cb5af459\") " Oct 30 05:00:51.943349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.943294 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145a9be0-0ea7-48ec-aab8-c186cb5af459-kube-api-access-hdxlt\" (UniqueName: \"kubernetes.io/projected/145a9be0-0ea7-48ec-aab8-c186cb5af459-kube-api-access-hdxlt\") pod \"145a9be0-0ea7-48ec-aab8-c186cb5af459\" (UID: \"145a9be0-0ea7-48ec-aab8-c186cb5af459\") " Oct 30 05:00:51.944110 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.943649 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-openshift-ca" pod "145a9be0-0ea7-48ec-aab8-c186cb5af459" (UID: "145a9be0-0ea7-48ec-aab8-c186cb5af459"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:51.944110 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.944076 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:51.945603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.945577 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-xks-ca" pod "145a9be0-0ea7-48ec-aab8-c186cb5af459" (UID: "145a9be0-0ea7-48ec-aab8-c186cb5af459"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:00:51.945855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.945828 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145a9be0-0ea7-48ec-aab8-c186cb5af459-kube-api-access-hdxlt" pod "145a9be0-0ea7-48ec-aab8-c186cb5af459" (UID: "145a9be0-0ea7-48ec-aab8-c186cb5af459"). InnerVolumeSpecName "kube-api-access-hdxlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:51.998422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.998395 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg"] Oct 30 05:00:51.999070 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.999054 6587 cpu_manager.go:414] "containerMap: removing container" podUID="145a9be0-0ea7-48ec-aab8-c186cb5af459" containerName="revoke-keys" Oct 30 05:00:51.999112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.999074 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145a9be0-0ea7-48ec-aab8-c186cb5af459" containerName="revoke-keys" Oct 30 05:00:51.999189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:51.999180 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="145a9be0-0ea7-48ec-aab8-c186cb5af459" containerName="revoke-keys" Oct 30 05:00:52.002481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.002465 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.023156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.023135 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg"] Oct 30 05:00:52.045129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.045052 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.045258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.045131 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.045258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.045245 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25829776-064f-435f-8969-28682c538f0d-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.045424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.045287 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.045424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.045312 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.045544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.045460 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25829776-064f-435f-8969-28682c538f0d-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.045601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.045588 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.045658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.045643 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l9rp\" (UniqueName: \"kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-kube-api-access-6l9rp\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.045716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.045681 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.045831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.045736 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdxlt\" (UniqueName: \"kubernetes.io/projected/145a9be0-0ea7-48ec-aab8-c186cb5af459-kube-api-access-hdxlt\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:52.045831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.045752 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145a9be0-0ea7-48ec-aab8-c186cb5af459-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:52.146943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.146915 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.147078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.146954 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.147078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.147072 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.147210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.147105 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25829776-064f-435f-8969-28682c538f0d-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.147210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.147146 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.147210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.147182 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.147471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.147240 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25829776-064f-435f-8969-28682c538f0d-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.147471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.147378 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.147586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.147469 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l9rp\" (UniqueName: \"kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-kube-api-access-6l9rp\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.148025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.147995 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.148133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.148086 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.148199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.148137 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.148339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.148288 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.148878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.148725 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25829776-064f-435f-8969-28682c538f0d-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.151805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.150850 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25829776-064f-435f-8969-28682c538f0d-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.151805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.151735 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.170890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.170863 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.171010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.170992 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l9rp\" (UniqueName: \"kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-kube-api-access-6l9rp\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.179237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.179213 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb"] Oct 30 05:00:52.183823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.183805 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.211279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.211250 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb"] Oct 30 05:00:52.248658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.248620 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.248838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.248699 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.248838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.248727 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.248954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.248870 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.248954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.248906 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.249034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.248975 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmc72\" (UniqueName: \"kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-kube-api-access-pmc72\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.249069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.249031 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.249069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.249055 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dc2a23-6d11-4863-b694-646afe196859-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.249185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.249165 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dc2a23-6d11-4863-b694-646afe196859-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.277585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.277552 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f"] Oct 30 05:00:52.282679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.282654 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.320311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.320272 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:52.327409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.323704 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f"] Oct 30 05:00:52.351465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.351336 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.351465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.351412 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.351465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.351456 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dc2a23-6d11-4863-b694-646afe196859-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.351727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.351574 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4f7k\" (UniqueName: \"kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-kube-api-access-g4f7k\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.351727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.351614 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.351727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.351663 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dc2a23-6d11-4863-b694-646afe196859-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.351727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.351689 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.351939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.351740 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.351939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.351798 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.351939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.351825 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.351939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.351907 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.351939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.351928 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.352188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.351982 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.352188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.352065 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.352188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.352097 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.352188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.352122 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.352188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.352149 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.352188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.352186 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmc72\" (UniqueName: \"kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-kube-api-access-pmc72\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.355018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.352876 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.355018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.353103 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.355018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.353329 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.355018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.353678 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.356629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.356171 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dc2a23-6d11-4863-b694-646afe196859-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.356869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.356773 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.358862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.358813 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dc2a23-6d11-4863-b694-646afe196859-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.372846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.372813 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmc72\" (UniqueName: \"kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-kube-api-access-pmc72\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.374727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.374691 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.453181 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.453139 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.453348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.453223 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.453348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.453299 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.453508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.453473 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.455279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.453556 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.455279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.454051 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.455279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.454298 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.455279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.454353 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.455279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.454434 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.455279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.454506 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.455279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.454518 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4f7k\" (UniqueName: \"kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-kube-api-access-g4f7k\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.455279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.454543 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.455279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.454795 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.455808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.455417 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.458141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.458109 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.458251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.458108 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.476063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.476032 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.476592 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.476567 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4f7k\" (UniqueName: \"kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-kube-api-access-g4f7k\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.504513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.504489 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:52.600102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.600070 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:52.661895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.661863 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg"] Oct 30 05:00:52.668411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.665024 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:52.668411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.665108 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:52.668411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.665153 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:52.697551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.697517 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" event={"ID":"145a9be0-0ea7-48ec-aab8-c186cb5af459","Type":"ContainerDied","Data":"6609e16281c3c078a4d42988478854a6a7d63d8474600934fedd38c67690a8b0"} Oct 30 05:00:52.697934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.697580 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6609e16281c3c078a4d42988478854a6a7d63d8474600934fedd38c67690a8b0" Oct 30 05:00:52.698080 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.698061 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs" Oct 30 05:00:52.699833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.699809 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" event={"ID":"25829776-064f-435f-8969-28682c538f0d","Type":"ContainerStarted","Data":"d3418deabfbcca85a9f0c8caf32be2de7b81b93efb239dc11d24007f1f42b3d7"} Oct 30 05:00:52.702411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.702390 6587 generic.go:356] "Generic (PLEG): container finished" podID="e58f88cf-af86-49b9-8e2e-fa14124410a9" containerID="9ce4d5bf55227fac29003397ee9787ccbb1e05ce8c096dc1401c96e314a0bb46" exitCode=0 Oct 30 05:00:52.702536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.702522 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" event={"ID":"e58f88cf-af86-49b9-8e2e-fa14124410a9","Type":"ContainerDied","Data":"9ce4d5bf55227fac29003397ee9787ccbb1e05ce8c096dc1401c96e314a0bb46"} Oct 30 05:00:52.809952 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:00:52.809916 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dc2a23_6d11_4863_b694_646afe196859.slice/crio-70347407bdc0420f97d7a5fba4f7be8fe0ae5356c81d945d4294cedcf3496d2b WatchSource:0}: Error finding container 70347407bdc0420f97d7a5fba4f7be8fe0ae5356c81d945d4294cedcf3496d2b: Status 404 returned error can't find the container with id 70347407bdc0420f97d7a5fba4f7be8fe0ae5356c81d945d4294cedcf3496d2b Oct 30 05:00:52.810922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.810895 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb"] Oct 30 05:00:52.813315 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.813275 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:52.813401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.813377 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:52.813439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.813422 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:52.918156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.918125 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f"] Oct 30 05:00:52.931534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.931493 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:52.931638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.931579 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:52.931638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:52.931622 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:00:53.849437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:53.849398 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" event={"ID":"25829776-064f-435f-8969-28682c538f0d","Type":"ContainerStarted","Data":"ff1fb278af756643d7776ef269a2d13c404ee694d959edbd8ab8b3e56fd5e09e"} Oct 30 05:00:53.896382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:53.894430 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" event={"ID":"b3dc2a23-6d11-4863-b694-646afe196859","Type":"ContainerStarted","Data":"37049dfdc0d93f19da9f0d501c692762120b9ac1f8628f66329bcd734eefd179"} Oct 30 05:00:53.896382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:53.894483 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" event={"ID":"b3dc2a23-6d11-4863-b694-646afe196859","Type":"ContainerStarted","Data":"70347407bdc0420f97d7a5fba4f7be8fe0ae5356c81d945d4294cedcf3496d2b"} Oct 30 05:00:53.926380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:53.898231 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" event={"ID":"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35","Type":"ContainerStarted","Data":"2f0504d501789f9135992f3f10898db5782c6dce766da487ad5f745b04eababd"} Oct 30 05:00:53.926380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:53.898271 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" event={"ID":"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35","Type":"ContainerStarted","Data":"0ff993309cda266fe9895eebdd30296ee0d39ce0ba2b4b4d65be67c3dea9f241"} Oct 30 05:00:53.963382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:53.962307 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" podStartSLOduration=2.962287287 podStartE2EDuration="2.962287287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:53.957856039 +0000 UTC m=+978.485622829" watchObservedRunningTime="2026-10-30 05:00:53.962287287 +0000 UTC m=+978.490054074" Oct 30 05:00:54.012552 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.012301 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" podStartSLOduration=2.012281317 podStartE2EDuration="2.012281317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:54.005845778 +0000 UTC m=+978.533612575" watchObservedRunningTime="2026-10-30 05:00:54.012281317 +0000 UTC m=+978.540048100" Oct 30 05:00:54.111473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.109625 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" podStartSLOduration=2.109603313 podStartE2EDuration="2.109603313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:54.080914791 +0000 UTC m=+978.608681579" watchObservedRunningTime="2026-10-30 05:00:54.109603313 +0000 UTC m=+978.637370100" Oct 30 05:00:54.346640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.329223 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:00:54.426431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.411709 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:00:54.426431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.411769 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" podUID="25829776-064f-435f-8969-28682c538f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 05:00:54.506835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.506756 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:00:54.533576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.532147 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:00:54.533576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.532219 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" podUID="b3dc2a23-6d11-4863-b694-646afe196859" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 05:00:54.612385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.608751 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:00:54.612385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.610523 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:00:54.612385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.610576 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" podUID="9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 05:00:54.794668 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.794583 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" Oct 30 05:00:54.975436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.975411 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" Oct 30 05:00:54.976205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.976178 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh" event={"ID":"e58f88cf-af86-49b9-8e2e-fa14124410a9","Type":"ContainerDied","Data":"28163ca6c87320dac282ab45dabe79620665715eb3519a3a95b8f99efe2e93b5"} Oct 30 05:00:54.992402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.976381 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28163ca6c87320dac282ab45dabe79620665715eb3519a3a95b8f99efe2e93b5" Oct 30 05:00:54.995712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.995679 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-xks-ca\") pod \"e58f88cf-af86-49b9-8e2e-fa14124410a9\" (UID: \"e58f88cf-af86-49b9-8e2e-fa14124410a9\") " Oct 30 05:00:54.996340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.996318 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-openshift-ca\") pod \"e58f88cf-af86-49b9-8e2e-fa14124410a9\" (UID: \"e58f88cf-af86-49b9-8e2e-fa14124410a9\") " Oct 30 05:00:54.996928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.996807 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-openshift-ca" pod "e58f88cf-af86-49b9-8e2e-fa14124410a9" (UID: "e58f88cf-af86-49b9-8e2e-fa14124410a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:54.997209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:54.997190 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58f88cf-af86-49b9-8e2e-fa14124410a9-kube-api-access-d8wx5\" (UniqueName: \"kubernetes.io/projected/e58f88cf-af86-49b9-8e2e-fa14124410a9-kube-api-access-d8wx5\") pod \"e58f88cf-af86-49b9-8e2e-fa14124410a9\" (UID: \"e58f88cf-af86-49b9-8e2e-fa14124410a9\") " Oct 30 05:00:55.034817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:55.034580 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58f88cf-af86-49b9-8e2e-fa14124410a9-kube-api-access-d8wx5" pod "e58f88cf-af86-49b9-8e2e-fa14124410a9" (UID: "e58f88cf-af86-49b9-8e2e-fa14124410a9"). InnerVolumeSpecName "kube-api-access-d8wx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:55.037415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:55.036818 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8wx5\" (UniqueName: \"kubernetes.io/projected/e58f88cf-af86-49b9-8e2e-fa14124410a9-kube-api-access-d8wx5\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:55.037415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:55.036850 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:55.043942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:55.043905 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-xks-ca" pod "e58f88cf-af86-49b9-8e2e-fa14124410a9" (UID: "e58f88cf-af86-49b9-8e2e-fa14124410a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:00:55.174649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:55.174616 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58f88cf-af86-49b9-8e2e-fa14124410a9-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:00:55.357860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:55.357820 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:00:55.358164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:55.358110 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" podUID="25829776-064f-435f-8969-28682c538f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 05:00:55.509888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:55.509855 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:00:55.527905 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:55.527857 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" podUID="b3dc2a23-6d11-4863-b694-646afe196859" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 05:00:55.660582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:55.660028 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:00:55.660582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:55.660087 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" podUID="9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 05:00:56.267384 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:00:56.247187 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:00:56.325754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:56.325716 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:00:56.326029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:56.325995 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" podUID="25829776-064f-435f-8969-28682c538f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 05:00:56.509433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:56.508897 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:00:56.509433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:56.508952 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" podUID="b3dc2a23-6d11-4863-b694-646afe196859" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 05:00:56.601382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:56.600662 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:00:56.601382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:56.600722 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" podUID="9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 05:00:57.332384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:57.324723 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:00:57.332384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:57.324783 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" podUID="25829776-064f-435f-8969-28682c538f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 05:00:57.514386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:57.511766 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:00:57.514386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:57.511837 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" podUID="b3dc2a23-6d11-4863-b694-646afe196859" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 05:00:57.616153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:57.616066 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:00:57.638477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:57.638430 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" podUID="9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 05:00:58.329383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:58.325740 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:00:58.329383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:58.325802 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" podUID="25829776-064f-435f-8969-28682c538f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 05:00:58.507380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:58.505438 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:00:58.507380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:58.505501 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" podUID="b3dc2a23-6d11-4863-b694-646afe196859" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 05:00:58.615462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:58.601746 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:00:58.615462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:58.601811 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" podUID="9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 05:00:59.326763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:59.326724 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:00:59.327056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:59.327023 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" podUID="25829776-064f-435f-8969-28682c538f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 05:00:59.504930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:59.504891 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:00:59.505135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:59.504970 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" podUID="b3dc2a23-6d11-4863-b694-646afe196859" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 05:00:59.601136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:59.601052 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:00:59.601136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:00:59.601112 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" podUID="9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 05:01:00.615385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:00.614690 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:01:00.618575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:00.616328 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:01:00.618575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:00.618501 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:01:01.029425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.028919 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:01:01.030787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.030760 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:01:01.034662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.034635 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:01:01.082928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.082894 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:01:01.084000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.083976 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:01:01.085002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.084972 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:01:01.566050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.566007 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz"] Oct 30 05:01:01.567092 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.567068 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e58f88cf-af86-49b9-8e2e-fa14124410a9" containerName="revoke-keys" Oct 30 05:01:01.567199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.567092 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58f88cf-af86-49b9-8e2e-fa14124410a9" containerName="revoke-keys" Oct 30 05:01:01.567268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.567240 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58f88cf-af86-49b9-8e2e-fa14124410a9" containerName="revoke-keys" Oct 30 05:01:01.572553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.572531 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.588484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.588460 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d55037\"/\"kube-root-ca.crt\"" Oct 30 05:01:01.590689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.590418 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d55037\"/\"openshift-service-ca.crt\"" Oct 30 05:01:01.590689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.590496 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d55037\"/\"default-dockercfg-w4dgw\"" Oct 30 05:01:01.591859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.591839 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz"] Oct 30 05:01:01.598395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.595430 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d55037\"/\"distinct-w4-d55037-kserve-self-signed-certs\"" Oct 30 05:01:01.599967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.599944 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4"] Oct 30 05:01:01.604691 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.604671 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" Oct 30 05:01:01.631337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.631313 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4"] Oct 30 05:01:01.688400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.688341 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b1030e5-r8xl4\" (UID: \"7b3f500a-2154-4497-bb3c-d61fddaac62a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" Oct 30 05:01:01.688605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.688577 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tls-certs\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.688700 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.688680 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79kb4\" (UniqueName: \"kubernetes.io/projected/7b3f500a-2154-4497-bb3c-d61fddaac62a-kube-api-access-79kb4\") pod \"maas-api-revoke-sub-8b1030e5-r8xl4\" (UID: \"7b3f500a-2154-4497-bb3c-d61fddaac62a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" Oct 30 05:01:01.688753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.688723 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-dshm\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.688800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.688746 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tmp-dir\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.688983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.688937 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:01:01.688983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.688968 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:01:01.689114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.688995 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-model-cache\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.689114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.689040 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:01:01.689114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.689061 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:01:01.689114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.689069 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b1030e5-r8xl4\" (UID: \"7b3f500a-2154-4497-bb3c-d61fddaac62a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" Oct 30 05:01:01.689331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.689129 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-home\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.689331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.689218 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltnzj\" (UniqueName: \"kubernetes.io/projected/5e79efb5-e47f-4837-8193-0cd9f6343bc4-kube-api-access-ltnzj\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.802383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.798533 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-home\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.802383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.798624 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltnzj\" (UniqueName: \"kubernetes.io/projected/5e79efb5-e47f-4837-8193-0cd9f6343bc4-kube-api-access-ltnzj\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.802383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.798720 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b1030e5-r8xl4\" (UID: \"7b3f500a-2154-4497-bb3c-d61fddaac62a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" Oct 30 05:01:01.802383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.798827 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tls-certs\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.802383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.798908 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79kb4\" (UniqueName: \"kubernetes.io/projected/7b3f500a-2154-4497-bb3c-d61fddaac62a-kube-api-access-79kb4\") pod \"maas-api-revoke-sub-8b1030e5-r8xl4\" (UID: \"7b3f500a-2154-4497-bb3c-d61fddaac62a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" Oct 30 05:01:01.802383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.798938 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-dshm\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.802383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.798962 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tmp-dir\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.802383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.799050 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-model-cache\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.802383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.799120 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b1030e5-r8xl4\" (UID: \"7b3f500a-2154-4497-bb3c-d61fddaac62a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" Oct 30 05:01:01.802383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.799997 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b1030e5-r8xl4\" (UID: \"7b3f500a-2154-4497-bb3c-d61fddaac62a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" Oct 30 05:01:01.802383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.800330 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-home\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.808228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.806095 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-model-cache\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.808228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.805984 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tmp-dir\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.808228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.807939 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b1030e5-r8xl4\" (UID: \"7b3f500a-2154-4497-bb3c-d61fddaac62a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" Oct 30 05:01:01.814379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.809550 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-dshm\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.814379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.811000 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tls-certs\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.832153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.824392 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltnzj\" (UniqueName: \"kubernetes.io/projected/5e79efb5-e47f-4837-8193-0cd9f6343bc4-kube-api-access-ltnzj\") pod \"distinct-w4-d55037-kserve-7dc5744b9c-dltpz\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.832153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.826858 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79kb4\" (UniqueName: \"kubernetes.io/projected/7b3f500a-2154-4497-bb3c-d61fddaac62a-kube-api-access-79kb4\") pod \"maas-api-revoke-sub-8b1030e5-r8xl4\" (UID: \"7b3f500a-2154-4497-bb3c-d61fddaac62a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" Oct 30 05:01:01.892902 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.892864 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:01.922460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:01.922424 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" Oct 30 05:01:02.233577 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:01:02.233536 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e79efb5_e47f_4837_8193_0cd9f6343bc4.slice/crio-7533e7e74596d15ebdca1d308787b15c2a8805ebe661dd4dc1604e9705831647 WatchSource:0}: Error finding container 7533e7e74596d15ebdca1d308787b15c2a8805ebe661dd4dc1604e9705831647: Status 404 returned error can't find the container with id 7533e7e74596d15ebdca1d308787b15c2a8805ebe661dd4dc1604e9705831647 Oct 30 05:01:02.237515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:02.237487 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz"] Oct 30 05:01:02.264119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:02.264094 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4"] Oct 30 05:01:02.267755 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:01:02.267725 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3f500a_2154_4497_bb3c_d61fddaac62a.slice/crio-6be2172a375cf791af0ca3e4c09d780dab56c9e507b38fa35beb79c95d0e1f73 WatchSource:0}: Error finding container 6be2172a375cf791af0ca3e4c09d780dab56c9e507b38fa35beb79c95d0e1f73: Status 404 returned error can't find the container with id 6be2172a375cf791af0ca3e4c09d780dab56c9e507b38fa35beb79c95d0e1f73 Oct 30 05:01:03.250118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:03.250077 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" event={"ID":"5e79efb5-e47f-4837-8193-0cd9f6343bc4","Type":"ContainerStarted","Data":"61a646507e98ca985d624b02ba40160aec40498453f64db0037ca3e39825c0d8"} Oct 30 05:01:03.250731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:03.250705 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" event={"ID":"5e79efb5-e47f-4837-8193-0cd9f6343bc4","Type":"ContainerStarted","Data":"7533e7e74596d15ebdca1d308787b15c2a8805ebe661dd4dc1604e9705831647"} Oct 30 05:01:03.254224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:03.254200 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" event={"ID":"7b3f500a-2154-4497-bb3c-d61fddaac62a","Type":"ContainerStarted","Data":"a22a26a03bd54dcaa78b0c0696688594f342c8f31647e229eba287de83628091"} Oct 30 05:01:03.254423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:03.254404 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" event={"ID":"7b3f500a-2154-4497-bb3c-d61fddaac62a","Type":"ContainerStarted","Data":"6be2172a375cf791af0ca3e4c09d780dab56c9e507b38fa35beb79c95d0e1f73"} Oct 30 05:01:03.321452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:03.311063 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podStartSLOduration=2.311042665 podStartE2EDuration="2.311042665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:03.291819894 +0000 UTC m=+987.819586683" watchObservedRunningTime="2026-10-30 05:01:03.311042665 +0000 UTC m=+987.838809452" Oct 30 05:01:03.348389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:03.337195 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" podStartSLOduration=2.337177296 podStartE2EDuration="2.337177296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:03.336280402 +0000 UTC m=+987.864047185" watchObservedRunningTime="2026-10-30 05:01:03.337177296 +0000 UTC m=+987.864944082" Oct 30 05:01:03.909382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:03.893475 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:04.407124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:04.406632 6587 generic.go:356] "Generic (PLEG): container finished" podID="7b3f500a-2154-4497-bb3c-d61fddaac62a" containerID="a22a26a03bd54dcaa78b0c0696688594f342c8f31647e229eba287de83628091" exitCode=0 Oct 30 05:01:04.407985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:04.407834 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" event={"ID":"7b3f500a-2154-4497-bb3c-d61fddaac62a","Type":"ContainerDied","Data":"a22a26a03bd54dcaa78b0c0696688594f342c8f31647e229eba287de83628091"} Oct 30 05:01:04.424829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:04.424792 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:04.716437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:04.715535 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:01:04.720472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:04.717294 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:01:05.427389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:05.426564 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:05.428030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:05.427429 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:01:05.465826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:05.465380 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:01:05.465826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:05.465439 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:05.503840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:05.503414 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:05.526935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:05.526477 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:05.662383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:05.621929 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:05.787435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:05.716440 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:05.787435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:05.746484 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:01:05.795383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:05.762476 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:06.432264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:06.432213 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:06.521948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:06.518119 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:06.583923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:06.583862 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:01:06.831822 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:06.831777 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 30 05:01:06.856382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:06.836793 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:01:07.014706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.014653 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:01:07.063134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.063089 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:01:07.145101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.120483 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:01:07.326287 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:01:07.202189 6587 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.626s" Oct 30 05:01:07.326717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.326696 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:01:07.326925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.326911 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:01:07.352715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.325854 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:07.386383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.382444 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:01:07.449936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.449902 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" Oct 30 05:01:07.535124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.434344 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:01:07.539384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.458983 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:01:07.539965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.539941 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d"] Oct 30 05:01:07.548386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.541140 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7b3f500a-2154-4497-bb3c-d61fddaac62a" containerName="revoke-keys" Oct 30 05:01:07.549338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.549307 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3f500a-2154-4497-bb3c-d61fddaac62a" containerName="revoke-keys" Oct 30 05:01:07.549846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.549825 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3f500a-2154-4497-bb3c-d61fddaac62a" containerName="revoke-keys" Oct 30 05:01:07.579493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.579463 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:01:07.579760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.579741 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:01:07.583378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.583330 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:07.713631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.490405 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:07.721268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.525054 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:07.721268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.711028 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:01:07.865014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.864222 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3f500a-2154-4497-bb3c-d61fddaac62a-kube-api-access-79kb4\" (UniqueName: \"kubernetes.io/projected/7b3f500a-2154-4497-bb3c-d61fddaac62a-kube-api-access-79kb4\") pod \"7b3f500a-2154-4497-bb3c-d61fddaac62a\" (UID: \"7b3f500a-2154-4497-bb3c-d61fddaac62a\") " Oct 30 05:01:07.865014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.864349 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-xks-ca\") pod \"7b3f500a-2154-4497-bb3c-d61fddaac62a\" (UID: \"7b3f500a-2154-4497-bb3c-d61fddaac62a\") " Oct 30 05:01:07.865014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.864450 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-openshift-ca\") pod \"7b3f500a-2154-4497-bb3c-d61fddaac62a\" (UID: \"7b3f500a-2154-4497-bb3c-d61fddaac62a\") " Oct 30 05:01:07.865014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.864746 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6wzz\" (UniqueName: \"kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-kube-api-access-s6wzz\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:07.865014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.864883 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:07.865014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.864927 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:07.865014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.864958 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:07.877535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.866509 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:07.877535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.866645 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:07.877535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.866708 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:07.877535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.866744 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:07.896529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.878166 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:07.896529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.884474 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-openshift-ca" pod "7b3f500a-2154-4497-bb3c-d61fddaac62a" (UID: "7b3f500a-2154-4497-bb3c-d61fddaac62a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:01:07.996711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:07.996682 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8"] Oct 30 05:01:08.003069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.003040 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d"] Oct 30 05:01:08.003297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.003281 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8"] Oct 30 05:01:08.003475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.003462 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:01:08.003692 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.003677 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.007485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.007238 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:01:08.010044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.010026 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:01:08.033808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.033717 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.033994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.033827 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.033994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.033862 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.035895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.035388 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:01:08.035895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.035850 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.036545 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.038459 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.039603 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.040820 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.048550 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.048591 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.048620 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.048737 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.048765 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6wzz\" (UniqueName: \"kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-kube-api-access-s6wzz\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.048887 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.041154 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.049443 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.042622 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.043538 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3f500a-2154-4497-bb3c-d61fddaac62a-kube-api-access-79kb4" pod "7b3f500a-2154-4497-bb3c-d61fddaac62a" (UID: "7b3f500a-2154-4497-bb3c-d61fddaac62a"). InnerVolumeSpecName "kube-api-access-79kb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.045186 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-xks-ca" pod "7b3f500a-2154-4497-bb3c-d61fddaac62a" (UID: "7b3f500a-2154-4497-bb3c-d61fddaac62a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.050320 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.052612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.050562 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.069114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.054118 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:01:08.079860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.079797 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6wzz\" (UniqueName: \"kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-kube-api-access-s6wzz\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.082548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.082525 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.140184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.140151 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:08.151593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.151562 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.151746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.151621 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7z4h\" (UniqueName: \"kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-kube-api-access-f7z4h\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.151834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.151766 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.151834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.151801 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.151834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.151824 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.152007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.151886 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.152097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.152079 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.152165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.152115 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.152165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.152145 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.152310 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.152266 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79kb4\" (UniqueName: \"kubernetes.io/projected/7b3f500a-2154-4497-bb3c-d61fddaac62a-kube-api-access-79kb4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:08.152310 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.152280 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3f500a-2154-4497-bb3c-d61fddaac62a-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:08.225825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.224126 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" Oct 30 05:01:08.226104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.226029 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4" event={"ID":"7b3f500a-2154-4497-bb3c-d61fddaac62a","Type":"ContainerDied","Data":"6be2172a375cf791af0ca3e4c09d780dab56c9e507b38fa35beb79c95d0e1f73"} Oct 30 05:01:08.226237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.226204 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6be2172a375cf791af0ca3e4c09d780dab56c9e507b38fa35beb79c95d0e1f73" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.253660 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.253726 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7z4h\" (UniqueName: \"kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-kube-api-access-f7z4h\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.253795 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.253821 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.253846 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.253896 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.254088 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.254121 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.254146 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.254632 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.257999 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.258532 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.258883 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.265624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.262630 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.266546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.266221 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.270696 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.270501 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.285525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.284196 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.285525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.284572 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7z4h\" (UniqueName: \"kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-kube-api-access-f7z4h\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.444817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.444782 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:08.484236 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:01:08.484192 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32ca7a5_ef18_422d_aa8b_cb90ddf88737.slice/crio-f575598cdeb72f8f9ede5b9a30efc37f9ccc05baf83c4eb6fb9e8bfb45215e7d WatchSource:0}: Error finding container f575598cdeb72f8f9ede5b9a30efc37f9ccc05baf83c4eb6fb9e8bfb45215e7d: Status 404 returned error can't find the container with id f575598cdeb72f8f9ede5b9a30efc37f9ccc05baf83c4eb6fb9e8bfb45215e7d Oct 30 05:01:08.486309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.486276 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d"] Oct 30 05:01:08.490038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.490003 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:08.490136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.490072 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:08.490136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.490113 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:08.767046 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:01:08.767014 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode298688e_c592_47ed_b2e0_2f06a53e6b9a.slice/crio-4aa9236dbf2a892e8edb97f9baa38e7ce231de37d649824030503c74861d0387 WatchSource:0}: Error finding container 4aa9236dbf2a892e8edb97f9baa38e7ce231de37d649824030503c74861d0387: Status 404 returned error can't find the container with id 4aa9236dbf2a892e8edb97f9baa38e7ce231de37d649824030503c74861d0387 Oct 30 05:01:08.768762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.768734 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8"] Oct 30 05:01:08.769808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.769775 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:08.769914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.769841 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:08.769914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:08.769869 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:09.257839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:09.257792 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" event={"ID":"e298688e-c592-47ed-b2e0-2f06a53e6b9a","Type":"ContainerStarted","Data":"6ae04dede44f0081ae741ddd111e38a6477660c193f8c1fcbe147e516d2a7866"} Oct 30 05:01:09.257994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:09.257869 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" event={"ID":"e298688e-c592-47ed-b2e0-2f06a53e6b9a","Type":"ContainerStarted","Data":"4aa9236dbf2a892e8edb97f9baa38e7ce231de37d649824030503c74861d0387"} Oct 30 05:01:09.260559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:09.260527 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" event={"ID":"e32ca7a5-ef18-422d-aa8b-cb90ddf88737","Type":"ContainerStarted","Data":"495044dd984b91a950fc3e87aa35449d381447b0c2a559085b54a7d4ed5f7ea2"} Oct 30 05:01:09.260703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:09.260602 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" event={"ID":"e32ca7a5-ef18-422d-aa8b-cb90ddf88737","Type":"ContainerStarted","Data":"f575598cdeb72f8f9ede5b9a30efc37f9ccc05baf83c4eb6fb9e8bfb45215e7d"} Oct 30 05:01:09.314104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:09.314031 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" podStartSLOduration=3.314012695 podStartE2EDuration="3.314012695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:09.301228522 +0000 UTC m=+993.828995308" watchObservedRunningTime="2026-10-30 05:01:09.314012695 +0000 UTC m=+993.841779481" Oct 30 05:01:09.369632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:09.369573 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" podStartSLOduration=3.3695472300000002 podStartE2EDuration="3.36954723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:09.349003312 +0000 UTC m=+993.876770099" watchObservedRunningTime="2026-10-30 05:01:09.36954723 +0000 UTC m=+993.897314016" Oct 30 05:01:09.448380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:09.447419 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:09.448380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:09.447587 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:01:09.448380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:09.447633 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" podUID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 05:01:10.143725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:10.143252 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:10.143725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:10.143403 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:01:10.143725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:10.143453 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" podUID="e32ca7a5-ef18-422d-aa8b-cb90ddf88737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 05:01:10.449437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:10.448823 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:01:10.449437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:10.448892 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" podUID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 05:01:11.144381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:11.144146 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:01:11.144381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:11.144205 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" podUID="e32ca7a5-ef18-422d-aa8b-cb90ddf88737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 05:01:11.465440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:11.464534 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:01:11.465440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:11.464614 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" podUID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 05:01:11.893990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:11.893949 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:11.906303 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:11.906270 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:12.162435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:12.142422 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:01:12.166379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:12.142578 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" podUID="e32ca7a5-ef18-422d-aa8b-cb90ddf88737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 05:01:12.376603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:12.376569 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:12.454440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:12.451081 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:01:12.454440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:12.451142 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" podUID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 05:01:13.141383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.140781 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:01:13.141383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.140847 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" podUID="e32ca7a5-ef18-422d-aa8b-cb90ddf88737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 05:01:13.198055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.198022 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl"] Oct 30 05:01:13.202914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.202890 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" Oct 30 05:01:13.260381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.259716 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:01:13.260381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.259829 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:01:13.260381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.259955 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:01:13.260381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.260027 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:01:13.263491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.263466 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl"] Oct 30 05:01:13.273433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.270557 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17915466-615b-467d-9483-211240c24e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-803c8e40-z26bl\" (UID: \"17915466-615b-467d-9483-211240c24e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" Oct 30 05:01:13.273433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.270650 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfgrd\" (UniqueName: \"kubernetes.io/projected/17915466-615b-467d-9483-211240c24e24-kube-api-access-tfgrd\") pod \"maas-api-revoke-sub-803c8e40-z26bl\" (UID: \"17915466-615b-467d-9483-211240c24e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" Oct 30 05:01:13.273433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.270780 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17915466-615b-467d-9483-211240c24e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-803c8e40-z26bl\" (UID: \"17915466-615b-467d-9483-211240c24e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" Oct 30 05:01:13.374522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.374342 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgrd\" (UniqueName: \"kubernetes.io/projected/17915466-615b-467d-9483-211240c24e24-kube-api-access-tfgrd\") pod \"maas-api-revoke-sub-803c8e40-z26bl\" (UID: \"17915466-615b-467d-9483-211240c24e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" Oct 30 05:01:13.374733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.374540 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17915466-615b-467d-9483-211240c24e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-803c8e40-z26bl\" (UID: \"17915466-615b-467d-9483-211240c24e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" Oct 30 05:01:13.374812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.374752 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17915466-615b-467d-9483-211240c24e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-803c8e40-z26bl\" (UID: \"17915466-615b-467d-9483-211240c24e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" Oct 30 05:01:13.376157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.376122 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17915466-615b-467d-9483-211240c24e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-803c8e40-z26bl\" (UID: \"17915466-615b-467d-9483-211240c24e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" Oct 30 05:01:13.378130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.378102 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17915466-615b-467d-9483-211240c24e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-803c8e40-z26bl\" (UID: \"17915466-615b-467d-9483-211240c24e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" Oct 30 05:01:13.398825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.398765 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgrd\" (UniqueName: \"kubernetes.io/projected/17915466-615b-467d-9483-211240c24e24-kube-api-access-tfgrd\") pod \"maas-api-revoke-sub-803c8e40-z26bl\" (UID: \"17915466-615b-467d-9483-211240c24e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" Oct 30 05:01:13.470430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.469820 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:01:13.470430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.469904 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" podUID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 05:01:13.537872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.536603 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" Oct 30 05:01:13.986846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:13.986818 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl"] Oct 30 05:01:14.149381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:14.148678 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:01:14.149381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:14.148737 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" podUID="e32ca7a5-ef18-422d-aa8b-cb90ddf88737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 05:01:14.367384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:14.365490 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" event={"ID":"17915466-615b-467d-9483-211240c24e24","Type":"ContainerStarted","Data":"13570287b4b6838cca52f00e580730eb72e6ad62a2aae9d8d219853ec467ff17"} Oct 30 05:01:14.448794 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:14.448753 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:01:14.449062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:14.449034 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" podUID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 05:01:15.143853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.143815 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:01:15.144160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.144118 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" podUID="e32ca7a5-ef18-422d-aa8b-cb90ddf88737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 05:01:15.376824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.376787 6587 generic.go:356] "Generic (PLEG): container finished" podID="17915466-615b-467d-9483-211240c24e24" containerID="db0ed870d46a158004a9f962edf724429cdf28df31586e5435b63cfb7a11ac5c" exitCode=0 Oct 30 05:01:15.377242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.376873 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" event={"ID":"17915466-615b-467d-9483-211240c24e24","Type":"ContainerDied","Data":"db0ed870d46a158004a9f962edf724429cdf28df31586e5435b63cfb7a11ac5c"} Oct 30 05:01:15.428527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.428437 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw"] Oct 30 05:01:15.434664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.434631 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" Oct 30 05:01:15.450423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.449276 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:01:15.450423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.449335 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" podUID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 05:01:15.458798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.458766 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw"] Oct 30 05:01:15.534842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.534797 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-334b30c4-7bszw\" (UID: \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" Oct 30 05:01:15.535050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.534873 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-334b30c4-7bszw\" (UID: \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" Oct 30 05:01:15.535050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.535005 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9vnx\" (UniqueName: \"kubernetes.io/projected/63492924-2c05-4dda-86d2-b0fb7b03b0f8-kube-api-access-z9vnx\") pod \"maas-api-revoke-sub-334b30c4-7bszw\" (UID: \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" Oct 30 05:01:15.636489 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.636448 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-334b30c4-7bszw\" (UID: \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" Oct 30 05:01:15.636740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.636719 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-334b30c4-7bszw\" (UID: \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" Oct 30 05:01:15.636925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.636908 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9vnx\" (UniqueName: \"kubernetes.io/projected/63492924-2c05-4dda-86d2-b0fb7b03b0f8-kube-api-access-z9vnx\") pod \"maas-api-revoke-sub-334b30c4-7bszw\" (UID: \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" Oct 30 05:01:15.638512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.638487 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-334b30c4-7bszw\" (UID: \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" Oct 30 05:01:15.641987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.641940 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-334b30c4-7bszw\" (UID: \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" Oct 30 05:01:15.658553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.658525 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9vnx\" (UniqueName: \"kubernetes.io/projected/63492924-2c05-4dda-86d2-b0fb7b03b0f8-kube-api-access-z9vnx\") pod \"maas-api-revoke-sub-334b30c4-7bszw\" (UID: \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" Oct 30 05:01:15.759051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:15.758958 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" Oct 30 05:01:16.098429 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:01:16.097994 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63492924_2c05_4dda_86d2_b0fb7b03b0f8.slice/crio-6a71edd091942469a853ba97dca6d805065f13b12639197e9110723f7b983259 WatchSource:0}: Error finding container 6a71edd091942469a853ba97dca6d805065f13b12639197e9110723f7b983259: Status 404 returned error can't find the container with id 6a71edd091942469a853ba97dca6d805065f13b12639197e9110723f7b983259 Oct 30 05:01:16.100998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.100969 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw"] Oct 30 05:01:16.388909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.388804 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" event={"ID":"63492924-2c05-4dda-86d2-b0fb7b03b0f8","Type":"ContainerStarted","Data":"6a71edd091942469a853ba97dca6d805065f13b12639197e9110723f7b983259"} Oct 30 05:01:16.493267 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.491684 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:16.498225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.498067 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:16.501712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.499681 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:01:16.595625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.595597 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" Oct 30 05:01:16.789731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.789697 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17915466-615b-467d-9483-211240c24e24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17915466-615b-467d-9483-211240c24e24-maas-api-xks-ca\") pod \"17915466-615b-467d-9483-211240c24e24\" (UID: \"17915466-615b-467d-9483-211240c24e24\") " Oct 30 05:01:16.789918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.789769 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17915466-615b-467d-9483-211240c24e24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17915466-615b-467d-9483-211240c24e24-maas-api-openshift-ca\") pod \"17915466-615b-467d-9483-211240c24e24\" (UID: \"17915466-615b-467d-9483-211240c24e24\") " Oct 30 05:01:16.790219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.790107 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17915466-615b-467d-9483-211240c24e24-maas-api-openshift-ca" pod "17915466-615b-467d-9483-211240c24e24" (UID: "17915466-615b-467d-9483-211240c24e24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:01:16.790219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.790141 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17915466-615b-467d-9483-211240c24e24-kube-api-access-tfgrd\" (UniqueName: \"kubernetes.io/projected/17915466-615b-467d-9483-211240c24e24-kube-api-access-tfgrd\") pod \"17915466-615b-467d-9483-211240c24e24\" (UID: \"17915466-615b-467d-9483-211240c24e24\") " Oct 30 05:01:16.790797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.790774 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17915466-615b-467d-9483-211240c24e24-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:16.791979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.791953 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17915466-615b-467d-9483-211240c24e24-maas-api-xks-ca" pod "17915466-615b-467d-9483-211240c24e24" (UID: "17915466-615b-467d-9483-211240c24e24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:01:16.792276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.792259 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17915466-615b-467d-9483-211240c24e24-kube-api-access-tfgrd" pod "17915466-615b-467d-9483-211240c24e24" (UID: "17915466-615b-467d-9483-211240c24e24"). InnerVolumeSpecName "kube-api-access-tfgrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:01:16.822431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.822386 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:16.823974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.823953 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:16.826741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.826721 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:01:16.892614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.892580 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfgrd\" (UniqueName: \"kubernetes.io/projected/17915466-615b-467d-9483-211240c24e24-kube-api-access-tfgrd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:16.892614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:16.892613 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17915466-615b-467d-9483-211240c24e24-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:17.398064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:17.397989 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" Oct 30 05:01:17.398064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:17.398036 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl" event={"ID":"17915466-615b-467d-9483-211240c24e24","Type":"ContainerDied","Data":"13570287b4b6838cca52f00e580730eb72e6ad62a2aae9d8d219853ec467ff17"} Oct 30 05:01:17.398746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:17.398092 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13570287b4b6838cca52f00e580730eb72e6ad62a2aae9d8d219853ec467ff17" Oct 30 05:01:17.399957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:17.399929 6587 generic.go:356] "Generic (PLEG): container finished" podID="63492924-2c05-4dda-86d2-b0fb7b03b0f8" containerID="6a9ccd435ca7a9a95ec26bf4b87dc06d70f6eb6b42662e19905a90e18912e578" exitCode=0 Oct 30 05:01:17.400060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:17.400020 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" event={"ID":"63492924-2c05-4dda-86d2-b0fb7b03b0f8","Type":"ContainerDied","Data":"6a9ccd435ca7a9a95ec26bf4b87dc06d70f6eb6b42662e19905a90e18912e578"} Oct 30 05:01:18.568177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:18.568146 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" Oct 30 05:01:18.638500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:18.638464 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-openshift-ca\") pod \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\" (UID: \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\") " Oct 30 05:01:18.638849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:18.638822 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63492924-2c05-4dda-86d2-b0fb7b03b0f8-kube-api-access-z9vnx\" (UniqueName: \"kubernetes.io/projected/63492924-2c05-4dda-86d2-b0fb7b03b0f8-kube-api-access-z9vnx\") pod \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\" (UID: \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\") " Oct 30 05:01:18.639135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:18.639114 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-xks-ca\") pod \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\" (UID: \"63492924-2c05-4dda-86d2-b0fb7b03b0f8\") " Oct 30 05:01:18.641421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:18.640946 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-openshift-ca" pod "63492924-2c05-4dda-86d2-b0fb7b03b0f8" (UID: "63492924-2c05-4dda-86d2-b0fb7b03b0f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:01:18.644757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:18.644729 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-xks-ca" pod "63492924-2c05-4dda-86d2-b0fb7b03b0f8" (UID: "63492924-2c05-4dda-86d2-b0fb7b03b0f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:01:18.646052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:18.646028 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63492924-2c05-4dda-86d2-b0fb7b03b0f8-kube-api-access-z9vnx" pod "63492924-2c05-4dda-86d2-b0fb7b03b0f8" (UID: "63492924-2c05-4dda-86d2-b0fb7b03b0f8"). InnerVolumeSpecName "kube-api-access-z9vnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:01:18.746391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:18.742687 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:18.746391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:18.742724 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63492924-2c05-4dda-86d2-b0fb7b03b0f8-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:18.746391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:18.742734 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9vnx\" (UniqueName: \"kubernetes.io/projected/63492924-2c05-4dda-86d2-b0fb7b03b0f8-kube-api-access-z9vnx\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:19.416246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:19.416216 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" Oct 30 05:01:19.416466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:19.416259 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw" event={"ID":"63492924-2c05-4dda-86d2-b0fb7b03b0f8","Type":"ContainerDied","Data":"6a71edd091942469a853ba97dca6d805065f13b12639197e9110723f7b983259"} Oct 30 05:01:19.416466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:19.416312 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a71edd091942469a853ba97dca6d805065f13b12639197e9110723f7b983259" Oct 30 05:01:21.706109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.706065 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq"] Oct 30 05:01:21.707171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.707143 6587 cpu_manager.go:414] "containerMap: removing container" podUID="17915466-615b-467d-9483-211240c24e24" containerName="revoke-keys" Oct 30 05:01:21.707171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.707171 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17915466-615b-467d-9483-211240c24e24" containerName="revoke-keys" Oct 30 05:01:21.707352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.707265 6587 cpu_manager.go:414] "containerMap: removing container" podUID="63492924-2c05-4dda-86d2-b0fb7b03b0f8" containerName="revoke-keys" Oct 30 05:01:21.707352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.707274 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63492924-2c05-4dda-86d2-b0fb7b03b0f8" containerName="revoke-keys" Oct 30 05:01:21.707512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.707468 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="17915466-615b-467d-9483-211240c24e24" containerName="revoke-keys" Oct 30 05:01:21.707512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.707503 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="63492924-2c05-4dda-86d2-b0fb7b03b0f8" containerName="revoke-keys" Oct 30 05:01:21.713267 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.713242 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.726834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.726805 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l"] Oct 30 05:01:21.760606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.760566 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq"] Oct 30 05:01:21.760780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.760627 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l"] Oct 30 05:01:21.760780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.760747 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.786845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.786810 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.787042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.787016 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.787114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.787077 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.787169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.787128 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.787221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.787166 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.787280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.787225 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bt7k\" (UniqueName: \"kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-kube-api-access-2bt7k\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.787337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.787296 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.787481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.787460 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.787569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.787551 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896a9789-3a1d-4ec0-abfb-39131100ba0d-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.891307 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.891399 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.891456 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.891488 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.891532 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.891573 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.891601 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.891660 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bt7k\" (UniqueName: \"kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-kube-api-access-2bt7k\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.891714 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.891759 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.891804 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj24b\" (UniqueName: \"kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-kube-api-access-bj24b\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.891906 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.891955 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.892012 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.892043 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896a9789-3a1d-4ec0-abfb-39131100ba0d-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.892120 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407c363e-e3da-40c7-adb6-0d5e8880fd55-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.892156 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.892197 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.892788 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.893380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.893035 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.920382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.901584 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.920382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.903287 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896a9789-3a1d-4ec0-abfb-39131100ba0d-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.920382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.903609 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.920382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.904177 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.951239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.951153 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.954393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.954348 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bt7k\" (UniqueName: \"kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-kube-api-access-2bt7k\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.956757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.956679 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:21.994407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.993930 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.994407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.994021 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.994407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.994083 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj24b\" (UniqueName: \"kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-kube-api-access-bj24b\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.994407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.994201 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.994407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.994259 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.994407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.994329 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407c363e-e3da-40c7-adb6-0d5e8880fd55-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.995652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.994914 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.995652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.995017 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:21.995652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:21.995075 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:22.007892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:22.007861 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:22.015704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:22.015677 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:22.057672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:22.023733 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407c363e-e3da-40c7-adb6-0d5e8880fd55-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:22.057672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:22.024014 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:22.057672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:22.024337 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:22.057672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:22.024588 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:22.066082 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:22.066051 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:22.169373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:22.169296 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:22.361765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:22.361731 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:22.369565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:22.366335 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj24b\" (UniqueName: \"kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-kube-api-access-bj24b\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:22.390383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:22.385651 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:23.021382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:23.019171 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:23.095382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:23.084345 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:23.998453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:23.980440 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:01:24.076753 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:01:24.071857 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896a9789_3a1d_4ec0_abfb_39131100ba0d.slice/crio-f263bbf38e7eacad94bb7a876adb06399f1ab5eb3ffbe0c198f4a1ca396e7c9c WatchSource:0}: Error finding container f263bbf38e7eacad94bb7a876adb06399f1ab5eb3ffbe0c198f4a1ca396e7c9c: Status 404 returned error can't find the container with id f263bbf38e7eacad94bb7a876adb06399f1ab5eb3ffbe0c198f4a1ca396e7c9c Oct 30 05:01:24.076753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.075447 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:01:24.076753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.076151 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:24.076753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.076467 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:24.288497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.249802 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:24.288497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.250415 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:01:24.363295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.363253 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq"] Oct 30 05:01:24.367572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.363599 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:01:24.419679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.405210 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:01:24.420969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.037598 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:01:24.420969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.420241 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:01:24.530060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.530031 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l"] Oct 30 05:01:24.920911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.904331 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:24.920911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.904806 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:24.920911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.905175 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:24.983727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:24.983663 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:25.147433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.085241 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:01:25.214515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.126848 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:01:25.308384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.127097 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:01:25.308384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.261065 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:01:25.308384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.127342 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:01:25.308777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.308734 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:01:25.309025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.308908 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:01:25.309789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.309765 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:01:25.455432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.422681 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:25.562378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.539791 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:01:25.611693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.611651 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:25.636124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.636079 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:25.639489 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.639450 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:25.694695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:25.694651 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:26.113471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:26.092617 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:26.210383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:26.203653 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:01:26.212062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:26.212039 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:26.235875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:26.235836 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:01:26.313397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:26.244903 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:26.353385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:26.325666 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:01:26.591445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:26.591402 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" event={"ID":"407c363e-e3da-40c7-adb6-0d5e8880fd55","Type":"ContainerStarted","Data":"84b9c4391fd29d0fda652ce86c9902706a6f7dc66116d31b25cb90a5a386475c"} Oct 30 05:01:26.598536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:26.598483 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:26.605388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:26.599015 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:26.623757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:26.623714 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" event={"ID":"896a9789-3a1d-4ec0-abfb-39131100ba0d","Type":"ContainerStarted","Data":"f263bbf38e7eacad94bb7a876adb06399f1ab5eb3ffbe0c198f4a1ca396e7c9c"} Oct 30 05:01:26.628478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:26.628438 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:26.695383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:26.695005 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:26.845440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:26.759312 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:27.172899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:27.145959 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:01:27.251385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:27.233323 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 05:01:27.274805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:27.274751 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:01:27.308383 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:01:27.295704 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:01:27.346698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:27.346652 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:27.395485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:27.395441 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:27.543239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:27.541723 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": context deadline exceeded" Oct 30 05:01:27.668385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:27.638284 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:27.853322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:27.801944 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:01:27.853322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:27.844616 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:01:27.982382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:27.941639 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:01:28.138430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.116949 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:28.213311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.213260 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:28.214569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.214537 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:01:28.220528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.220489 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:28.247569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.238717 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:01:28.247569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.238757 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv"] Oct 30 05:01:28.247569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.244568 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv"] Oct 30 05:01:28.247569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.244654 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.263379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.259590 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:01:28.263379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.262773 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:01:28.290430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.265819 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-397179\"/\"default-dockercfg-lmv86\"" Oct 30 05:01:28.290430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.265898 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:01:28.302818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.302786 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-397179\"/\"distinct-2-w2-397179-kserve-self-signed-certs\"" Oct 30 05:01:28.305036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.304827 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-397179\"/\"kube-root-ca.crt\"" Oct 30 05:01:28.305188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.304916 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-397179\"/\"openshift-service-ca.crt\"" Oct 30 05:01:28.308322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.308299 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:01:28.401820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.401575 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:01:28.450467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.440228 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-model-cache\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.450467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.445521 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-dshm\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.450467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.445619 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-tmp-dir\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.450467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.445686 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3030b121-7591-4c40-b891-52a6009228c9-tls-certs\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.450467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.445725 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47zn2\" (UniqueName: \"kubernetes.io/projected/3030b121-7591-4c40-b891-52a6009228c9-kube-api-access-47zn2\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.450467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.445832 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-home\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.547857 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.547819 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-model-cache\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.548150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.548129 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-dshm\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.548306 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.548292 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-tmp-dir\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.548452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.548437 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3030b121-7591-4c40-b891-52a6009228c9-tls-certs\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.548571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.548555 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47zn2\" (UniqueName: \"kubernetes.io/projected/3030b121-7591-4c40-b891-52a6009228c9-kube-api-access-47zn2\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.548712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.548698 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-home\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.549730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.549707 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-model-cache\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.550970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.550948 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-tmp-dir\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.552696 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.551403 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-home\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.577398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.577371 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-dshm\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.577580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.577501 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47zn2\" (UniqueName: \"kubernetes.io/projected/3030b121-7591-4c40-b891-52a6009228c9-kube-api-access-47zn2\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.577705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.577614 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3030b121-7591-4c40-b891-52a6009228c9-tls-certs\") pod \"distinct-2-w2-397179-kserve-7cc95f4984-qtszv\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:28.676681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:28.676602 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:29.013719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:29.013680 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv"] Oct 30 05:01:29.084546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:29.084415 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" event={"ID":"407c363e-e3da-40c7-adb6-0d5e8880fd55","Type":"ContainerStarted","Data":"09e74d2b7299a5b9e37967a45f9742092e61eb988905b0a44966a8f4dabce27f"} Oct 30 05:01:29.086708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:29.086297 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" event={"ID":"896a9789-3a1d-4ec0-abfb-39131100ba0d","Type":"ContainerStarted","Data":"9c2072d7b1782a9bd14cd49f90cfe60b44d032e7654e22d2a265304002344def"} Oct 30 05:01:29.162448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:29.161911 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" podStartSLOduration=8.161891775 podStartE2EDuration="8.161891775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:29.143178097 +0000 UTC m=+1013.670944877" watchObservedRunningTime="2026-10-30 05:01:29.161891775 +0000 UTC m=+1013.689658561" Oct 30 05:01:29.201042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:29.200989 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" podStartSLOduration=8.200973069 podStartE2EDuration="8.200973069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:29.187065895 +0000 UTC m=+1013.714832703" watchObservedRunningTime="2026-10-30 05:01:29.200973069 +0000 UTC m=+1013.728739854" Oct 30 05:01:29.407259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:29.407218 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:29.408640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:29.408609 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 05:01:29.408837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:29.408816 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" podUID="407c363e-e3da-40c7-adb6-0d5e8880fd55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 05:01:30.074517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:30.074480 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:01:30.074802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:30.074774 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" podUID="896a9789-3a1d-4ec0-abfb-39131100ba0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 05:01:30.075166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:30.075145 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:30.096246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:30.095960 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" event={"ID":"3030b121-7591-4c40-b891-52a6009228c9","Type":"ContainerStarted","Data":"cf2e7e44c211ec48b0c2aba944ded4496eceba945be1c6907788b4eab4aa6a84"} Oct 30 05:01:30.096246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:30.096003 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" event={"ID":"3030b121-7591-4c40-b891-52a6009228c9","Type":"ContainerStarted","Data":"a66318250e3ac1f13be2cc419791987a691f6b71f25d7d6fd6e1f18bfc7857d1"} Oct 30 05:01:30.157383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:30.156111 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podStartSLOduration=3.156090275 podStartE2EDuration="3.156090275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:30.136123584 +0000 UTC m=+1014.663890374" watchObservedRunningTime="2026-10-30 05:01:30.156090275 +0000 UTC m=+1014.683857064" Oct 30 05:01:30.409352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:30.409253 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 05:01:30.409352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:30.409326 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" podUID="407c363e-e3da-40c7-adb6-0d5e8880fd55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 05:01:30.682458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:30.679661 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:31.072384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:31.068880 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:01:31.072384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:31.068953 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" podUID="896a9789-3a1d-4ec0-abfb-39131100ba0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 05:01:31.408052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:31.407972 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 05:01:31.408186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:31.408045 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" podUID="407c363e-e3da-40c7-adb6-0d5e8880fd55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 05:01:32.077386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:32.076592 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:01:32.077386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:32.076666 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" podUID="896a9789-3a1d-4ec0-abfb-39131100ba0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 05:01:32.406848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:32.406759 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 05:01:32.406848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:32.406817 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" podUID="407c363e-e3da-40c7-adb6-0d5e8880fd55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 05:01:33.072381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:33.071761 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:01:33.072381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:33.071841 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" podUID="896a9789-3a1d-4ec0-abfb-39131100ba0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 05:01:33.406684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:33.406609 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 05:01:33.406684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:33.406655 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" podUID="407c363e-e3da-40c7-adb6-0d5e8880fd55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 05:01:34.068179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.068132 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:01:34.068410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.068197 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" podUID="896a9789-3a1d-4ec0-abfb-39131100ba0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 05:01:34.360541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.360425 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm"] Oct 30 05:01:34.371460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.371432 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" Oct 30 05:01:34.397653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.394886 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm"] Oct 30 05:01:34.402859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.402830 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:01:34.403025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.402876 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:01:34.403025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.402953 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:01:34.403025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.403000 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:01:34.406702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.406679 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 05:01:34.407042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.406719 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" podUID="407c363e-e3da-40c7-adb6-0d5e8880fd55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 05:01:34.436712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.436668 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa08835e-4sxqm\" (UID: \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" Oct 30 05:01:34.437040 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.437016 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa08835e-4sxqm\" (UID: \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" Oct 30 05:01:34.446393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.437258 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb9d7\" (UniqueName: \"kubernetes.io/projected/3dc6a393-2c3c-41be-b898-bcc9b87b5390-kube-api-access-zb9d7\") pod \"maas-api-revoke-sub-fa08835e-4sxqm\" (UID: \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" Oct 30 05:01:34.569650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.569614 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa08835e-4sxqm\" (UID: \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" Oct 30 05:01:34.569650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.569665 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa08835e-4sxqm\" (UID: \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" Oct 30 05:01:34.569924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.569720 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb9d7\" (UniqueName: \"kubernetes.io/projected/3dc6a393-2c3c-41be-b898-bcc9b87b5390-kube-api-access-zb9d7\") pod \"maas-api-revoke-sub-fa08835e-4sxqm\" (UID: \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" Oct 30 05:01:34.570480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.570447 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa08835e-4sxqm\" (UID: \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" Oct 30 05:01:34.572265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.572238 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa08835e-4sxqm\" (UID: \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" Oct 30 05:01:34.591884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.591852 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb9d7\" (UniqueName: \"kubernetes.io/projected/3dc6a393-2c3c-41be-b898-bcc9b87b5390-kube-api-access-zb9d7\") pod \"maas-api-revoke-sub-fa08835e-4sxqm\" (UID: \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" Oct 30 05:01:34.706450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:34.706338 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" Oct 30 05:01:35.030409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:35.030253 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm"] Oct 30 05:01:35.067629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:35.067597 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:01:35.067745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:35.067654 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" podUID="896a9789-3a1d-4ec0-abfb-39131100ba0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 05:01:35.184180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:35.184132 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" event={"ID":"3dc6a393-2c3c-41be-b898-bcc9b87b5390","Type":"ContainerStarted","Data":"2b6aac91fae890652d9a1951b0a775d2c274929d7c846ed6a04728dfaf77b7f5"} Oct 30 05:01:35.893149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:35.892726 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:35.895247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:35.894443 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:35.895710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:35.895687 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:01:36.164434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:36.158794 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:36.164434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:36.160278 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:36.164434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:36.162704 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:01:36.196698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:36.196668 6587 generic.go:356] "Generic (PLEG): container finished" podID="3dc6a393-2c3c-41be-b898-bcc9b87b5390" containerID="dc5c2960751ff27c51e46521f8a3855b32e2af2cbaa6b56e9bf0ab631adb0b64" exitCode=0 Oct 30 05:01:36.196868 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:36.196724 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" event={"ID":"3dc6a393-2c3c-41be-b898-bcc9b87b5390","Type":"ContainerDied","Data":"dc5c2960751ff27c51e46521f8a3855b32e2af2cbaa6b56e9bf0ab631adb0b64"} Oct 30 05:01:37.387748 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.387723 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" Oct 30 05:01:37.531762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.531680 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc6a393-2c3c-41be-b898-bcc9b87b5390-kube-api-access-zb9d7\" (UniqueName: \"kubernetes.io/projected/3dc6a393-2c3c-41be-b898-bcc9b87b5390-kube-api-access-zb9d7\") pod \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\" (UID: \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\") " Oct 30 05:01:37.531762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.531731 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-openshift-ca\") pod \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\" (UID: \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\") " Oct 30 05:01:37.531994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.531844 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-xks-ca\") pod \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\" (UID: \"3dc6a393-2c3c-41be-b898-bcc9b87b5390\") " Oct 30 05:01:37.532196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.532166 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-openshift-ca" pod "3dc6a393-2c3c-41be-b898-bcc9b87b5390" (UID: "3dc6a393-2c3c-41be-b898-bcc9b87b5390"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:01:37.532477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.532457 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:37.533895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.533852 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-xks-ca" pod "3dc6a393-2c3c-41be-b898-bcc9b87b5390" (UID: "3dc6a393-2c3c-41be-b898-bcc9b87b5390"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:01:37.533996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.533909 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc6a393-2c3c-41be-b898-bcc9b87b5390-kube-api-access-zb9d7" pod "3dc6a393-2c3c-41be-b898-bcc9b87b5390" (UID: "3dc6a393-2c3c-41be-b898-bcc9b87b5390"). InnerVolumeSpecName "kube-api-access-zb9d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:01:37.633688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.633656 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb9d7\" (UniqueName: \"kubernetes.io/projected/3dc6a393-2c3c-41be-b898-bcc9b87b5390-kube-api-access-zb9d7\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:37.633688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.633688 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc6a393-2c3c-41be-b898-bcc9b87b5390-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:37.750647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.750616 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj"] Oct 30 05:01:37.751278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.751265 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3dc6a393-2c3c-41be-b898-bcc9b87b5390" containerName="revoke-keys" Oct 30 05:01:37.751333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.751279 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc6a393-2c3c-41be-b898-bcc9b87b5390" containerName="revoke-keys" Oct 30 05:01:37.751444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.751434 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc6a393-2c3c-41be-b898-bcc9b87b5390" containerName="revoke-keys" Oct 30 05:01:37.755274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.755255 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" Oct 30 05:01:37.775251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.775223 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj"] Oct 30 05:01:37.837545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.837506 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45446341-jtmpj\" (UID: \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" Oct 30 05:01:37.837738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.837685 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qclzw\" (UniqueName: \"kubernetes.io/projected/5d81aeb2-a9f4-4061-ad64-4fc317582efb-kube-api-access-qclzw\") pod \"maas-api-revoke-sub-45446341-jtmpj\" (UID: \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" Oct 30 05:01:37.837738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.837727 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45446341-jtmpj\" (UID: \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" Oct 30 05:01:37.938824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.938784 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45446341-jtmpj\" (UID: \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" Oct 30 05:01:37.939002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.938930 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qclzw\" (UniqueName: \"kubernetes.io/projected/5d81aeb2-a9f4-4061-ad64-4fc317582efb-kube-api-access-qclzw\") pod \"maas-api-revoke-sub-45446341-jtmpj\" (UID: \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" Oct 30 05:01:37.939002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.938954 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45446341-jtmpj\" (UID: \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" Oct 30 05:01:37.939478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.939452 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45446341-jtmpj\" (UID: \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" Oct 30 05:01:37.941352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.941331 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45446341-jtmpj\" (UID: \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" Oct 30 05:01:37.959389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:37.959331 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qclzw\" (UniqueName: \"kubernetes.io/projected/5d81aeb2-a9f4-4061-ad64-4fc317582efb-kube-api-access-qclzw\") pod \"maas-api-revoke-sub-45446341-jtmpj\" (UID: \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" Oct 30 05:01:38.069706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:38.069673 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" Oct 30 05:01:38.223560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:38.223521 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" event={"ID":"3dc6a393-2c3c-41be-b898-bcc9b87b5390","Type":"ContainerDied","Data":"2b6aac91fae890652d9a1951b0a775d2c274929d7c846ed6a04728dfaf77b7f5"} Oct 30 05:01:38.223726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:38.223607 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b6aac91fae890652d9a1951b0a775d2c274929d7c846ed6a04728dfaf77b7f5" Oct 30 05:01:38.223726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:38.223536 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm" Oct 30 05:01:38.374352 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:01:38.374315 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d81aeb2_a9f4_4061_ad64_4fc317582efb.slice/crio-2e9df31a675c4e0c4cb6a698254c673c7c9ce22a80afa51eb3693c708368666c WatchSource:0}: Error finding container 2e9df31a675c4e0c4cb6a698254c673c7c9ce22a80afa51eb3693c708368666c: Status 404 returned error can't find the container with id 2e9df31a675c4e0c4cb6a698254c673c7c9ce22a80afa51eb3693c708368666c Oct 30 05:01:38.376546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:38.376476 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj"] Oct 30 05:01:38.677561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:38.677524 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:38.691032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:38.691005 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:38.925798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:38.925721 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb"] Oct 30 05:01:38.931706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:38.931684 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" Oct 30 05:01:38.947069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:38.947035 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb"] Oct 30 05:01:38.955319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:38.955282 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5psl\" (UniqueName: \"kubernetes.io/projected/231fbddd-367a-493a-ab21-983bfb881d2b-kube-api-access-x5psl\") pod \"maas-api-revoke-sub-a3e424d0-l68bb\" (UID: \"231fbddd-367a-493a-ab21-983bfb881d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" Oct 30 05:01:38.955532 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:38.955481 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e424d0-l68bb\" (UID: \"231fbddd-367a-493a-ab21-983bfb881d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" Oct 30 05:01:38.955755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:38.955735 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e424d0-l68bb\" (UID: \"231fbddd-367a-493a-ab21-983bfb881d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" Oct 30 05:01:39.057841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:39.057801 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e424d0-l68bb\" (UID: \"231fbddd-367a-493a-ab21-983bfb881d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" Oct 30 05:01:39.058025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:39.057895 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5psl\" (UniqueName: \"kubernetes.io/projected/231fbddd-367a-493a-ab21-983bfb881d2b-kube-api-access-x5psl\") pod \"maas-api-revoke-sub-a3e424d0-l68bb\" (UID: \"231fbddd-367a-493a-ab21-983bfb881d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" Oct 30 05:01:39.058025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:39.057989 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e424d0-l68bb\" (UID: \"231fbddd-367a-493a-ab21-983bfb881d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" Oct 30 05:01:39.058689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:39.058667 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e424d0-l68bb\" (UID: \"231fbddd-367a-493a-ab21-983bfb881d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" Oct 30 05:01:39.060366 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:39.060333 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e424d0-l68bb\" (UID: \"231fbddd-367a-493a-ab21-983bfb881d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" Oct 30 05:01:39.080411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:39.080382 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5psl\" (UniqueName: \"kubernetes.io/projected/231fbddd-367a-493a-ab21-983bfb881d2b-kube-api-access-x5psl\") pod \"maas-api-revoke-sub-a3e424d0-l68bb\" (UID: \"231fbddd-367a-493a-ab21-983bfb881d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" Oct 30 05:01:39.233630 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:39.233523 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" event={"ID":"5d81aeb2-a9f4-4061-ad64-4fc317582efb","Type":"ContainerStarted","Data":"7b286febe342bed57e749c7f1413e2e32f854ee9879c4295f835f04b727d61f7"} Oct 30 05:01:39.233630 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:39.233594 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" event={"ID":"5d81aeb2-a9f4-4061-ad64-4fc317582efb","Type":"ContainerStarted","Data":"2e9df31a675c4e0c4cb6a698254c673c7c9ce22a80afa51eb3693c708368666c"} Oct 30 05:01:39.245198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:39.245170 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:39.256781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:39.256760 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" Oct 30 05:01:39.278266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:39.278216 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" podStartSLOduration=2.278201878 podStartE2EDuration="2.278201878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:39.274720378 +0000 UTC m=+1023.802487164" watchObservedRunningTime="2026-10-30 05:01:39.278201878 +0000 UTC m=+1023.805968663" Oct 30 05:01:39.557526 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:01:39.557491 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231fbddd_367a_493a_ab21_983bfb881d2b.slice/crio-915f82cdbadd0f2281e11911b561caae489c8b9606df4d26124dadeb2988a427 WatchSource:0}: Error finding container 915f82cdbadd0f2281e11911b561caae489c8b9606df4d26124dadeb2988a427: Status 404 returned error can't find the container with id 915f82cdbadd0f2281e11911b561caae489c8b9606df4d26124dadeb2988a427 Oct 30 05:01:39.561011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:39.560991 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb"] Oct 30 05:01:40.268284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:40.268181 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" event={"ID":"231fbddd-367a-493a-ab21-983bfb881d2b","Type":"ContainerStarted","Data":"751d199507a4610ae6be9128bbaf3bde38d7360ad47f8239f3c713bb92238f53"} Oct 30 05:01:40.268284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:40.268253 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" event={"ID":"231fbddd-367a-493a-ab21-983bfb881d2b","Type":"ContainerStarted","Data":"915f82cdbadd0f2281e11911b561caae489c8b9606df4d26124dadeb2988a427"} Oct 30 05:01:40.273840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:40.273807 6587 generic.go:356] "Generic (PLEG): container finished" podID="5d81aeb2-a9f4-4061-ad64-4fc317582efb" containerID="7b286febe342bed57e749c7f1413e2e32f854ee9879c4295f835f04b727d61f7" exitCode=0 Oct 30 05:01:40.274966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:40.274943 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" event={"ID":"5d81aeb2-a9f4-4061-ad64-4fc317582efb","Type":"ContainerDied","Data":"7b286febe342bed57e749c7f1413e2e32f854ee9879c4295f835f04b727d61f7"} Oct 30 05:01:40.317181 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:40.317132 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" podStartSLOduration=2.3171172 podStartE2EDuration="2.3171172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:40.310893233 +0000 UTC m=+1024.838660021" watchObservedRunningTime="2026-10-30 05:01:40.3171172 +0000 UTC m=+1024.844883986" Oct 30 05:01:41.283961 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:41.283920 6587 generic.go:356] "Generic (PLEG): container finished" podID="231fbddd-367a-493a-ab21-983bfb881d2b" containerID="751d199507a4610ae6be9128bbaf3bde38d7360ad47f8239f3c713bb92238f53" exitCode=0 Oct 30 05:01:41.284397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:41.284008 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" event={"ID":"231fbddd-367a-493a-ab21-983bfb881d2b","Type":"ContainerDied","Data":"751d199507a4610ae6be9128bbaf3bde38d7360ad47f8239f3c713bb92238f53"} Oct 30 05:01:41.467545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:41.467520 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" Oct 30 05:01:41.501855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:41.501815 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-openshift-ca\") pod \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\" (UID: \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\") " Oct 30 05:01:41.502039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:41.501941 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-xks-ca\") pod \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\" (UID: \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\") " Oct 30 05:01:41.502039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:41.502003 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d81aeb2-a9f4-4061-ad64-4fc317582efb-kube-api-access-qclzw\" (UniqueName: \"kubernetes.io/projected/5d81aeb2-a9f4-4061-ad64-4fc317582efb-kube-api-access-qclzw\") pod \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\" (UID: \"5d81aeb2-a9f4-4061-ad64-4fc317582efb\") " Oct 30 05:01:41.502284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:41.502256 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-openshift-ca" pod "5d81aeb2-a9f4-4061-ad64-4fc317582efb" (UID: "5d81aeb2-a9f4-4061-ad64-4fc317582efb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:01:41.502764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:41.502743 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:41.504230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:41.504206 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d81aeb2-a9f4-4061-ad64-4fc317582efb-kube-api-access-qclzw" pod "5d81aeb2-a9f4-4061-ad64-4fc317582efb" (UID: "5d81aeb2-a9f4-4061-ad64-4fc317582efb"). InnerVolumeSpecName "kube-api-access-qclzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:01:41.504549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:41.504524 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-xks-ca" pod "5d81aeb2-a9f4-4061-ad64-4fc317582efb" (UID: "5d81aeb2-a9f4-4061-ad64-4fc317582efb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:01:41.604871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:41.604839 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d81aeb2-a9f4-4061-ad64-4fc317582efb-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:41.604871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:41.604872 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qclzw\" (UniqueName: \"kubernetes.io/projected/5d81aeb2-a9f4-4061-ad64-4fc317582efb-kube-api-access-qclzw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:42.292829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:42.292801 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" Oct 30 05:01:42.293222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:42.292804 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj" event={"ID":"5d81aeb2-a9f4-4061-ad64-4fc317582efb","Type":"ContainerDied","Data":"2e9df31a675c4e0c4cb6a698254c673c7c9ce22a80afa51eb3693c708368666c"} Oct 30 05:01:42.293222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:42.292927 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e9df31a675c4e0c4cb6a698254c673c7c9ce22a80afa51eb3693c708368666c" Oct 30 05:01:42.472511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:42.472488 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" Oct 30 05:01:42.517293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:42.517252 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231fbddd-367a-493a-ab21-983bfb881d2b-kube-api-access-x5psl\" (UniqueName: \"kubernetes.io/projected/231fbddd-367a-493a-ab21-983bfb881d2b-kube-api-access-x5psl\") pod \"231fbddd-367a-493a-ab21-983bfb881d2b\" (UID: \"231fbddd-367a-493a-ab21-983bfb881d2b\") " Oct 30 05:01:42.517495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:42.517335 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-openshift-ca\") pod \"231fbddd-367a-493a-ab21-983bfb881d2b\" (UID: \"231fbddd-367a-493a-ab21-983bfb881d2b\") " Oct 30 05:01:42.517591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:42.517515 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-xks-ca\") pod \"231fbddd-367a-493a-ab21-983bfb881d2b\" (UID: \"231fbddd-367a-493a-ab21-983bfb881d2b\") " Oct 30 05:01:42.518517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:42.518298 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-openshift-ca" pod "231fbddd-367a-493a-ab21-983bfb881d2b" (UID: "231fbddd-367a-493a-ab21-983bfb881d2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:01:42.518517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:42.518486 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:42.540403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:42.524585 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-xks-ca" pod "231fbddd-367a-493a-ab21-983bfb881d2b" (UID: "231fbddd-367a-493a-ab21-983bfb881d2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:01:42.540403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:42.529059 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231fbddd-367a-493a-ab21-983bfb881d2b-kube-api-access-x5psl" pod "231fbddd-367a-493a-ab21-983bfb881d2b" (UID: "231fbddd-367a-493a-ab21-983bfb881d2b"). InnerVolumeSpecName "kube-api-access-x5psl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:01:42.619937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:42.619886 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231fbddd-367a-493a-ab21-983bfb881d2b-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:42.619937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:42.619922 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5psl\" (UniqueName: \"kubernetes.io/projected/231fbddd-367a-493a-ab21-983bfb881d2b-kube-api-access-x5psl\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:01:43.302244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:43.302211 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" Oct 30 05:01:43.302640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:43.302236 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb" event={"ID":"231fbddd-367a-493a-ab21-983bfb881d2b","Type":"ContainerDied","Data":"915f82cdbadd0f2281e11911b561caae489c8b9606df4d26124dadeb2988a427"} Oct 30 05:01:43.302640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:43.302295 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="915f82cdbadd0f2281e11911b561caae489c8b9606df4d26124dadeb2988a427" Oct 30 05:01:48.828242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.821691 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl"] Oct 30 05:01:48.828242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.822931 6587 cpu_manager.go:414] "containerMap: removing container" podUID="231fbddd-367a-493a-ab21-983bfb881d2b" containerName="revoke-keys" Oct 30 05:01:48.828242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.822957 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231fbddd-367a-493a-ab21-983bfb881d2b" containerName="revoke-keys" Oct 30 05:01:48.828242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.822978 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5d81aeb2-a9f4-4061-ad64-4fc317582efb" containerName="revoke-keys" Oct 30 05:01:48.828242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.822987 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d81aeb2-a9f4-4061-ad64-4fc317582efb" containerName="revoke-keys" Oct 30 05:01:48.828242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.823260 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="231fbddd-367a-493a-ab21-983bfb881d2b" containerName="revoke-keys" Oct 30 05:01:48.828242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.823277 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d81aeb2-a9f4-4061-ad64-4fc317582efb" containerName="revoke-keys" Oct 30 05:01:48.829315 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.829274 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:48.861416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.858493 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl"] Oct 30 05:01:48.861416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.859717 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 05:01:48.861416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.859794 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 05:01:48.861416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.859837 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8lhdl\"" Oct 30 05:01:48.861416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.859845 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 05:01:48.928838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.928791 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbc1851-59c4-451d-9345-aff6d78e5a2d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:48.929023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.928907 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:48.929023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.928939 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:48.929023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.928998 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgchb\" (UniqueName: \"kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-kube-api-access-wgchb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:48.929216 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.929094 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:48.929216 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.929121 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:48.929329 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:48.929214 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.031255 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.031219 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgchb\" (UniqueName: \"kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-kube-api-access-wgchb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.031470 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.031311 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.031470 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.031331 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.031470 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.031416 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.031654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.031630 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbc1851-59c4-451d-9345-aff6d78e5a2d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.031729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.031668 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.031729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.031699 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.032682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.032647 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.033933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.033890 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.034040 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.033921 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.034237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.034215 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbc1851-59c4-451d-9345-aff6d78e5a2d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.034277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.034227 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.066558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.054557 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgchb\" (UniqueName: \"kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-kube-api-access-wgchb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.066558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.054767 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nxlpl\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.170136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.170062 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:01:49.568347 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:01:49.568299 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbc1851_59c4_451d_9345_aff6d78e5a2d.slice/crio-9aaf80eb68bc7c82801bf1eedcb000643f33e99dfe4088d97625bd52f0901e91 WatchSource:0}: Error finding container 9aaf80eb68bc7c82801bf1eedcb000643f33e99dfe4088d97625bd52f0901e91: Status 404 returned error can't find the container with id 9aaf80eb68bc7c82801bf1eedcb000643f33e99dfe4088d97625bd52f0901e91 Oct 30 05:01:49.569996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.569964 6587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:01:49.637529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:49.632174 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl"] Oct 30 05:01:50.508382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:50.508289 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" event={"ID":"4bbc1851-59c4-451d-9345-aff6d78e5a2d","Type":"ContainerStarted","Data":"9aaf80eb68bc7c82801bf1eedcb000643f33e99dfe4088d97625bd52f0901e91"} Oct 30 05:01:51.054384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:51.053440 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:51.129383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:51.129248 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:51.242383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:51.241431 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:51.242593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:51.242475 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:51.411439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:51.402456 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:51.442383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:51.441464 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:51.643384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:51.621128 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:51.643384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:51.630280 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:51.716440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:51.706479 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:51.823583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:51.823540 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:52.179807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.141241 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:52.245385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.210041 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:01:52.396245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.291436 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:52.396245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.291880 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:52.396245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.298438 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:52.396245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.311532 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:01:52.457065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.456947 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:52.579382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.472486 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:01:52.579382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.474161 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:01:52.579382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.500518 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:01:52.650838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.650795 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:52.680421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.658243 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:01:52.804888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.788880 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk"] Oct 30 05:01:52.804888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.794386 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:52.804888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.797834 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:52.804888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.798475 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:52.881380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.842699 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:52.881380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.855189 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:01:52.915779 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.906823 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:53.019384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:52.994646 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:53.056439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.048553 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:53.056439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.048889 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:53.056439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.049008 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:53.056439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.049374 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:53.056439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.049694 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:53.056439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.049960 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:53.323771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.323675 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:53.364945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.364898 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ttfg\" (UniqueName: \"kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-kube-api-access-8ttfg\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:53.475386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.237576 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:01:53.634452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.459756 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:01:53.676384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.662862 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:53.693381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.631555 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:01:53.704388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.586817 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:53.942162 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.942031 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" podUID="cd43eb0e-0dca-47cd-9fda-bb4a0dbce266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Oct 30 05:01:53.983379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.966701 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:53.983379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:53.967028 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:01:54.267955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.257861 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk"] Oct 30 05:01:54.269097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.269058 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-cnqvb" podUID="cd43eb0e-0dca-47cd-9fda-bb4a0dbce266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Oct 30 05:01:54.367402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.367321 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:01:54.367614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.367419 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:54.470382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.467093 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" podUID="ac919fe3-ac98-44cd-8d95-6df70a406fa4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.18:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:54.470382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.468222 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:01:54.500382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.481708 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:01:54.501328 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.483277 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:54.538136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.538032 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.538413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.538391 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.538653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.538634 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.538871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.538853 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.539007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.538992 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.539201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.539167 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.539445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.539417 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ttfg\" (UniqueName: \"kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-kube-api-access-8ttfg\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.539760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.539742 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.539955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.539937 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.540812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.540792 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.616754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.616721 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.617245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.617219 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.637775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.637743 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.640725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.640695 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:54.646779 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.646745 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:01:54.708098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.708056 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:54.753604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.753514 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:54.778410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.774876 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:01:54.778746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.778690 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:54.964508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.964458 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:01:54.990678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.990602 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:55.016452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:54.990540 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:55.040393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:55.034836 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ttfg\" (UniqueName: \"kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-kube-api-access-8ttfg\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:55.066801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:55.066728 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:55.156418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:55.151597 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:55.216940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:55.216765 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:55.292384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:55.288639 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:55.483098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:55.482995 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:55.518571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:55.518528 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:01:55.519165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:55.519133 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:01:55.666058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:55.666012 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:55.711249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:55.710640 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:01:55.763539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:55.763438 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:55.996533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:55.996483 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:01:56.003382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:55.999621 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:01:56.030694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:56.030600 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:56.458388 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:01:56.447636 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:01:56.610503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:56.593764 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:01:56.611086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:56.593804 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:01:56.617382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:56.595178 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:01:56.687276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:56.687230 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 30 05:01:56.712906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:56.712462 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:01:56.800847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:56.727431 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:56.800847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:56.778043 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:57.007018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:57.001050 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:57.060990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:57.060584 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:57.208213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:57.164342 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:57.266433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:57.243077 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:57.617946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:57.617900 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": context deadline exceeded" Oct 30 05:01:57.696968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:57.633758 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:01:57.699133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:57.640160 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:01:57.700209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:57.640268 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:01:57.722769 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:57.719209 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:01:57.917160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:57.881754 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:01:57.917160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:57.883685 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:01:57.917160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:57.884950 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:58.035952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.031460 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:58.172422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.172325 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:01:58.179381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.178534 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:01:58.181231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.172756 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:01:58.184287 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.184256 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:58.184434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.184349 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:58.184434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.184411 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:01:58.184624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.184601 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:01:58.184708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.184655 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:01:58.185710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.184984 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:01:58.187881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.187793 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:01:58.188165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.188153 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:01:58.198476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.198452 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2"] Oct 30 05:01:58.232895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.232868 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2"] Oct 30 05:01:58.233118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.233104 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk"] Oct 30 05:01:58.233295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.233274 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:01:58.233526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.233025 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" Oct 30 05:01:58.248765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.248729 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" event={"ID":"eb42bb9d-5a6f-4124-a5c1-b200eee2901d","Type":"ContainerStarted","Data":"45c16902a6c30c6a0da9f48af8ecf7f085cba85b1c13c768711e4b4a6b743dab"} Oct 30 05:01:58.269420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.268842 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:01:58.269420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.268983 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:01:58.269420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.269054 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:01:58.269420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.269135 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:01:58.340990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.340957 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcd1a95c-tsvg2\" (UID: \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" Oct 30 05:01:58.342934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.342907 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzzsk\" (UniqueName: \"kubernetes.io/projected/ecfea321-d7af-4ee0-83f2-1aee2c643b01-kube-api-access-fzzsk\") pod \"maas-api-revoke-sub-bcd1a95c-tsvg2\" (UID: \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" Oct 30 05:01:58.343551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.343518 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcd1a95c-tsvg2\" (UID: \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" Oct 30 05:01:58.450704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.450614 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcd1a95c-tsvg2\" (UID: \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" Oct 30 05:01:58.450854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.450718 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzzsk\" (UniqueName: \"kubernetes.io/projected/ecfea321-d7af-4ee0-83f2-1aee2c643b01-kube-api-access-fzzsk\") pod \"maas-api-revoke-sub-bcd1a95c-tsvg2\" (UID: \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" Oct 30 05:01:58.450939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.450917 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcd1a95c-tsvg2\" (UID: \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" Oct 30 05:01:58.454119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.454082 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcd1a95c-tsvg2\" (UID: \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" Oct 30 05:01:58.482930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.482867 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzzsk\" (UniqueName: \"kubernetes.io/projected/ecfea321-d7af-4ee0-83f2-1aee2c643b01-kube-api-access-fzzsk\") pod \"maas-api-revoke-sub-bcd1a95c-tsvg2\" (UID: \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" Oct 30 05:01:58.496417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.493260 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcd1a95c-tsvg2\" (UID: \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" Oct 30 05:01:58.567129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.566879 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" Oct 30 05:01:58.979554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:58.979527 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2"] Oct 30 05:01:59.262684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:59.261808 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" event={"ID":"eb42bb9d-5a6f-4124-a5c1-b200eee2901d","Type":"ContainerStarted","Data":"78dd06b19f73cd4d87261952ef85d88d14a79f6898eb8fa9ab8af52849b341b3"} Oct 30 05:01:59.263589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:59.263551 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" event={"ID":"ecfea321-d7af-4ee0-83f2-1aee2c643b01","Type":"ContainerStarted","Data":"44524dcba1765e550a28e2c6df7bf9e40c426e61129dc43929862e086ad747af"} Oct 30 05:01:59.290407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:59.289688 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:01:59.292867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:59.291538 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:01:59.292867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:01:59.291593 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" podUID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 05:02:00.275137 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:00.275106 6587 generic.go:356] "Generic (PLEG): container finished" podID="ecfea321-d7af-4ee0-83f2-1aee2c643b01" containerID="1ed6e93e922b6024a398ecf1a307fc0a55c7fdadedf545db3c54d51acd1763fa" exitCode=0 Oct 30 05:02:00.275791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:00.275763 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" event={"ID":"ecfea321-d7af-4ee0-83f2-1aee2c643b01","Type":"ContainerDied","Data":"1ed6e93e922b6024a398ecf1a307fc0a55c7fdadedf545db3c54d51acd1763fa"} Oct 30 05:02:00.290097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:00.290070 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:02:00.290235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:00.290120 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" podUID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 05:02:00.348895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:00.348822 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" podStartSLOduration=8.348800572 podStartE2EDuration="8.348800572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:59.310875791 +0000 UTC m=+1043.838642578" watchObservedRunningTime="2026-10-30 05:02:00.348800572 +0000 UTC m=+1044.876567359" Oct 30 05:02:01.290752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.289605 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:02:01.290752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.289660 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" podUID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 05:02:01.595624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.595577 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" Oct 30 05:02:01.613027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.612996 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfea321-d7af-4ee0-83f2-1aee2c643b01-kube-api-access-fzzsk\" (UniqueName: \"kubernetes.io/projected/ecfea321-d7af-4ee0-83f2-1aee2c643b01-kube-api-access-fzzsk\") pod \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\" (UID: \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\") " Oct 30 05:02:01.613211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.613191 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-xks-ca\") pod \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\" (UID: \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\") " Oct 30 05:02:01.613336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.613318 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-openshift-ca\") pod \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\" (UID: \"ecfea321-d7af-4ee0-83f2-1aee2c643b01\") " Oct 30 05:02:01.613719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.613694 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-openshift-ca" pod "ecfea321-d7af-4ee0-83f2-1aee2c643b01" (UID: "ecfea321-d7af-4ee0-83f2-1aee2c643b01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:01.614712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.614687 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:01.662155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.647995 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfea321-d7af-4ee0-83f2-1aee2c643b01-kube-api-access-fzzsk" pod "ecfea321-d7af-4ee0-83f2-1aee2c643b01" (UID: "ecfea321-d7af-4ee0-83f2-1aee2c643b01"). InnerVolumeSpecName "kube-api-access-fzzsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:01.662155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.649477 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-xks-ca" pod "ecfea321-d7af-4ee0-83f2-1aee2c643b01" (UID: "ecfea321-d7af-4ee0-83f2-1aee2c643b01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:01.751384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.742180 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecfea321-d7af-4ee0-83f2-1aee2c643b01-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:01.751384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.742219 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzzsk\" (UniqueName: \"kubernetes.io/projected/ecfea321-d7af-4ee0-83f2-1aee2c643b01-kube-api-access-fzzsk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:01.790780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.790752 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:02:01.790852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:01.790818 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:02:02.290352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:02.290226 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:02:02.290352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:02.290292 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" podUID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 05:02:02.300380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:02.299252 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" event={"ID":"ecfea321-d7af-4ee0-83f2-1aee2c643b01","Type":"ContainerDied","Data":"44524dcba1765e550a28e2c6df7bf9e40c426e61129dc43929862e086ad747af"} Oct 30 05:02:02.300380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:02.299314 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44524dcba1765e550a28e2c6df7bf9e40c426e61129dc43929862e086ad747af" Oct 30 05:02:02.300380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:02.299405 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2" Oct 30 05:02:02.304404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:02.302975 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" event={"ID":"4bbc1851-59c4-451d-9345-aff6d78e5a2d","Type":"ContainerStarted","Data":"b6a9ec40fe70f95a79109d61ddac1df214d7ddc2e0f999c4f6213d8d6f3210df"} Oct 30 05:02:02.304678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:02.304655 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:02:02.306276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:02.306252 6587 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:02:02.306422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:02.306295 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" podUID="4bbc1851-59c4-451d-9345-aff6d78e5a2d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:02:02.368371 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:02.368294 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" podStartSLOduration=2.149392702 podStartE2EDuration="14.368279283s" totalImagesPullingTime="12.218886581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:02.351863186 +0000 UTC m=+1046.879629972" watchObservedRunningTime="2026-10-30 05:02:02.368279283 +0000 UTC m=+1046.896046068" Oct 30 05:02:03.290172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.290131 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:02:03.290348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.290210 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" podUID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 05:02:03.351293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.350629 6587 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:02:03.351293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.350704 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" podUID="4bbc1851-59c4-451d-9345-aff6d78e5a2d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:02:03.746967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.746924 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm"] Oct 30 05:02:03.748002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.747977 6587 cpu_manager.go:414] "containerMap: removing container" podUID="ecfea321-d7af-4ee0-83f2-1aee2c643b01" containerName="revoke-keys" Oct 30 05:02:03.748002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.748004 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfea321-d7af-4ee0-83f2-1aee2c643b01" containerName="revoke-keys" Oct 30 05:02:03.748162 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.748151 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfea321-d7af-4ee0-83f2-1aee2c643b01" containerName="revoke-keys" Oct 30 05:02:03.756044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.756022 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm"] Oct 30 05:02:03.756151 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.756108 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" Oct 30 05:02:03.772273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.772250 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:02:03.785994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.785976 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:02:03.786089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.786057 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:02:03.786126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.786110 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:02:03.894113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.894075 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ad250be-vbvfm\" (UID: \"c092142b-a383-485a-9c1a-cfd750e2f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" Oct 30 05:02:03.894276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.894252 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ad250be-vbvfm\" (UID: \"c092142b-a383-485a-9c1a-cfd750e2f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" Oct 30 05:02:03.894451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.894436 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzt7n\" (UniqueName: \"kubernetes.io/projected/c092142b-a383-485a-9c1a-cfd750e2f9e8-kube-api-access-gzt7n\") pod \"maas-api-revoke-sub-5ad250be-vbvfm\" (UID: \"c092142b-a383-485a-9c1a-cfd750e2f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" Oct 30 05:02:03.996225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.996189 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzt7n\" (UniqueName: \"kubernetes.io/projected/c092142b-a383-485a-9c1a-cfd750e2f9e8-kube-api-access-gzt7n\") pod \"maas-api-revoke-sub-5ad250be-vbvfm\" (UID: \"c092142b-a383-485a-9c1a-cfd750e2f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" Oct 30 05:02:03.996443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.996296 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ad250be-vbvfm\" (UID: \"c092142b-a383-485a-9c1a-cfd750e2f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" Oct 30 05:02:03.996443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.996419 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ad250be-vbvfm\" (UID: \"c092142b-a383-485a-9c1a-cfd750e2f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" Oct 30 05:02:03.997149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.997086 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ad250be-vbvfm\" (UID: \"c092142b-a383-485a-9c1a-cfd750e2f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" Oct 30 05:02:03.998882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:03.998855 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ad250be-vbvfm\" (UID: \"c092142b-a383-485a-9c1a-cfd750e2f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" Oct 30 05:02:04.026388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:04.024954 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzt7n\" (UniqueName: \"kubernetes.io/projected/c092142b-a383-485a-9c1a-cfd750e2f9e8-kube-api-access-gzt7n\") pod \"maas-api-revoke-sub-5ad250be-vbvfm\" (UID: \"c092142b-a383-485a-9c1a-cfd750e2f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" Oct 30 05:02:04.071389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:04.071345 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" Oct 30 05:02:04.290034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:04.289951 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:02:04.290034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:04.290000 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" podUID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 05:02:04.351424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:04.351397 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:02:04.429414 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:02:04.415426 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc092142b_a383_485a_9c1a_cfd750e2f9e8.slice/crio-eb19f7ab9c390347cc0e8dd09addbb82c6a956e90dea420747a34c609acc9bee WatchSource:0}: Error finding container eb19f7ab9c390347cc0e8dd09addbb82c6a956e90dea420747a34c609acc9bee: Status 404 returned error can't find the container with id eb19f7ab9c390347cc0e8dd09addbb82c6a956e90dea420747a34c609acc9bee Oct 30 05:02:04.469043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:04.469004 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm"] Oct 30 05:02:05.290268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:05.290223 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:02:05.290505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:05.290294 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" podUID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 05:02:05.361317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:05.361259 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" event={"ID":"c092142b-a383-485a-9c1a-cfd750e2f9e8","Type":"ContainerStarted","Data":"fb993ccdcdc854383879d76b8f9221ca69956a890e9709cb7bcb5489aa2bd87d"} Oct 30 05:02:05.361847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:05.361343 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" event={"ID":"c092142b-a383-485a-9c1a-cfd750e2f9e8","Type":"ContainerStarted","Data":"eb19f7ab9c390347cc0e8dd09addbb82c6a956e90dea420747a34c609acc9bee"} Oct 30 05:02:05.419788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:05.419716 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" podStartSLOduration=2.41969871 podStartE2EDuration="2.41969871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:05.415731548 +0000 UTC m=+1049.943498336" watchObservedRunningTime="2026-10-30 05:02:05.41969871 +0000 UTC m=+1049.947465494" Oct 30 05:02:06.373091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:06.373042 6587 generic.go:356] "Generic (PLEG): container finished" podID="c092142b-a383-485a-9c1a-cfd750e2f9e8" containerID="fb993ccdcdc854383879d76b8f9221ca69956a890e9709cb7bcb5489aa2bd87d" exitCode=0 Oct 30 05:02:06.373599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:06.373129 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" event={"ID":"c092142b-a383-485a-9c1a-cfd750e2f9e8","Type":"ContainerDied","Data":"fb993ccdcdc854383879d76b8f9221ca69956a890e9709cb7bcb5489aa2bd87d"} Oct 30 05:02:06.386402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:06.384039 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:02:06.386402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:06.384564 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:02:06.386402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:06.385911 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:02:07.564966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:07.564920 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" Oct 30 05:02:07.667895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:07.667796 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-openshift-ca\") pod \"c092142b-a383-485a-9c1a-cfd750e2f9e8\" (UID: \"c092142b-a383-485a-9c1a-cfd750e2f9e8\") " Oct 30 05:02:07.668080 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:07.667953 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-xks-ca\") pod \"c092142b-a383-485a-9c1a-cfd750e2f9e8\" (UID: \"c092142b-a383-485a-9c1a-cfd750e2f9e8\") " Oct 30 05:02:07.668152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:07.668129 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c092142b-a383-485a-9c1a-cfd750e2f9e8-kube-api-access-gzt7n\" (UniqueName: \"kubernetes.io/projected/c092142b-a383-485a-9c1a-cfd750e2f9e8-kube-api-access-gzt7n\") pod \"c092142b-a383-485a-9c1a-cfd750e2f9e8\" (UID: \"c092142b-a383-485a-9c1a-cfd750e2f9e8\") " Oct 30 05:02:07.668294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:07.668247 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-openshift-ca" pod "c092142b-a383-485a-9c1a-cfd750e2f9e8" (UID: "c092142b-a383-485a-9c1a-cfd750e2f9e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:07.668813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:07.668792 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:07.670311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:07.670278 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c092142b-a383-485a-9c1a-cfd750e2f9e8-kube-api-access-gzt7n" pod "c092142b-a383-485a-9c1a-cfd750e2f9e8" (UID: "c092142b-a383-485a-9c1a-cfd750e2f9e8"). InnerVolumeSpecName "kube-api-access-gzt7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:07.670436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:07.670340 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-xks-ca" pod "c092142b-a383-485a-9c1a-cfd750e2f9e8" (UID: "c092142b-a383-485a-9c1a-cfd750e2f9e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:07.769909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:07.769876 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzt7n\" (UniqueName: \"kubernetes.io/projected/c092142b-a383-485a-9c1a-cfd750e2f9e8-kube-api-access-gzt7n\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:07.769909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:07.769907 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c092142b-a383-485a-9c1a-cfd750e2f9e8-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:08.390511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:08.390479 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" Oct 30 05:02:08.390764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:08.390508 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm" event={"ID":"c092142b-a383-485a-9c1a-cfd750e2f9e8","Type":"ContainerDied","Data":"eb19f7ab9c390347cc0e8dd09addbb82c6a956e90dea420747a34c609acc9bee"} Oct 30 05:02:08.390764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:08.390562 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb19f7ab9c390347cc0e8dd09addbb82c6a956e90dea420747a34c609acc9bee" Oct 30 05:02:11.810495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:11.810451 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr"] Oct 30 05:02:11.813022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:11.812943 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c092142b-a383-485a-9c1a-cfd750e2f9e8" containerName="revoke-keys" Oct 30 05:02:11.813022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:11.812982 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c092142b-a383-485a-9c1a-cfd750e2f9e8" containerName="revoke-keys" Oct 30 05:02:11.817187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:11.813722 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c092142b-a383-485a-9c1a-cfd750e2f9e8" containerName="revoke-keys" Oct 30 05:02:11.822155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:11.822132 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" Oct 30 05:02:11.834723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:11.834057 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr"] Oct 30 05:02:11.841378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:11.841299 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:02:11.841553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:11.841374 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:02:11.841553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:11.841451 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:02:11.841553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:11.841500 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:02:11.940557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:11.940524 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjcz7\" (UniqueName: \"kubernetes.io/projected/4bd17d06-25da-4cbc-9e7a-a744f882a585-kube-api-access-fjcz7\") pod \"maas-api-revoke-sub-091ef8a9-7cprr\" (UID: \"4bd17d06-25da-4cbc-9e7a-a744f882a585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" Oct 30 05:02:11.940768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:11.940569 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-091ef8a9-7cprr\" (UID: \"4bd17d06-25da-4cbc-9e7a-a744f882a585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" Oct 30 05:02:11.940768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:11.940658 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-091ef8a9-7cprr\" (UID: \"4bd17d06-25da-4cbc-9e7a-a744f882a585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" Oct 30 05:02:12.042573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:12.042537 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjcz7\" (UniqueName: \"kubernetes.io/projected/4bd17d06-25da-4cbc-9e7a-a744f882a585-kube-api-access-fjcz7\") pod \"maas-api-revoke-sub-091ef8a9-7cprr\" (UID: \"4bd17d06-25da-4cbc-9e7a-a744f882a585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" Oct 30 05:02:12.042787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:12.042589 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-091ef8a9-7cprr\" (UID: \"4bd17d06-25da-4cbc-9e7a-a744f882a585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" Oct 30 05:02:12.042787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:12.042647 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-091ef8a9-7cprr\" (UID: \"4bd17d06-25da-4cbc-9e7a-a744f882a585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" Oct 30 05:02:12.043373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:12.043329 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-091ef8a9-7cprr\" (UID: \"4bd17d06-25da-4cbc-9e7a-a744f882a585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" Oct 30 05:02:12.045931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:12.045906 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-091ef8a9-7cprr\" (UID: \"4bd17d06-25da-4cbc-9e7a-a744f882a585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" Oct 30 05:02:12.065605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:12.065536 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjcz7\" (UniqueName: \"kubernetes.io/projected/4bd17d06-25da-4cbc-9e7a-a744f882a585-kube-api-access-fjcz7\") pod \"maas-api-revoke-sub-091ef8a9-7cprr\" (UID: \"4bd17d06-25da-4cbc-9e7a-a744f882a585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" Oct 30 05:02:12.156721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:12.156669 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" Oct 30 05:02:12.469171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:12.469087 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr"] Oct 30 05:02:13.442401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:13.442342 6587 generic.go:356] "Generic (PLEG): container finished" podID="4bd17d06-25da-4cbc-9e7a-a744f882a585" containerID="da77a721c45b74bf89b8ab9727c4395901182634128e070e3726c66f8cddca22" exitCode=0 Oct 30 05:02:13.442749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:13.442415 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" event={"ID":"4bd17d06-25da-4cbc-9e7a-a744f882a585","Type":"ContainerDied","Data":"da77a721c45b74bf89b8ab9727c4395901182634128e070e3726c66f8cddca22"} Oct 30 05:02:13.442749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:13.442475 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" event={"ID":"4bd17d06-25da-4cbc-9e7a-a744f882a585","Type":"ContainerStarted","Data":"a3273fc73b5665d5c5efc05d3187bdbfcca2560bcbb1f03c10e3f8093114214e"} Oct 30 05:02:14.652270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:14.652238 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" Oct 30 05:02:14.684643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:14.684608 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-xks-ca\") pod \"4bd17d06-25da-4cbc-9e7a-a744f882a585\" (UID: \"4bd17d06-25da-4cbc-9e7a-a744f882a585\") " Oct 30 05:02:14.684821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:14.684708 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd17d06-25da-4cbc-9e7a-a744f882a585-kube-api-access-fjcz7\" (UniqueName: \"kubernetes.io/projected/4bd17d06-25da-4cbc-9e7a-a744f882a585-kube-api-access-fjcz7\") pod \"4bd17d06-25da-4cbc-9e7a-a744f882a585\" (UID: \"4bd17d06-25da-4cbc-9e7a-a744f882a585\") " Oct 30 05:02:14.684886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:14.684815 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-openshift-ca\") pod \"4bd17d06-25da-4cbc-9e7a-a744f882a585\" (UID: \"4bd17d06-25da-4cbc-9e7a-a744f882a585\") " Oct 30 05:02:14.685144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:14.685117 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-openshift-ca" pod "4bd17d06-25da-4cbc-9e7a-a744f882a585" (UID: "4bd17d06-25da-4cbc-9e7a-a744f882a585"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:14.685522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:14.685499 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:14.686829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:14.686807 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-xks-ca" pod "4bd17d06-25da-4cbc-9e7a-a744f882a585" (UID: "4bd17d06-25da-4cbc-9e7a-a744f882a585"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:14.686914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:14.686854 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd17d06-25da-4cbc-9e7a-a744f882a585-kube-api-access-fjcz7" pod "4bd17d06-25da-4cbc-9e7a-a744f882a585" (UID: "4bd17d06-25da-4cbc-9e7a-a744f882a585"). InnerVolumeSpecName "kube-api-access-fjcz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:14.787703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:14.787612 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd17d06-25da-4cbc-9e7a-a744f882a585-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:14.787703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:14.787646 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjcz7\" (UniqueName: \"kubernetes.io/projected/4bd17d06-25da-4cbc-9e7a-a744f882a585-kube-api-access-fjcz7\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:15.463138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:15.463108 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" Oct 30 05:02:15.463311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:15.463107 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr" event={"ID":"4bd17d06-25da-4cbc-9e7a-a744f882a585","Type":"ContainerDied","Data":"a3273fc73b5665d5c5efc05d3187bdbfcca2560bcbb1f03c10e3f8093114214e"} Oct 30 05:02:15.463311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:15.463246 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3273fc73b5665d5c5efc05d3187bdbfcca2560bcbb1f03c10e3f8093114214e" Oct 30 05:02:16.299946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.299907 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b"] Oct 30 05:02:16.300633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.300616 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4bd17d06-25da-4cbc-9e7a-a744f882a585" containerName="revoke-keys" Oct 30 05:02:16.300633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.300632 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd17d06-25da-4cbc-9e7a-a744f882a585" containerName="revoke-keys" Oct 30 05:02:16.300789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.300779 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd17d06-25da-4cbc-9e7a-a744f882a585" containerName="revoke-keys" Oct 30 05:02:16.304942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.304926 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" Oct 30 05:02:16.325814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.325092 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b"] Oct 30 05:02:16.330401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.330353 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:02:16.330974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.330955 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:02:16.331053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.331041 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:02:16.339191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.339168 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:02:16.412288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.412258 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5117d40-9t74b\" (UID: \"1a063d68-3337-4c89-9630-c48676fae9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" Oct 30 05:02:16.412480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.412300 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg2fh\" (UniqueName: \"kubernetes.io/projected/1a063d68-3337-4c89-9630-c48676fae9b6-kube-api-access-vg2fh\") pod \"maas-api-revoke-sub-e5117d40-9t74b\" (UID: \"1a063d68-3337-4c89-9630-c48676fae9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" Oct 30 05:02:16.412480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.412452 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5117d40-9t74b\" (UID: \"1a063d68-3337-4c89-9630-c48676fae9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" Oct 30 05:02:16.513569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.513536 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5117d40-9t74b\" (UID: \"1a063d68-3337-4c89-9630-c48676fae9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" Oct 30 05:02:16.513762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.513587 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg2fh\" (UniqueName: \"kubernetes.io/projected/1a063d68-3337-4c89-9630-c48676fae9b6-kube-api-access-vg2fh\") pod \"maas-api-revoke-sub-e5117d40-9t74b\" (UID: \"1a063d68-3337-4c89-9630-c48676fae9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" Oct 30 05:02:16.513762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.513692 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5117d40-9t74b\" (UID: \"1a063d68-3337-4c89-9630-c48676fae9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" Oct 30 05:02:16.514243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.514219 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5117d40-9t74b\" (UID: \"1a063d68-3337-4c89-9630-c48676fae9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" Oct 30 05:02:16.516443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.516419 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5117d40-9t74b\" (UID: \"1a063d68-3337-4c89-9630-c48676fae9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" Oct 30 05:02:16.534310 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.534289 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg2fh\" (UniqueName: \"kubernetes.io/projected/1a063d68-3337-4c89-9630-c48676fae9b6-kube-api-access-vg2fh\") pod \"maas-api-revoke-sub-e5117d40-9t74b\" (UID: \"1a063d68-3337-4c89-9630-c48676fae9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" Oct 30 05:02:16.630489 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.630403 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" Oct 30 05:02:16.937852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:16.937816 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b"] Oct 30 05:02:17.482768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:17.482727 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" event={"ID":"1a063d68-3337-4c89-9630-c48676fae9b6","Type":"ContainerStarted","Data":"8f7f4444fcd0bf2c9f229ac0dba83b341140c78d5b9fdd51e8149781c6bb255f"} Oct 30 05:02:17.483210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:17.482797 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" event={"ID":"1a063d68-3337-4c89-9630-c48676fae9b6","Type":"ContainerStarted","Data":"71f20e75e61268f3a5b43d1b1282fb621748082cf6679145aeade50e39c12f62"} Oct 30 05:02:17.531004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:17.530934 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" podStartSLOduration=1.530915925 podStartE2EDuration="1.530915925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:17.524228894 +0000 UTC m=+1062.051995680" watchObservedRunningTime="2026-10-30 05:02:17.530915925 +0000 UTC m=+1062.058682711" Oct 30 05:02:17.987988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:17.987953 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c"] Oct 30 05:02:17.992932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:17.992912 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" Oct 30 05:02:18.009973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.009945 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c"] Oct 30 05:02:18.053040 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.053010 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-929a649b-5xq7c\" (UID: \"032e3aad-0862-4063-b39c-0eb858bf7584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" Oct 30 05:02:18.053227 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.053133 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-929a649b-5xq7c\" (UID: \"032e3aad-0862-4063-b39c-0eb858bf7584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" Oct 30 05:02:18.053420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.053396 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zwfp\" (UniqueName: \"kubernetes.io/projected/032e3aad-0862-4063-b39c-0eb858bf7584-kube-api-access-8zwfp\") pod \"maas-api-revoke-sub-929a649b-5xq7c\" (UID: \"032e3aad-0862-4063-b39c-0eb858bf7584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" Oct 30 05:02:18.154608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.154521 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zwfp\" (UniqueName: \"kubernetes.io/projected/032e3aad-0862-4063-b39c-0eb858bf7584-kube-api-access-8zwfp\") pod \"maas-api-revoke-sub-929a649b-5xq7c\" (UID: \"032e3aad-0862-4063-b39c-0eb858bf7584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" Oct 30 05:02:18.154754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.154665 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-929a649b-5xq7c\" (UID: \"032e3aad-0862-4063-b39c-0eb858bf7584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" Oct 30 05:02:18.154841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.154754 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-929a649b-5xq7c\" (UID: \"032e3aad-0862-4063-b39c-0eb858bf7584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" Oct 30 05:02:18.155445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.155348 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-929a649b-5xq7c\" (UID: \"032e3aad-0862-4063-b39c-0eb858bf7584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" Oct 30 05:02:18.157707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.157681 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-929a649b-5xq7c\" (UID: \"032e3aad-0862-4063-b39c-0eb858bf7584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" Oct 30 05:02:18.175008 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.174982 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zwfp\" (UniqueName: \"kubernetes.io/projected/032e3aad-0862-4063-b39c-0eb858bf7584-kube-api-access-8zwfp\") pod \"maas-api-revoke-sub-929a649b-5xq7c\" (UID: \"032e3aad-0862-4063-b39c-0eb858bf7584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" Oct 30 05:02:18.310433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.310402 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" Oct 30 05:02:18.500965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.500879 6587 generic.go:356] "Generic (PLEG): container finished" podID="1a063d68-3337-4c89-9630-c48676fae9b6" containerID="8f7f4444fcd0bf2c9f229ac0dba83b341140c78d5b9fdd51e8149781c6bb255f" exitCode=0 Oct 30 05:02:18.501498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.500953 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" event={"ID":"1a063d68-3337-4c89-9630-c48676fae9b6","Type":"ContainerDied","Data":"8f7f4444fcd0bf2c9f229ac0dba83b341140c78d5b9fdd51e8149781c6bb255f"} Oct 30 05:02:18.632059 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:18.632023 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c"] Oct 30 05:02:19.519290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:19.518115 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" event={"ID":"032e3aad-0862-4063-b39c-0eb858bf7584","Type":"ContainerStarted","Data":"4c434c105254c39a4e939f3b7cc21b7cefd1294efaff0f586207190c2b5e0f2b"} Oct 30 05:02:19.519290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:19.518177 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" event={"ID":"032e3aad-0862-4063-b39c-0eb858bf7584","Type":"ContainerStarted","Data":"70d368be2edb004050c3064d5f40c48bab7e50012837b1737cc5711244a3ae1a"} Oct 30 05:02:19.573402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:19.573174 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" podStartSLOduration=2.573126103 podStartE2EDuration="2.573126103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:19.557863724 +0000 UTC m=+1064.085630511" watchObservedRunningTime="2026-10-30 05:02:19.573126103 +0000 UTC m=+1064.100892890" Oct 30 05:02:20.007379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.005561 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" Oct 30 05:02:20.110739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.108582 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-openshift-ca\") pod \"1a063d68-3337-4c89-9630-c48676fae9b6\" (UID: \"1a063d68-3337-4c89-9630-c48676fae9b6\") " Oct 30 05:02:20.110739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.108722 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-xks-ca\") pod \"1a063d68-3337-4c89-9630-c48676fae9b6\" (UID: \"1a063d68-3337-4c89-9630-c48676fae9b6\") " Oct 30 05:02:20.110739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.108928 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a063d68-3337-4c89-9630-c48676fae9b6-kube-api-access-vg2fh\" (UniqueName: \"kubernetes.io/projected/1a063d68-3337-4c89-9630-c48676fae9b6-kube-api-access-vg2fh\") pod \"1a063d68-3337-4c89-9630-c48676fae9b6\" (UID: \"1a063d68-3337-4c89-9630-c48676fae9b6\") " Oct 30 05:02:20.119624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.119588 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-openshift-ca" pod "1a063d68-3337-4c89-9630-c48676fae9b6" (UID: "1a063d68-3337-4c89-9630-c48676fae9b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:20.239394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.227183 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:20.239394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.227410 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a063d68-3337-4c89-9630-c48676fae9b6-kube-api-access-vg2fh" pod "1a063d68-3337-4c89-9630-c48676fae9b6" (UID: "1a063d68-3337-4c89-9630-c48676fae9b6"). InnerVolumeSpecName "kube-api-access-vg2fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:20.265275 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.265230 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-xks-ca" pod "1a063d68-3337-4c89-9630-c48676fae9b6" (UID: "1a063d68-3337-4c89-9630-c48676fae9b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:20.330131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.328712 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a063d68-3337-4c89-9630-c48676fae9b6-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:20.330131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.328751 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg2fh\" (UniqueName: \"kubernetes.io/projected/1a063d68-3337-4c89-9630-c48676fae9b6-kube-api-access-vg2fh\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:20.727384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.723476 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" event={"ID":"1a063d68-3337-4c89-9630-c48676fae9b6","Type":"ContainerDied","Data":"71f20e75e61268f3a5b43d1b1282fb621748082cf6679145aeade50e39c12f62"} Oct 30 05:02:20.727384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.723562 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f20e75e61268f3a5b43d1b1282fb621748082cf6679145aeade50e39c12f62" Oct 30 05:02:20.727384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.723689 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b" Oct 30 05:02:20.798382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.787484 6587 generic.go:356] "Generic (PLEG): container finished" podID="032e3aad-0862-4063-b39c-0eb858bf7584" containerID="4c434c105254c39a4e939f3b7cc21b7cefd1294efaff0f586207190c2b5e0f2b" exitCode=0 Oct 30 05:02:20.798382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:20.787720 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" event={"ID":"032e3aad-0862-4063-b39c-0eb858bf7584","Type":"ContainerDied","Data":"4c434c105254c39a4e939f3b7cc21b7cefd1294efaff0f586207190c2b5e0f2b"} Oct 30 05:02:21.501832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:21.501781 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:21.502876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:21.502843 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:02:21.625292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:21.623448 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:21.625292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:21.624622 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:21.722390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:21.721515 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:21.722390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:21.721903 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:21.992382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:21.956491 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:21.992382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:21.971510 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:02:21.992382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:21.971946 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:02:22.518388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:22.508712 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:22.518388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:22.509020 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:02:22.518388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:22.509623 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:02:22.518388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:22.509712 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:02:22.686070 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:22.686023 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:22.708385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:22.687603 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:02:22.787941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:22.709425 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:02:22.788232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:22.788211 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:02:22.789705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:22.731387 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:22.790875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:22.731462 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:02:22.791234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:22.791216 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:02:22.791410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:22.791395 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:02:22.997765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:22.972461 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:23.036385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:23.001244 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:23.036385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:23.002034 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:23.036385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:23.003019 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:02:23.036385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:23.003104 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:02:23.036385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:23.003159 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:02:23.570386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:23.553509 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/health\": context deadline exceeded" Oct 30 05:02:23.768953 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:23.768898 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:02:23.882146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:23.807519 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:02:23.882146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:23.869711 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:24.072491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:24.050091 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:24.072491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:24.069215 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:02:24.072491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:24.071581 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:02:24.622507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:24.563688 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" podUID="da31dd7c-961b-46e8-9f28-1e7e6780fa10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 05:02:24.825405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:24.824509 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:24.837520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:24.837466 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:02:24.980735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:24.980611 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:02:25.046516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.046476 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:02:25.067274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.067223 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:25.067870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.067838 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:02:25.069503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.069454 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:02:25.087989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.069664 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:02:25.106261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.106216 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" podUID="d02fa1f2-298d-4a23-a302-6a59e1cbd188" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.30:8443/ready\": context deadline exceeded" Oct 30 05:02:25.106735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.106689 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:25.106989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.106962 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:25.107300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.107272 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:25.329990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.329946 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" podUID="c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:25.604455 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:02:25.603469 6587 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.026s" Oct 30 05:02:25.629383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.606495 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" podUID="da31dd7c-961b-46e8-9f28-1e7e6780fa10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 05:02:25.831383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.827481 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:25.860464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.860039 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:25.903386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.885229 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 05:02:25.933522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.903745 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n" Oct 30 05:02:25.984384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.983939 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" Oct 30 05:02:25.990427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:25.986647 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:26.073659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.073612 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:26.132439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.098626 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:26.184956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.110735 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:26.245384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.207145 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-xks-ca\") pod \"032e3aad-0862-4063-b39c-0eb858bf7584\" (UID: \"032e3aad-0862-4063-b39c-0eb858bf7584\") " Oct 30 05:02:26.245384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.207212 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032e3aad-0862-4063-b39c-0eb858bf7584-kube-api-access-8zwfp\" (UniqueName: \"kubernetes.io/projected/032e3aad-0862-4063-b39c-0eb858bf7584-kube-api-access-8zwfp\") pod \"032e3aad-0862-4063-b39c-0eb858bf7584\" (UID: \"032e3aad-0862-4063-b39c-0eb858bf7584\") " Oct 30 05:02:26.245384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.207376 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-openshift-ca\") pod \"032e3aad-0862-4063-b39c-0eb858bf7584\" (UID: \"032e3aad-0862-4063-b39c-0eb858bf7584\") " Oct 30 05:02:26.245384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.223345 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:02:26.418330 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.390316 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-xks-ca" pod "032e3aad-0862-4063-b39c-0eb858bf7584" (UID: "032e3aad-0862-4063-b39c-0eb858bf7584"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:26.468480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.468433 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:02:26.567420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.469447 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:02:26.567420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.488667 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032e3aad-0862-4063-b39c-0eb858bf7584-kube-api-access-8zwfp" pod "032e3aad-0862-4063-b39c-0eb858bf7584" (UID: "032e3aad-0862-4063-b39c-0eb858bf7584"). InnerVolumeSpecName "kube-api-access-8zwfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:26.567420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.536590 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-openshift-ca" pod "032e3aad-0862-4063-b39c-0eb858bf7584" (UID: "032e3aad-0862-4063-b39c-0eb858bf7584"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:26.672510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.672413 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-openshift-ca\") pod \"032e3aad-0862-4063-b39c-0eb858bf7584\" (UID: \"032e3aad-0862-4063-b39c-0eb858bf7584\") " Oct 30 05:02:26.681384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.673523 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:26.681632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.681610 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zwfp\" (UniqueName: \"kubernetes.io/projected/032e3aad-0862-4063-b39c-0eb858bf7584-kube-api-access-8zwfp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:26.681728 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:02:26.680641 6587 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/032e3aad-0862-4063-b39c-0eb858bf7584/volumes/kubernetes.io~configmap/maas-api-openshift-ca Oct 30 05:02:26.681816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.681799 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-openshift-ca" pod "032e3aad-0862-4063-b39c-0eb858bf7584" (UID: "032e3aad-0862-4063-b39c-0eb858bf7584"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:26.767543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.763525 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:26.847771 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:02:26.847727 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:02:26.853942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.853914 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" Oct 30 05:02:26.861498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.857705 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c" event={"ID":"032e3aad-0862-4063-b39c-0eb858bf7584","Type":"ContainerDied","Data":"70d368be2edb004050c3064d5f40c48bab7e50012837b1737cc5711244a3ae1a"} Oct 30 05:02:26.861498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.857769 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70d368be2edb004050c3064d5f40c48bab7e50012837b1737cc5711244a3ae1a" Oct 30 05:02:26.931450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.920055 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:26.931450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.921421 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:02:26.996116 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:26.995537 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:02:27.006385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:27.002688 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032e3aad-0862-4063-b39c-0eb858bf7584-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:27.086629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:27.086586 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:27.298416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:27.267149 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:27.338515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:27.301460 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:27.341392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:27.324132 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 05:02:27.345481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:27.324533 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:27.544693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:27.472911 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:27.544693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:27.481818 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:27.546946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:27.546903 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": context deadline exceeded" Oct 30 05:02:27.924933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:27.922111 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:27.924933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:27.922492 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:27.993320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:27.993273 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:28.022753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.022706 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:28.292864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.289079 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:28.380382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.364816 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:02:28.380382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.378738 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:02:28.480043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.480004 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf"] Oct 30 05:02:28.481735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.481704 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1a063d68-3337-4c89-9630-c48676fae9b6" containerName="revoke-keys" Oct 30 05:02:28.481735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.481736 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a063d68-3337-4c89-9630-c48676fae9b6" containerName="revoke-keys" Oct 30 05:02:28.481916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.481770 6587 cpu_manager.go:414] "containerMap: removing container" podUID="032e3aad-0862-4063-b39c-0eb858bf7584" containerName="revoke-keys" Oct 30 05:02:28.481916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.481783 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032e3aad-0862-4063-b39c-0eb858bf7584" containerName="revoke-keys" Oct 30 05:02:28.482280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.482215 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a063d68-3337-4c89-9630-c48676fae9b6" containerName="revoke-keys" Oct 30 05:02:28.482280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.482247 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="032e3aad-0862-4063-b39c-0eb858bf7584" containerName="revoke-keys" Oct 30 05:02:28.484971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.484939 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:28.486644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.486617 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:28.498286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.489483 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:02:28.498286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.489543 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:02:28.498286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.489609 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:02:28.498286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.490088 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:02:28.498286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.490737 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" Oct 30 05:02:28.498286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.492569 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:02:28.508499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.508472 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:02:28.509973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.509951 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:02:28.528793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.528764 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:02:28.528930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.528857 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:02:28.528930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.528872 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:02:28.529109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.529030 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:02:28.533619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.533402 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:02:28.533619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.533461 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:02:28.537613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.537545 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf"] Oct 30 05:02:28.562929 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:02:28.562897 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032e3aad_0862_4063_b39c_0eb858bf7584.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032e3aad_0862_4063_b39c_0eb858bf7584.slice/crio-70d368be2edb004050c3064d5f40c48bab7e50012837b1737cc5711244a3ae1a\": RecentStats: unable to find data in memory cache]" Oct 30 05:02:28.581658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.581626 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dk6l\" (UniqueName: \"kubernetes.io/projected/214e7e6b-359a-4c41-8a15-7fa47ae39c78-kube-api-access-6dk6l\") pod \"maas-api-revoke-sub-a8e1afcc-cq5cf\" (UID: \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" Oct 30 05:02:28.581808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.581741 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8e1afcc-cq5cf\" (UID: \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" Oct 30 05:02:28.582528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.582350 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8e1afcc-cq5cf\" (UID: \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" Oct 30 05:02:28.684319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.684257 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8e1afcc-cq5cf\" (UID: \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" Oct 30 05:02:28.686138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.685235 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8e1afcc-cq5cf\" (UID: \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" Oct 30 05:02:28.686782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.686535 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dk6l\" (UniqueName: \"kubernetes.io/projected/214e7e6b-359a-4c41-8a15-7fa47ae39c78-kube-api-access-6dk6l\") pod \"maas-api-revoke-sub-a8e1afcc-cq5cf\" (UID: \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" Oct 30 05:02:28.687542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.687514 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8e1afcc-cq5cf\" (UID: \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" Oct 30 05:02:28.691419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.691396 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8e1afcc-cq5cf\" (UID: \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" Oct 30 05:02:28.708761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.708736 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dk6l\" (UniqueName: \"kubernetes.io/projected/214e7e6b-359a-4c41-8a15-7fa47ae39c78-kube-api-access-6dk6l\") pod \"maas-api-revoke-sub-a8e1afcc-cq5cf\" (UID: \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" Oct 30 05:02:28.831423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:28.831309 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" Oct 30 05:02:29.195100 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:29.195069 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf"] Oct 30 05:02:29.196647 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:02:29.196617 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214e7e6b_359a_4c41_8a15_7fa47ae39c78.slice/crio-4d52d3e49c2de190b13e74facd193841d154600cbe4469cd52d9758022771f82 WatchSource:0}: Error finding container 4d52d3e49c2de190b13e74facd193841d154600cbe4469cd52d9758022771f82: Status 404 returned error can't find the container with id 4d52d3e49c2de190b13e74facd193841d154600cbe4469cd52d9758022771f82 Oct 30 05:02:29.495738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:29.495662 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" event={"ID":"214e7e6b-359a-4c41-8a15-7fa47ae39c78","Type":"ContainerStarted","Data":"4d52d3e49c2de190b13e74facd193841d154600cbe4469cd52d9758022771f82"} Oct 30 05:02:30.507645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:30.507610 6587 generic.go:356] "Generic (PLEG): container finished" podID="214e7e6b-359a-4c41-8a15-7fa47ae39c78" containerID="be2d0590e2d30a3757f70aed20ee703b176113ccc9e27de362f1765285f3e24d" exitCode=0 Oct 30 05:02:30.508084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:30.507691 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" event={"ID":"214e7e6b-359a-4c41-8a15-7fa47ae39c78","Type":"ContainerDied","Data":"be2d0590e2d30a3757f70aed20ee703b176113ccc9e27de362f1765285f3e24d"} Oct 30 05:02:31.720223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:31.720196 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" Oct 30 05:02:31.847228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:31.847194 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-xks-ca\") pod \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\" (UID: \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\") " Oct 30 05:02:31.847536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:31.847521 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/214e7e6b-359a-4c41-8a15-7fa47ae39c78-kube-api-access-6dk6l\" (UniqueName: \"kubernetes.io/projected/214e7e6b-359a-4c41-8a15-7fa47ae39c78-kube-api-access-6dk6l\") pod \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\" (UID: \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\") " Oct 30 05:02:31.847627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:31.847547 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-openshift-ca\") pod \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\" (UID: \"214e7e6b-359a-4c41-8a15-7fa47ae39c78\") " Oct 30 05:02:31.848020 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:31.847989 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-openshift-ca" pod "214e7e6b-359a-4c41-8a15-7fa47ae39c78" (UID: "214e7e6b-359a-4c41-8a15-7fa47ae39c78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:31.848717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:31.848683 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:31.849603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:31.849572 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-xks-ca" pod "214e7e6b-359a-4c41-8a15-7fa47ae39c78" (UID: "214e7e6b-359a-4c41-8a15-7fa47ae39c78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:31.849705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:31.849675 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214e7e6b-359a-4c41-8a15-7fa47ae39c78-kube-api-access-6dk6l" pod "214e7e6b-359a-4c41-8a15-7fa47ae39c78" (UID: "214e7e6b-359a-4c41-8a15-7fa47ae39c78"). InnerVolumeSpecName "kube-api-access-6dk6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:31.949960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:31.949927 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dk6l\" (UniqueName: \"kubernetes.io/projected/214e7e6b-359a-4c41-8a15-7fa47ae39c78-kube-api-access-6dk6l\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:31.949960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:31.949957 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/214e7e6b-359a-4c41-8a15-7fa47ae39c78-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:32.527402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:32.527351 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" Oct 30 05:02:32.527582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:32.527391 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf" event={"ID":"214e7e6b-359a-4c41-8a15-7fa47ae39c78","Type":"ContainerDied","Data":"4d52d3e49c2de190b13e74facd193841d154600cbe4469cd52d9758022771f82"} Oct 30 05:02:32.527582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:32.527492 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d52d3e49c2de190b13e74facd193841d154600cbe4469cd52d9758022771f82" Oct 30 05:02:36.974337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:36.974294 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5"] Oct 30 05:02:36.975459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:36.975438 6587 cpu_manager.go:414] "containerMap: removing container" podUID="214e7e6b-359a-4c41-8a15-7fa47ae39c78" containerName="revoke-keys" Oct 30 05:02:36.975459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:36.975459 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="214e7e6b-359a-4c41-8a15-7fa47ae39c78" containerName="revoke-keys" Oct 30 05:02:36.975639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:36.975627 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="214e7e6b-359a-4c41-8a15-7fa47ae39c78" containerName="revoke-keys" Oct 30 05:02:36.980396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:36.980373 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" Oct 30 05:02:36.995830 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:36.995560 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5"] Oct 30 05:02:37.002011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.001429 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:02:37.002011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.001450 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:02:37.002011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.001552 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:02:37.002011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.001588 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:02:37.134447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.134412 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c614e530-kbsc5\" (UID: \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" Oct 30 05:02:37.134727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.134706 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c614e530-kbsc5\" (UID: \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" Oct 30 05:02:37.134904 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.134882 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvhrj\" (UniqueName: \"kubernetes.io/projected/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-kube-api-access-fvhrj\") pod \"maas-api-revoke-sub-c614e530-kbsc5\" (UID: \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" Oct 30 05:02:37.236120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.236029 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c614e530-kbsc5\" (UID: \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" Oct 30 05:02:37.236285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.236160 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvhrj\" (UniqueName: \"kubernetes.io/projected/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-kube-api-access-fvhrj\") pod \"maas-api-revoke-sub-c614e530-kbsc5\" (UID: \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" Oct 30 05:02:37.236285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.236225 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c614e530-kbsc5\" (UID: \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" Oct 30 05:02:37.236834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.236794 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c614e530-kbsc5\" (UID: \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" Oct 30 05:02:37.238960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.238930 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c614e530-kbsc5\" (UID: \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" Oct 30 05:02:37.256757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.256726 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvhrj\" (UniqueName: \"kubernetes.io/projected/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-kube-api-access-fvhrj\") pod \"maas-api-revoke-sub-c614e530-kbsc5\" (UID: \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" Oct 30 05:02:37.299842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.299810 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" Oct 30 05:02:37.613380 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:02:37.610720 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf5b2b3_385d_4cbc_9693_e5acb0bb561a.slice/crio-19a8fe6fc0de3890995608145869388df7a91c3a848c098302cf6d72b98d5c57 WatchSource:0}: Error finding container 19a8fe6fc0de3890995608145869388df7a91c3a848c098302cf6d72b98d5c57: Status 404 returned error can't find the container with id 19a8fe6fc0de3890995608145869388df7a91c3a848c098302cf6d72b98d5c57 Oct 30 05:02:37.617883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:37.617854 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5"] Oct 30 05:02:38.597575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:38.597527 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" event={"ID":"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a","Type":"ContainerStarted","Data":"9a084897c25fb371ca4f0548b783d76ae6570a6de4d180b83e5012d72bd2795c"} Oct 30 05:02:38.598044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:38.597592 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" event={"ID":"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a","Type":"ContainerStarted","Data":"19a8fe6fc0de3890995608145869388df7a91c3a848c098302cf6d72b98d5c57"} Oct 30 05:02:38.643269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:38.643207 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" podStartSLOduration=2.643187255 podStartE2EDuration="2.643187255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:38.639786221 +0000 UTC m=+1083.167553008" watchObservedRunningTime="2026-10-30 05:02:38.643187255 +0000 UTC m=+1083.170954044" Oct 30 05:02:38.981832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:38.981754 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq"] Oct 30 05:02:38.986482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:38.986459 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" Oct 30 05:02:39.003948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.003924 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq"] Oct 30 05:02:39.167200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.167128 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f33b40d8-7cjpq\" (UID: \"a498169d-c8ad-49b6-b836-e6898140eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" Oct 30 05:02:39.167200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.167173 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f33b40d8-7cjpq\" (UID: \"a498169d-c8ad-49b6-b836-e6898140eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" Oct 30 05:02:39.167872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.167428 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc58q\" (UniqueName: \"kubernetes.io/projected/a498169d-c8ad-49b6-b836-e6898140eb8e-kube-api-access-lc58q\") pod \"maas-api-revoke-sub-f33b40d8-7cjpq\" (UID: \"a498169d-c8ad-49b6-b836-e6898140eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" Oct 30 05:02:39.268937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.268831 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f33b40d8-7cjpq\" (UID: \"a498169d-c8ad-49b6-b836-e6898140eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" Oct 30 05:02:39.268937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.268876 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f33b40d8-7cjpq\" (UID: \"a498169d-c8ad-49b6-b836-e6898140eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" Oct 30 05:02:39.269199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.269056 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc58q\" (UniqueName: \"kubernetes.io/projected/a498169d-c8ad-49b6-b836-e6898140eb8e-kube-api-access-lc58q\") pod \"maas-api-revoke-sub-f33b40d8-7cjpq\" (UID: \"a498169d-c8ad-49b6-b836-e6898140eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" Oct 30 05:02:39.269853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.269824 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f33b40d8-7cjpq\" (UID: \"a498169d-c8ad-49b6-b836-e6898140eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" Oct 30 05:02:39.271353 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.271331 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f33b40d8-7cjpq\" (UID: \"a498169d-c8ad-49b6-b836-e6898140eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" Oct 30 05:02:39.289451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.289426 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc58q\" (UniqueName: \"kubernetes.io/projected/a498169d-c8ad-49b6-b836-e6898140eb8e-kube-api-access-lc58q\") pod \"maas-api-revoke-sub-f33b40d8-7cjpq\" (UID: \"a498169d-c8ad-49b6-b836-e6898140eb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" Oct 30 05:02:39.302394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.302334 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" Oct 30 05:02:39.615588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.615560 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq"] Oct 30 05:02:39.615978 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:02:39.615832 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda498169d_c8ad_49b6_b836_e6898140eb8e.slice/crio-ae7b6018628d38a91f867c48fdd389c1f7e72a2044e2e01f2d3ac3d3ed02715a WatchSource:0}: Error finding container ae7b6018628d38a91f867c48fdd389c1f7e72a2044e2e01f2d3ac3d3ed02715a: Status 404 returned error can't find the container with id ae7b6018628d38a91f867c48fdd389c1f7e72a2044e2e01f2d3ac3d3ed02715a Oct 30 05:02:39.619000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.618964 6587 generic.go:356] "Generic (PLEG): container finished" podID="eaf5b2b3-385d-4cbc-9693-e5acb0bb561a" containerID="9a084897c25fb371ca4f0548b783d76ae6570a6de4d180b83e5012d72bd2795c" exitCode=0 Oct 30 05:02:39.619118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:39.619090 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" event={"ID":"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a","Type":"ContainerDied","Data":"9a084897c25fb371ca4f0548b783d76ae6570a6de4d180b83e5012d72bd2795c"} Oct 30 05:02:40.630663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:40.630625 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" event={"ID":"a498169d-c8ad-49b6-b836-e6898140eb8e","Type":"ContainerStarted","Data":"0c234ee2b4b495280571d336ca346424239d290e85b3ccaaad9733f19b27cba7"} Oct 30 05:02:40.631093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:40.630691 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" event={"ID":"a498169d-c8ad-49b6-b836-e6898140eb8e","Type":"ContainerStarted","Data":"ae7b6018628d38a91f867c48fdd389c1f7e72a2044e2e01f2d3ac3d3ed02715a"} Oct 30 05:02:40.679629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:40.679565 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" podStartSLOduration=2.679545316 podStartE2EDuration="2.679545316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:40.672810425 +0000 UTC m=+1085.200577213" watchObservedRunningTime="2026-10-30 05:02:40.679545316 +0000 UTC m=+1085.207312104" Oct 30 05:02:40.853213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:40.853185 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" Oct 30 05:02:40.906021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:40.905943 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-xks-ca\") pod \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\" (UID: \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\") " Oct 30 05:02:40.906186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:40.906043 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-openshift-ca\") pod \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\" (UID: \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\") " Oct 30 05:02:40.906186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:40.906135 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-kube-api-access-fvhrj\" (UniqueName: \"kubernetes.io/projected/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-kube-api-access-fvhrj\") pod \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\" (UID: \"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a\") " Oct 30 05:02:40.906462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:40.906430 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-openshift-ca" pod "eaf5b2b3-385d-4cbc-9693-e5acb0bb561a" (UID: "eaf5b2b3-385d-4cbc-9693-e5acb0bb561a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:40.906943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:40.906915 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:40.908231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:40.908198 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-xks-ca" pod "eaf5b2b3-385d-4cbc-9693-e5acb0bb561a" (UID: "eaf5b2b3-385d-4cbc-9693-e5acb0bb561a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:40.908337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:40.908311 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-kube-api-access-fvhrj" pod "eaf5b2b3-385d-4cbc-9693-e5acb0bb561a" (UID: "eaf5b2b3-385d-4cbc-9693-e5acb0bb561a"). InnerVolumeSpecName "kube-api-access-fvhrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:41.008658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:41.008622 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvhrj\" (UniqueName: \"kubernetes.io/projected/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-kube-api-access-fvhrj\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:41.008658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:41.008654 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:41.644192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:41.644150 6587 generic.go:356] "Generic (PLEG): container finished" podID="a498169d-c8ad-49b6-b836-e6898140eb8e" containerID="0c234ee2b4b495280571d336ca346424239d290e85b3ccaaad9733f19b27cba7" exitCode=0 Oct 30 05:02:41.644660 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:41.644245 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" event={"ID":"a498169d-c8ad-49b6-b836-e6898140eb8e","Type":"ContainerDied","Data":"0c234ee2b4b495280571d336ca346424239d290e85b3ccaaad9733f19b27cba7"} Oct 30 05:02:41.646044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:41.646026 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" Oct 30 05:02:41.646174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:41.646045 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5" event={"ID":"eaf5b2b3-385d-4cbc-9693-e5acb0bb561a","Type":"ContainerDied","Data":"19a8fe6fc0de3890995608145869388df7a91c3a848c098302cf6d72b98d5c57"} Oct 30 05:02:41.646174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:41.646078 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19a8fe6fc0de3890995608145869388df7a91c3a848c098302cf6d72b98d5c57" Oct 30 05:02:42.864124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:42.864097 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" Oct 30 05:02:42.882505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:42.882480 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-xks-ca\") pod \"a498169d-c8ad-49b6-b836-e6898140eb8e\" (UID: \"a498169d-c8ad-49b6-b836-e6898140eb8e\") " Oct 30 05:02:42.882622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:42.882536 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a498169d-c8ad-49b6-b836-e6898140eb8e-kube-api-access-lc58q\" (UniqueName: \"kubernetes.io/projected/a498169d-c8ad-49b6-b836-e6898140eb8e-kube-api-access-lc58q\") pod \"a498169d-c8ad-49b6-b836-e6898140eb8e\" (UID: \"a498169d-c8ad-49b6-b836-e6898140eb8e\") " Oct 30 05:02:42.882622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:42.882562 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-openshift-ca\") pod \"a498169d-c8ad-49b6-b836-e6898140eb8e\" (UID: \"a498169d-c8ad-49b6-b836-e6898140eb8e\") " Oct 30 05:02:42.883029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:42.883001 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-openshift-ca" pod "a498169d-c8ad-49b6-b836-e6898140eb8e" (UID: "a498169d-c8ad-49b6-b836-e6898140eb8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:42.883519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:42.883498 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:42.884609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:42.884587 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-xks-ca" pod "a498169d-c8ad-49b6-b836-e6898140eb8e" (UID: "a498169d-c8ad-49b6-b836-e6898140eb8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:42.884674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:42.884647 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a498169d-c8ad-49b6-b836-e6898140eb8e-kube-api-access-lc58q" pod "a498169d-c8ad-49b6-b836-e6898140eb8e" (UID: "a498169d-c8ad-49b6-b836-e6898140eb8e"). InnerVolumeSpecName "kube-api-access-lc58q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:42.985473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:42.985385 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a498169d-c8ad-49b6-b836-e6898140eb8e-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:42.985473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:42.985419 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc58q\" (UniqueName: \"kubernetes.io/projected/a498169d-c8ad-49b6-b836-e6898140eb8e-kube-api-access-lc58q\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:43.668459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:43.668429 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" Oct 30 05:02:43.668640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:43.668476 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq" event={"ID":"a498169d-c8ad-49b6-b836-e6898140eb8e","Type":"ContainerDied","Data":"ae7b6018628d38a91f867c48fdd389c1f7e72a2044e2e01f2d3ac3d3ed02715a"} Oct 30 05:02:43.668640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:43.668530 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae7b6018628d38a91f867c48fdd389c1f7e72a2044e2e01f2d3ac3d3ed02715a" Oct 30 05:02:47.101056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.100975 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm"] Oct 30 05:02:47.101899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.101879 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a498169d-c8ad-49b6-b836-e6898140eb8e" containerName="revoke-keys" Oct 30 05:02:47.101948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.101900 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a498169d-c8ad-49b6-b836-e6898140eb8e" containerName="revoke-keys" Oct 30 05:02:47.101948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.101925 6587 cpu_manager.go:414] "containerMap: removing container" podUID="eaf5b2b3-385d-4cbc-9693-e5acb0bb561a" containerName="revoke-keys" Oct 30 05:02:47.101948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.101931 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf5b2b3-385d-4cbc-9693-e5acb0bb561a" containerName="revoke-keys" Oct 30 05:02:47.102169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.102028 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf5b2b3-385d-4cbc-9693-e5acb0bb561a" containerName="revoke-keys" Oct 30 05:02:47.102169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.102045 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a498169d-c8ad-49b6-b836-e6898140eb8e" containerName="revoke-keys" Oct 30 05:02:47.109538 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.109509 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" Oct 30 05:02:47.126618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.126577 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm"] Oct 30 05:02:47.130760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.130735 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:02:47.130946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.130932 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:02:47.131121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.131105 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:02:47.139234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.139208 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:02:47.161432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.161398 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2k24\" (UniqueName: \"kubernetes.io/projected/90e878d6-d55f-4ca9-a3d8-66e6d8379172-kube-api-access-r2k24\") pod \"maas-api-revoke-sub-d5399c41-qdbvm\" (UID: \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" Oct 30 05:02:47.161674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.161643 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5399c41-qdbvm\" (UID: \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" Oct 30 05:02:47.161771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.161723 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5399c41-qdbvm\" (UID: \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" Oct 30 05:02:47.263247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.263206 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5399c41-qdbvm\" (UID: \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" Oct 30 05:02:47.263474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.263273 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5399c41-qdbvm\" (UID: \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" Oct 30 05:02:47.263474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.263435 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2k24\" (UniqueName: \"kubernetes.io/projected/90e878d6-d55f-4ca9-a3d8-66e6d8379172-kube-api-access-r2k24\") pod \"maas-api-revoke-sub-d5399c41-qdbvm\" (UID: \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" Oct 30 05:02:47.264016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.263989 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5399c41-qdbvm\" (UID: \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" Oct 30 05:02:47.265841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.265820 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5399c41-qdbvm\" (UID: \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" Oct 30 05:02:47.284186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.284153 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2k24\" (UniqueName: \"kubernetes.io/projected/90e878d6-d55f-4ca9-a3d8-66e6d8379172-kube-api-access-r2k24\") pod \"maas-api-revoke-sub-d5399c41-qdbvm\" (UID: \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" Oct 30 05:02:47.433122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.433030 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" Oct 30 05:02:47.752657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:47.752578 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm"] Oct 30 05:02:47.757398 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:02:47.757347 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e878d6_d55f_4ca9_a3d8_66e6d8379172.slice/crio-a2c32464e2f8f572f9019b902dede9157e8aca072a78048b505ef297708f9a9e WatchSource:0}: Error finding container a2c32464e2f8f572f9019b902dede9157e8aca072a78048b505ef297708f9a9e: Status 404 returned error can't find the container with id a2c32464e2f8f572f9019b902dede9157e8aca072a78048b505ef297708f9a9e Oct 30 05:02:48.841917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:48.841869 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" event={"ID":"90e878d6-d55f-4ca9-a3d8-66e6d8379172","Type":"ContainerStarted","Data":"72911799b67081faf4f99a0184ed7f064787b5a12f4f44be7d1a03ffd873cf57"} Oct 30 05:02:48.842547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:48.842519 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" event={"ID":"90e878d6-d55f-4ca9-a3d8-66e6d8379172","Type":"ContainerStarted","Data":"a2c32464e2f8f572f9019b902dede9157e8aca072a78048b505ef297708f9a9e"} Oct 30 05:02:48.973382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:48.966228 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" podStartSLOduration=1.96620672 podStartE2EDuration="1.96620672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:48.896801766 +0000 UTC m=+1093.424568564" watchObservedRunningTime="2026-10-30 05:02:48.96620672 +0000 UTC m=+1093.493973508" Oct 30 05:02:49.492392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:49.488448 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:49.492392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:49.488865 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:02:49.684385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:49.683114 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:49.825510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:49.825457 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:02:49.873249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:49.873200 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:50.034194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:50.034149 6587 generic.go:356] "Generic (PLEG): container finished" podID="90e878d6-d55f-4ca9-a3d8-66e6d8379172" containerID="72911799b67081faf4f99a0184ed7f064787b5a12f4f44be7d1a03ffd873cf57" exitCode=0 Oct 30 05:02:50.073425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:50.054408 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" event={"ID":"90e878d6-d55f-4ca9-a3d8-66e6d8379172","Type":"ContainerDied","Data":"72911799b67081faf4f99a0184ed7f064787b5a12f4f44be7d1a03ffd873cf57"} Oct 30 05:02:50.073425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:50.060444 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:50.202465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:50.198061 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:50.306956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:50.220581 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:02:50.306956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:50.243500 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:02:50.306956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:50.273823 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:50.892384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:50.876464 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:02:51.003384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:50.923914 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:51.028981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:50.924186 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:02:51.354384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:50.995129 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:02:51.354384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:50.995231 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:02:51.354384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:51.024685 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:02:51.354384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:51.336619 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:02:51.354384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:51.336697 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:02:51.354384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:51.336756 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:02:51.354384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:51.071127 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:51.354384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:51.117303 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:51.354384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:51.273963 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:51.354384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:51.274203 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:02:51.354384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:51.274759 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dmbs8" podUID="c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 05:02:51.476930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:51.476879 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:02:51.629716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:51.629621 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:02:52.023574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.003654 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:52.044385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.030902 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:02:52.052383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.041186 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:02:52.079524 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.079490 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:02:52.080403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.080381 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:02:52.385378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.379717 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:52.385378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.380075 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:02:52.385378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.380332 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:02:52.385378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.381723 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:02:52.696441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.624791 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:02:52.805383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.629441 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:02:52.826392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.629483 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:52.826996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.629506 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:02:52.968437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:52.966310 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:53.043390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:53.041034 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:02:53.082386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:53.078579 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:02:53.396384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:53.396125 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:53.420385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:53.402453 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:02:53.420385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:53.416380 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:02:53.817547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:53.817484 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:53.818730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:53.818681 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:02:53.847033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:53.846465 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:53.847033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:53.846827 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:02:53.971393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:53.961378 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl"] Oct 30 05:02:54.043570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:54.043520 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:54.202441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:54.199552 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:02:54.420623 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:54.420564 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:02:54.424377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:54.421716 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:02:54.489478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:54.454999 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:54.787438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:54.786672 6587 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:02:54.787438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:54.786779 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:54.830563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:54.825609 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:02:54.839194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:54.834083 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:54.929382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:54.927034 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:02:54.929382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:54.927409 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:02:54.930464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:54.929874 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl"] Oct 30 05:02:54.938932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:54.930942 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" Oct 30 05:02:55.071386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.070438 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.097464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.078651 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d52ecbf-wjzcl\" (UID: \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" Oct 30 05:02:55.097464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.078864 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9cnh\" (UniqueName: \"kubernetes.io/projected/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-kube-api-access-x9cnh\") pod \"maas-api-revoke-sub-7d52ecbf-wjzcl\" (UID: \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" Oct 30 05:02:55.097464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.079033 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d52ecbf-wjzcl\" (UID: \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" Oct 30 05:02:55.218141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.217494 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.233395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.220107 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d52ecbf-wjzcl\" (UID: \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" Oct 30 05:02:55.233395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.226609 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d52ecbf-wjzcl\" (UID: \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" Oct 30 05:02:55.233395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.226801 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9cnh\" (UniqueName: \"kubernetes.io/projected/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-kube-api-access-x9cnh\") pod \"maas-api-revoke-sub-7d52ecbf-wjzcl\" (UID: \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" Oct 30 05:02:55.291626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.291579 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d52ecbf-wjzcl\" (UID: \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" Oct 30 05:02:55.387817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.309800 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d52ecbf-wjzcl\" (UID: \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" Oct 30 05:02:55.437597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.434520 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.437597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.435052 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:02:55.494104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.494054 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.553388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.528124 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9cnh\" (UniqueName: \"kubernetes.io/projected/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-kube-api-access-x9cnh\") pod \"maas-api-revoke-sub-7d52ecbf-wjzcl\" (UID: \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" Oct 30 05:02:55.731679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.731577 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" Oct 30 05:02:55.839317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.839262 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.884323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.856677 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:55.944384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.944101 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:02:55.944384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:55.944350 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:02:56.101382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.080600 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:56.180401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.160491 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" Oct 30 05:02:56.219436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.218207 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:56.529389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.526544 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e878d6-d55f-4ca9-a3d8-66e6d8379172-kube-api-access-r2k24\" (UniqueName: \"kubernetes.io/projected/90e878d6-d55f-4ca9-a3d8-66e6d8379172-kube-api-access-r2k24\") pod \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\" (UID: \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\") " Oct 30 05:02:56.529389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.526749 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-openshift-ca\") pod \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\" (UID: \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\") " Oct 30 05:02:56.529389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.526856 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-xks-ca\") pod \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\" (UID: \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\") " Oct 30 05:02:56.529829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.529792 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:02:56.627552 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.627509 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-openshift-ca" pod "90e878d6-d55f-4ca9-a3d8-66e6d8379172" (UID: "90e878d6-d55f-4ca9-a3d8-66e6d8379172"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:56.627840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.627806 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:02:56.685507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.685457 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:02:56.800410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.746774 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 30 05:02:56.932388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.901588 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:56.932388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.902272 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:56.932388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.905252 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" event={"ID":"90e878d6-d55f-4ca9-a3d8-66e6d8379172","Type":"ContainerDied","Data":"a2c32464e2f8f572f9019b902dede9157e8aca072a78048b505ef297708f9a9e"} Oct 30 05:02:56.932388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.905312 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2c32464e2f8f572f9019b902dede9157e8aca072a78048b505ef297708f9a9e" Oct 30 05:02:56.932388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.905409 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm" Oct 30 05:02:56.938655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.938601 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e878d6-d55f-4ca9-a3d8-66e6d8379172-kube-api-access-r2k24" pod "90e878d6-d55f-4ca9-a3d8-66e6d8379172" (UID: "90e878d6-d55f-4ca9-a3d8-66e6d8379172"). InnerVolumeSpecName "kube-api-access-r2k24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:56.949405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.947528 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-xks-ca" pod "90e878d6-d55f-4ca9-a3d8-66e6d8379172" (UID: "90e878d6-d55f-4ca9-a3d8-66e6d8379172"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:56.949405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.947629 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:56.949405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.948029 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:02:56.954621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.954587 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-xks-ca\") pod \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\" (UID: \"90e878d6-d55f-4ca9-a3d8-66e6d8379172\") " Oct 30 05:02:56.957887 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:02:56.957823 6587 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/90e878d6-d55f-4ca9-a3d8-66e6d8379172/volumes/kubernetes.io~secret/maas-api-xks-ca Oct 30 05:02:56.957887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.957850 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-xks-ca" pod "90e878d6-d55f-4ca9-a3d8-66e6d8379172" (UID: "90e878d6-d55f-4ca9-a3d8-66e6d8379172"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:56.983674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.970546 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:56.983674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.970593 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e878d6-d55f-4ca9-a3d8-66e6d8379172-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:56.983674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:56.970607 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2k24\" (UniqueName: \"kubernetes.io/projected/90e878d6-d55f-4ca9-a3d8-66e6d8379172-kube-api-access-r2k24\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:02:57.300822 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:57.297168 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:02:57.315589 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:02:57.315547 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:02:57.409743 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:57.358665 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:57.479197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:57.479151 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:02:57.569524 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:57.560002 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:02:57.977599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:57.906812 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:58.456618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:57.950644 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:58.456618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.002921 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": context deadline exceeded" Oct 30 05:02:58.456618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.026201 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:02:58.456618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.026255 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:02:58.456618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.060494 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": context deadline exceeded" Oct 30 05:02:58.456618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.061064 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:02:58.456618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.061093 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:02:58.456618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.310433 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:02:58.496375 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.495548 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:02:58.560620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.560527 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:02:58.617251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.617221 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl"] Oct 30 05:02:58.878545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.878513 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:02:58.930009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.929981 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:02:58.969639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.959667 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:02:58.969910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.969891 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:02:58.981454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:58.981427 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:02:59.014124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:59.014097 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:02:59.042208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:59.042174 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:02:59.079594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:59.079561 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:02:59.091385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:59.090593 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:02:59.231129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:02:59.231099 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" event={"ID":"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7","Type":"ContainerStarted","Data":"d301131382cd970003bdd9bf7a1e9d67bc94007807b1954857f800ac412879af"} Oct 30 05:03:00.245510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:00.245477 6587 generic.go:356] "Generic (PLEG): container finished" podID="80d0d0ca-0c75-4e62-b22f-4eeba8b557b7" containerID="c33f3c03473c87e1075919da48e5ed994b71adcf3f40ee3620df9271b316d7f5" exitCode=0 Oct 30 05:03:00.245883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:00.245516 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" event={"ID":"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7","Type":"ContainerDied","Data":"c33f3c03473c87e1075919da48e5ed994b71adcf3f40ee3620df9271b316d7f5"} Oct 30 05:03:01.428782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:01.428754 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" Oct 30 05:03:01.543490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:01.541531 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-openshift-ca\") pod \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\" (UID: \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\") " Oct 30 05:03:01.543490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:01.541588 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-xks-ca\") pod \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\" (UID: \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\") " Oct 30 05:03:01.543490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:01.541653 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-kube-api-access-x9cnh\" (UniqueName: \"kubernetes.io/projected/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-kube-api-access-x9cnh\") pod \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\" (UID: \"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7\") " Oct 30 05:03:01.545668 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:01.544167 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-openshift-ca" pod "80d0d0ca-0c75-4e62-b22f-4eeba8b557b7" (UID: "80d0d0ca-0c75-4e62-b22f-4eeba8b557b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:03:01.548154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:01.548116 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-xks-ca" pod "80d0d0ca-0c75-4e62-b22f-4eeba8b557b7" (UID: "80d0d0ca-0c75-4e62-b22f-4eeba8b557b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:03:01.548285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:01.548218 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-kube-api-access-x9cnh" pod "80d0d0ca-0c75-4e62-b22f-4eeba8b557b7" (UID: "80d0d0ca-0c75-4e62-b22f-4eeba8b557b7"). InnerVolumeSpecName "kube-api-access-x9cnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:03:01.645259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:01.645220 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:01.645259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:01.645260 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:01.645669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:01.645285 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9cnh\" (UniqueName: \"kubernetes.io/projected/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7-kube-api-access-x9cnh\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:02.267751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:02.267723 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" Oct 30 05:03:02.267935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:02.267752 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl" event={"ID":"80d0d0ca-0c75-4e62-b22f-4eeba8b557b7","Type":"ContainerDied","Data":"d301131382cd970003bdd9bf7a1e9d67bc94007807b1954857f800ac412879af"} Oct 30 05:03:02.267935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:02.267833 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d301131382cd970003bdd9bf7a1e9d67bc94007807b1954857f800ac412879af" Oct 30 05:03:05.479682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.479652 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g"] Oct 30 05:03:05.480498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.480481 6587 cpu_manager.go:414] "containerMap: removing container" podUID="90e878d6-d55f-4ca9-a3d8-66e6d8379172" containerName="revoke-keys" Oct 30 05:03:05.480498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.480498 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e878d6-d55f-4ca9-a3d8-66e6d8379172" containerName="revoke-keys" Oct 30 05:03:05.480603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.480516 6587 cpu_manager.go:414] "containerMap: removing container" podUID="80d0d0ca-0c75-4e62-b22f-4eeba8b557b7" containerName="revoke-keys" Oct 30 05:03:05.480603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.480521 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d0d0ca-0c75-4e62-b22f-4eeba8b557b7" containerName="revoke-keys" Oct 30 05:03:05.480666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.480655 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e878d6-d55f-4ca9-a3d8-66e6d8379172" containerName="revoke-keys" Oct 30 05:03:05.480704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.480666 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d0d0ca-0c75-4e62-b22f-4eeba8b557b7" containerName="revoke-keys" Oct 30 05:03:05.496401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.496350 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" Oct 30 05:03:05.518026 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.517986 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g"] Oct 30 05:03:05.520619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.520591 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:03:05.528727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.528697 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:03:05.528960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.528944 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:03:05.529163 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.529147 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:03:05.628616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.628568 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5e5e447-sng2g\" (UID: \"7d78db49-762e-45df-a7fc-eadb81752d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" Oct 30 05:03:05.628798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.628694 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6k6w\" (UniqueName: \"kubernetes.io/projected/7d78db49-762e-45df-a7fc-eadb81752d98-kube-api-access-k6k6w\") pod \"maas-api-revoke-sub-b5e5e447-sng2g\" (UID: \"7d78db49-762e-45df-a7fc-eadb81752d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" Oct 30 05:03:05.628798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.628787 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5e5e447-sng2g\" (UID: \"7d78db49-762e-45df-a7fc-eadb81752d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" Oct 30 05:03:05.730955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.730859 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5e5e447-sng2g\" (UID: \"7d78db49-762e-45df-a7fc-eadb81752d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" Oct 30 05:03:05.731135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.730968 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6k6w\" (UniqueName: \"kubernetes.io/projected/7d78db49-762e-45df-a7fc-eadb81752d98-kube-api-access-k6k6w\") pod \"maas-api-revoke-sub-b5e5e447-sng2g\" (UID: \"7d78db49-762e-45df-a7fc-eadb81752d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" Oct 30 05:03:05.731135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.731117 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5e5e447-sng2g\" (UID: \"7d78db49-762e-45df-a7fc-eadb81752d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" Oct 30 05:03:05.731668 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.731640 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5e5e447-sng2g\" (UID: \"7d78db49-762e-45df-a7fc-eadb81752d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" Oct 30 05:03:05.733796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.733771 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5e5e447-sng2g\" (UID: \"7d78db49-762e-45df-a7fc-eadb81752d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" Oct 30 05:03:05.751284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.751255 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6k6w\" (UniqueName: \"kubernetes.io/projected/7d78db49-762e-45df-a7fc-eadb81752d98-kube-api-access-k6k6w\") pod \"maas-api-revoke-sub-b5e5e447-sng2g\" (UID: \"7d78db49-762e-45df-a7fc-eadb81752d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" Oct 30 05:03:05.819683 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:05.819647 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" Oct 30 05:03:06.126619 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:03:06.126573 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d78db49_762e_45df_a7fc_eadb81752d98.slice/crio-75d5c20dd4e594effde42ebbe52512441be56adf8f9cc7a4aceb77f0b0366cce WatchSource:0}: Error finding container 75d5c20dd4e594effde42ebbe52512441be56adf8f9cc7a4aceb77f0b0366cce: Status 404 returned error can't find the container with id 75d5c20dd4e594effde42ebbe52512441be56adf8f9cc7a4aceb77f0b0366cce Oct 30 05:03:06.129008 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:06.128980 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g"] Oct 30 05:03:06.306687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:06.306650 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" event={"ID":"7d78db49-762e-45df-a7fc-eadb81752d98","Type":"ContainerStarted","Data":"75d5c20dd4e594effde42ebbe52512441be56adf8f9cc7a4aceb77f0b0366cce"} Oct 30 05:03:07.319158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:07.319123 6587 generic.go:356] "Generic (PLEG): container finished" podID="7d78db49-762e-45df-a7fc-eadb81752d98" containerID="f5587e89959c30fdbef5ba6e515773e4dea318fb2bdf75366235a2b037e91bf2" exitCode=0 Oct 30 05:03:07.319620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:07.319202 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" event={"ID":"7d78db49-762e-45df-a7fc-eadb81752d98","Type":"ContainerDied","Data":"f5587e89959c30fdbef5ba6e515773e4dea318fb2bdf75366235a2b037e91bf2"} Oct 30 05:03:08.537767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:08.537736 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" Oct 30 05:03:08.700085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:08.699990 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d78db49-762e-45df-a7fc-eadb81752d98-kube-api-access-k6k6w\" (UniqueName: \"kubernetes.io/projected/7d78db49-762e-45df-a7fc-eadb81752d98-kube-api-access-k6k6w\") pod \"7d78db49-762e-45df-a7fc-eadb81752d98\" (UID: \"7d78db49-762e-45df-a7fc-eadb81752d98\") " Oct 30 05:03:08.700257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:08.700200 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-xks-ca\") pod \"7d78db49-762e-45df-a7fc-eadb81752d98\" (UID: \"7d78db49-762e-45df-a7fc-eadb81752d98\") " Oct 30 05:03:08.700324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:08.700310 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-openshift-ca\") pod \"7d78db49-762e-45df-a7fc-eadb81752d98\" (UID: \"7d78db49-762e-45df-a7fc-eadb81752d98\") " Oct 30 05:03:08.700678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:08.700650 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-openshift-ca" pod "7d78db49-762e-45df-a7fc-eadb81752d98" (UID: "7d78db49-762e-45df-a7fc-eadb81752d98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:03:08.701062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:08.701040 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:08.702273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:08.702249 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-xks-ca" pod "7d78db49-762e-45df-a7fc-eadb81752d98" (UID: "7d78db49-762e-45df-a7fc-eadb81752d98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:03:08.702468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:08.702415 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d78db49-762e-45df-a7fc-eadb81752d98-kube-api-access-k6k6w" pod "7d78db49-762e-45df-a7fc-eadb81752d98" (UID: "7d78db49-762e-45df-a7fc-eadb81752d98"). InnerVolumeSpecName "kube-api-access-k6k6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:03:08.802302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:08.802269 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6k6w\" (UniqueName: \"kubernetes.io/projected/7d78db49-762e-45df-a7fc-eadb81752d98-kube-api-access-k6k6w\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:08.802302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:08.802299 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d78db49-762e-45df-a7fc-eadb81752d98-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:09.339543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:09.339514 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" Oct 30 05:03:09.339816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:09.339551 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g" event={"ID":"7d78db49-762e-45df-a7fc-eadb81752d98","Type":"ContainerDied","Data":"75d5c20dd4e594effde42ebbe52512441be56adf8f9cc7a4aceb77f0b0366cce"} Oct 30 05:03:09.339926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:09.339863 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75d5c20dd4e594effde42ebbe52512441be56adf8f9cc7a4aceb77f0b0366cce" Oct 30 05:03:14.574168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.574125 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j"] Oct 30 05:03:14.575528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.575294 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7d78db49-762e-45df-a7fc-eadb81752d98" containerName="revoke-keys" Oct 30 05:03:14.575528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.575321 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d78db49-762e-45df-a7fc-eadb81752d98" containerName="revoke-keys" Oct 30 05:03:14.575756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.575534 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d78db49-762e-45df-a7fc-eadb81752d98" containerName="revoke-keys" Oct 30 05:03:14.580615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.580593 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" Oct 30 05:03:14.600101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.600043 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j"] Oct 30 05:03:14.607564 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.607539 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:03:14.607702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.607574 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:03:14.607702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.607613 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:03:14.617410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.610495 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:03:14.636623 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.636590 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2q2n\" (UniqueName: \"kubernetes.io/projected/69f66128-c09c-4032-9a79-3149b07e4812-kube-api-access-p2q2n\") pod \"maas-api-revoke-sub-49933366-l826j\" (UID: \"69f66128-c09c-4032-9a79-3149b07e4812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" Oct 30 05:03:14.636864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.636843 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49933366-l826j\" (UID: \"69f66128-c09c-4032-9a79-3149b07e4812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" Oct 30 05:03:14.637675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.637646 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49933366-l826j\" (UID: \"69f66128-c09c-4032-9a79-3149b07e4812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" Oct 30 05:03:14.740129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.740093 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2q2n\" (UniqueName: \"kubernetes.io/projected/69f66128-c09c-4032-9a79-3149b07e4812-kube-api-access-p2q2n\") pod \"maas-api-revoke-sub-49933366-l826j\" (UID: \"69f66128-c09c-4032-9a79-3149b07e4812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" Oct 30 05:03:14.740306 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.740144 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49933366-l826j\" (UID: \"69f66128-c09c-4032-9a79-3149b07e4812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" Oct 30 05:03:14.740444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.740425 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49933366-l826j\" (UID: \"69f66128-c09c-4032-9a79-3149b07e4812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" Oct 30 05:03:14.740900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.740879 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49933366-l826j\" (UID: \"69f66128-c09c-4032-9a79-3149b07e4812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" Oct 30 05:03:14.743557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.743500 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49933366-l826j\" (UID: \"69f66128-c09c-4032-9a79-3149b07e4812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" Oct 30 05:03:14.768830 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.768802 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2q2n\" (UniqueName: \"kubernetes.io/projected/69f66128-c09c-4032-9a79-3149b07e4812-kube-api-access-p2q2n\") pod \"maas-api-revoke-sub-49933366-l826j\" (UID: \"69f66128-c09c-4032-9a79-3149b07e4812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" Oct 30 05:03:14.904547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:14.904463 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" Oct 30 05:03:15.226254 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:15.226222 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j"] Oct 30 05:03:15.424863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:15.424822 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" event={"ID":"69f66128-c09c-4032-9a79-3149b07e4812","Type":"ContainerStarted","Data":"5a69a9d2ebc823d7a94a3b43671e15c770f131c079ed887e6f0ecf8ab34749a7"} Oct 30 05:03:16.450936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:16.450890 6587 generic.go:356] "Generic (PLEG): container finished" podID="69f66128-c09c-4032-9a79-3149b07e4812" containerID="04214d72b50b344b3e02f65cf9a68e43c4041a0780210b12ece763be5f93dec0" exitCode=0 Oct 30 05:03:16.451481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:16.451096 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" event={"ID":"69f66128-c09c-4032-9a79-3149b07e4812","Type":"ContainerDied","Data":"04214d72b50b344b3e02f65cf9a68e43c4041a0780210b12ece763be5f93dec0"} Oct 30 05:03:18.085377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.084348 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" Oct 30 05:03:18.199378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.195322 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-openshift-ca\") pod \"69f66128-c09c-4032-9a79-3149b07e4812\" (UID: \"69f66128-c09c-4032-9a79-3149b07e4812\") " Oct 30 05:03:18.199378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.195402 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f66128-c09c-4032-9a79-3149b07e4812-kube-api-access-p2q2n\" (UniqueName: \"kubernetes.io/projected/69f66128-c09c-4032-9a79-3149b07e4812-kube-api-access-p2q2n\") pod \"69f66128-c09c-4032-9a79-3149b07e4812\" (UID: \"69f66128-c09c-4032-9a79-3149b07e4812\") " Oct 30 05:03:18.199378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.195666 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-xks-ca\") pod \"69f66128-c09c-4032-9a79-3149b07e4812\" (UID: \"69f66128-c09c-4032-9a79-3149b07e4812\") " Oct 30 05:03:18.236476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.235490 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f66128-c09c-4032-9a79-3149b07e4812-kube-api-access-p2q2n" pod "69f66128-c09c-4032-9a79-3149b07e4812" (UID: "69f66128-c09c-4032-9a79-3149b07e4812"). InnerVolumeSpecName "kube-api-access-p2q2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:03:18.238561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.236798 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-openshift-ca" pod "69f66128-c09c-4032-9a79-3149b07e4812" (UID: "69f66128-c09c-4032-9a79-3149b07e4812"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:03:18.304735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.304678 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-xks-ca" pod "69f66128-c09c-4032-9a79-3149b07e4812" (UID: "69f66128-c09c-4032-9a79-3149b07e4812"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:03:18.328561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.325698 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:18.328561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.325735 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f66128-c09c-4032-9a79-3149b07e4812-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:18.328561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.325750 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2q2n\" (UniqueName: \"kubernetes.io/projected/69f66128-c09c-4032-9a79-3149b07e4812-kube-api-access-p2q2n\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:18.637379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.627030 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf"] Oct 30 05:03:18.637379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.628325 6587 cpu_manager.go:414] "containerMap: removing container" podUID="69f66128-c09c-4032-9a79-3149b07e4812" containerName="revoke-keys" Oct 30 05:03:18.637379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.628348 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f66128-c09c-4032-9a79-3149b07e4812" containerName="revoke-keys" Oct 30 05:03:18.637379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.628583 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f66128-c09c-4032-9a79-3149b07e4812" containerName="revoke-keys" Oct 30 05:03:18.705385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.689815 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" Oct 30 05:03:18.766582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.764502 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wsq8\" (UniqueName: \"kubernetes.io/projected/afb90856-8a71-4042-a36b-d0d3d6529ad2-kube-api-access-5wsq8\") pod \"maas-api-revoke-sub-1cf2229c-kzbtf\" (UID: \"afb90856-8a71-4042-a36b-d0d3d6529ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" Oct 30 05:03:18.766582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.764613 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf2229c-kzbtf\" (UID: \"afb90856-8a71-4042-a36b-d0d3d6529ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" Oct 30 05:03:18.766582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.764781 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf2229c-kzbtf\" (UID: \"afb90856-8a71-4042-a36b-d0d3d6529ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" Oct 30 05:03:18.849851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.849810 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" event={"ID":"69f66128-c09c-4032-9a79-3149b07e4812","Type":"ContainerDied","Data":"5a69a9d2ebc823d7a94a3b43671e15c770f131c079ed887e6f0ecf8ab34749a7"} Oct 30 05:03:18.873582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.853427 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a69a9d2ebc823d7a94a3b43671e15c770f131c079ed887e6f0ecf8ab34749a7" Oct 30 05:03:18.889756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.860048 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j" Oct 30 05:03:18.891877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.875100 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf2229c-kzbtf\" (UID: \"afb90856-8a71-4042-a36b-d0d3d6529ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" Oct 30 05:03:18.892176 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.875983 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf2229c-kzbtf\" (UID: \"afb90856-8a71-4042-a36b-d0d3d6529ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" Oct 30 05:03:18.892974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.892500 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf2229c-kzbtf\" (UID: \"afb90856-8a71-4042-a36b-d0d3d6529ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" Oct 30 05:03:18.893665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.893632 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wsq8\" (UniqueName: \"kubernetes.io/projected/afb90856-8a71-4042-a36b-d0d3d6529ad2-kube-api-access-5wsq8\") pod \"maas-api-revoke-sub-1cf2229c-kzbtf\" (UID: \"afb90856-8a71-4042-a36b-d0d3d6529ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" Oct 30 05:03:18.966385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.948226 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf"] Oct 30 05:03:18.978757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:18.978005 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wsq8\" (UniqueName: \"kubernetes.io/projected/afb90856-8a71-4042-a36b-d0d3d6529ad2-kube-api-access-5wsq8\") pod \"maas-api-revoke-sub-1cf2229c-kzbtf\" (UID: \"afb90856-8a71-4042-a36b-d0d3d6529ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" Oct 30 05:03:19.088791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:19.088743 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf2229c-kzbtf\" (UID: \"afb90856-8a71-4042-a36b-d0d3d6529ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" Oct 30 05:03:19.331093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:19.329659 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" Oct 30 05:03:19.497349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:19.497303 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:19.513046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:19.512995 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:19.841724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:19.840709 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:19.888765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:19.888470 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:19.927593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:19.927532 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:03:20.055026 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.054405 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:20.186440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.177039 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:20.392383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.390899 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:20.404956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.396799 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:20.439435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.439051 6587 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:03:20.439435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.439118 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:20.516380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.515667 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:03:20.545808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.533138 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:03:20.545808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.534877 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:03:20.545808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.534939 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:03:20.803447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.800138 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:03:20.923030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.894076 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:20.923030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.914929 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:03:20.923030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.918874 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:03:20.923030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:20.918934 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:03:21.080996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:21.058381 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:21.080996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:21.059064 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:21.080996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:21.059273 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:21.080996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:21.059635 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:03:21.406431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:21.401499 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:21.463542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:21.463489 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:03:21.465138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:21.465109 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:03:21.465399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:21.465382 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:03:21.533497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:21.533446 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:03:21.935416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:21.922284 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:03:21.958940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:21.947750 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:22.019389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:22.010802 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:22.106892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:22.099449 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:22.488443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:22.484041 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:03:22.488443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:22.485402 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:22.490120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:22.490066 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:22.526691 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:22.526647 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:22.612390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:22.610832 6587 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:03:22.612693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:22.612655 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" podUID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:22.713693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:22.713581 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:22.929568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:22.929512 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:23.091158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.091107 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:03:23.102828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.091554 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/health\": context deadline exceeded" Oct 30 05:03:23.103093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.091658 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:23.103244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.102420 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:03:23.310419 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:03:23.290382 6587 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.456s" Oct 30 05:03:23.310419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.290466 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:03:23.310419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.290573 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:03:23.310419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.290609 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:03:23.329017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.328982 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:03:23.430629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.430542 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:03:23.469737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.469665 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:03:23.518139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.514980 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:03:23.518139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.515045 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:03:23.518139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.515073 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:03:23.518139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.515095 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:03:23.524009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.522350 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf"] Oct 30 05:03:23.524009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.522422 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" event={"ID":"afb90856-8a71-4042-a36b-d0d3d6529ad2","Type":"ContainerStarted","Data":"cf48e5169438a9093da8b31687a9af752ffa6576f31d7e1f4b3f31692c61f4b9"} Oct 30 05:03:23.533982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.533946 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:23.534612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.534588 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:03:23.567115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:23.567084 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:03:24.534429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:24.534389 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" event={"ID":"afb90856-8a71-4042-a36b-d0d3d6529ad2","Type":"ContainerStarted","Data":"3086cb06eb8733851631394bc61088e3787eb3d2c44548d229b68caf567e99f4"} Oct 30 05:03:25.545969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:25.545934 6587 generic.go:356] "Generic (PLEG): container finished" podID="afb90856-8a71-4042-a36b-d0d3d6529ad2" containerID="3086cb06eb8733851631394bc61088e3787eb3d2c44548d229b68caf567e99f4" exitCode=0 Oct 30 05:03:25.546564 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:25.546023 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" event={"ID":"afb90856-8a71-4042-a36b-d0d3d6529ad2","Type":"ContainerDied","Data":"3086cb06eb8733851631394bc61088e3787eb3d2c44548d229b68caf567e99f4"} Oct 30 05:03:25.735142 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:25.735116 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" Oct 30 05:03:25.808280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:25.808235 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb90856-8a71-4042-a36b-d0d3d6529ad2-kube-api-access-5wsq8\" (UniqueName: \"kubernetes.io/projected/afb90856-8a71-4042-a36b-d0d3d6529ad2-kube-api-access-5wsq8\") pod \"afb90856-8a71-4042-a36b-d0d3d6529ad2\" (UID: \"afb90856-8a71-4042-a36b-d0d3d6529ad2\") " Oct 30 05:03:25.808486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:25.808384 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-openshift-ca\") pod \"afb90856-8a71-4042-a36b-d0d3d6529ad2\" (UID: \"afb90856-8a71-4042-a36b-d0d3d6529ad2\") " Oct 30 05:03:25.808486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:25.808420 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-xks-ca\") pod \"afb90856-8a71-4042-a36b-d0d3d6529ad2\" (UID: \"afb90856-8a71-4042-a36b-d0d3d6529ad2\") " Oct 30 05:03:25.808920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:25.808894 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-openshift-ca" pod "afb90856-8a71-4042-a36b-d0d3d6529ad2" (UID: "afb90856-8a71-4042-a36b-d0d3d6529ad2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:03:25.809150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:25.809125 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:25.811005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:25.810975 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb90856-8a71-4042-a36b-d0d3d6529ad2-kube-api-access-5wsq8" pod "afb90856-8a71-4042-a36b-d0d3d6529ad2" (UID: "afb90856-8a71-4042-a36b-d0d3d6529ad2"). InnerVolumeSpecName "kube-api-access-5wsq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:03:25.819518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:25.819488 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-xks-ca" pod "afb90856-8a71-4042-a36b-d0d3d6529ad2" (UID: "afb90856-8a71-4042-a36b-d0d3d6529ad2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:03:25.910539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:25.910502 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wsq8\" (UniqueName: \"kubernetes.io/projected/afb90856-8a71-4042-a36b-d0d3d6529ad2-kube-api-access-5wsq8\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:25.910539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:25.910534 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb90856-8a71-4042-a36b-d0d3d6529ad2-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:25.990343 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:03:25.990306 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:03:26.556824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:26.556791 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" event={"ID":"afb90856-8a71-4042-a36b-d0d3d6529ad2","Type":"ContainerDied","Data":"cf48e5169438a9093da8b31687a9af752ffa6576f31d7e1f4b3f31692c61f4b9"} Oct 30 05:03:26.557304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:26.556842 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf48e5169438a9093da8b31687a9af752ffa6576f31d7e1f4b3f31692c61f4b9" Oct 30 05:03:26.557304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:26.556877 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf" Oct 30 05:03:28.911450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:28.911414 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x"] Oct 30 05:03:28.912191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:28.912175 6587 cpu_manager.go:414] "containerMap: removing container" podUID="afb90856-8a71-4042-a36b-d0d3d6529ad2" containerName="revoke-keys" Oct 30 05:03:28.912234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:28.912192 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb90856-8a71-4042-a36b-d0d3d6529ad2" containerName="revoke-keys" Oct 30 05:03:28.912388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:28.912331 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb90856-8a71-4042-a36b-d0d3d6529ad2" containerName="revoke-keys" Oct 30 05:03:28.916852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:28.916829 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" Oct 30 05:03:28.933482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:28.932977 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:03:28.940346 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:28.940318 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x"] Oct 30 05:03:28.941136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:28.941103 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:03:28.985007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:28.984972 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8a28de1-jtf8x\" (UID: \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" Oct 30 05:03:28.985160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:28.985118 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8a28de1-jtf8x\" (UID: \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" Oct 30 05:03:28.985391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:28.985349 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcfjn\" (UniqueName: \"kubernetes.io/projected/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-kube-api-access-pcfjn\") pod \"maas-api-revoke-sub-d8a28de1-jtf8x\" (UID: \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" Oct 30 05:03:28.985792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:28.985775 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:03:28.985934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:28.985916 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:03:29.087182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:29.087147 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8a28de1-jtf8x\" (UID: \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" Oct 30 05:03:29.087352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:29.087312 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcfjn\" (UniqueName: \"kubernetes.io/projected/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-kube-api-access-pcfjn\") pod \"maas-api-revoke-sub-d8a28de1-jtf8x\" (UID: \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" Oct 30 05:03:29.087444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:29.087431 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8a28de1-jtf8x\" (UID: \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" Oct 30 05:03:29.087969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:29.087951 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8a28de1-jtf8x\" (UID: \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" Oct 30 05:03:29.089848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:29.089829 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8a28de1-jtf8x\" (UID: \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" Oct 30 05:03:29.107671 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:29.107644 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcfjn\" (UniqueName: \"kubernetes.io/projected/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-kube-api-access-pcfjn\") pod \"maas-api-revoke-sub-d8a28de1-jtf8x\" (UID: \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" Oct 30 05:03:29.235534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:29.235429 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" Oct 30 05:03:29.566494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:29.565386 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x"] Oct 30 05:03:29.566494 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:03:29.565580 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a048a1c_93c6_4e4d_a511_6e9378eb65ab.slice/crio-fa04f6a301cbaded6ce842fcfb15d8793bc2f6ba4cb9809e44870843f575e039 WatchSource:0}: Error finding container fa04f6a301cbaded6ce842fcfb15d8793bc2f6ba4cb9809e44870843f575e039: Status 404 returned error can't find the container with id fa04f6a301cbaded6ce842fcfb15d8793bc2f6ba4cb9809e44870843f575e039 Oct 30 05:03:29.589859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:29.589822 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" event={"ID":"4a048a1c-93c6-4e4d-a511-6e9378eb65ab","Type":"ContainerStarted","Data":"fa04f6a301cbaded6ce842fcfb15d8793bc2f6ba4cb9809e44870843f575e039"} Oct 30 05:03:30.600256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:30.600220 6587 generic.go:356] "Generic (PLEG): container finished" podID="4a048a1c-93c6-4e4d-a511-6e9378eb65ab" containerID="923e175705a6e8baaaa5542115457ef09ec110673db3086114ff14cb3791d40a" exitCode=0 Oct 30 05:03:30.600704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:30.600305 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" event={"ID":"4a048a1c-93c6-4e4d-a511-6e9378eb65ab","Type":"ContainerDied","Data":"923e175705a6e8baaaa5542115457ef09ec110673db3086114ff14cb3791d40a"} Oct 30 05:03:31.802572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:31.802545 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" Oct 30 05:03:31.940122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:31.940034 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-xks-ca\") pod \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\" (UID: \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\") " Oct 30 05:03:31.940320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:31.940281 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-kube-api-access-pcfjn\" (UniqueName: \"kubernetes.io/projected/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-kube-api-access-pcfjn\") pod \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\" (UID: \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\") " Oct 30 05:03:31.940542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:31.940473 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-openshift-ca\") pod \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\" (UID: \"4a048a1c-93c6-4e4d-a511-6e9378eb65ab\") " Oct 30 05:03:31.940911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:31.940880 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-openshift-ca" pod "4a048a1c-93c6-4e4d-a511-6e9378eb65ab" (UID: "4a048a1c-93c6-4e4d-a511-6e9378eb65ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:03:31.941150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:31.941129 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:31.942305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:31.942280 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-xks-ca" pod "4a048a1c-93c6-4e4d-a511-6e9378eb65ab" (UID: "4a048a1c-93c6-4e4d-a511-6e9378eb65ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:03:31.942828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:31.942805 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-kube-api-access-pcfjn" pod "4a048a1c-93c6-4e4d-a511-6e9378eb65ab" (UID: "4a048a1c-93c6-4e4d-a511-6e9378eb65ab"). InnerVolumeSpecName "kube-api-access-pcfjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:03:32.042491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:32.042458 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcfjn\" (UniqueName: \"kubernetes.io/projected/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-kube-api-access-pcfjn\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:32.042491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:32.042488 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a048a1c-93c6-4e4d-a511-6e9378eb65ab-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:32.619705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:32.619674 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" Oct 30 05:03:32.619705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:32.619692 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x" event={"ID":"4a048a1c-93c6-4e4d-a511-6e9378eb65ab","Type":"ContainerDied","Data":"fa04f6a301cbaded6ce842fcfb15d8793bc2f6ba4cb9809e44870843f575e039"} Oct 30 05:03:32.619936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:32.619745 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa04f6a301cbaded6ce842fcfb15d8793bc2f6ba4cb9809e44870843f575e039" Oct 30 05:03:33.035699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:33.035602 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94"] Oct 30 05:03:33.042958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:33.042925 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e188d6-9mp94"] Oct 30 05:03:33.586533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:33.586500 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad" path="/var/lib/kubelet/pods/f7fb8719-3b2a-40ce-a0f0-0c437b5d13ad/volumes" Oct 30 05:03:35.254587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.254546 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4"] Oct 30 05:03:35.255685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.255664 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4a048a1c-93c6-4e4d-a511-6e9378eb65ab" containerName="revoke-keys" Oct 30 05:03:35.255756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.255688 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a048a1c-93c6-4e4d-a511-6e9378eb65ab" containerName="revoke-keys" Oct 30 05:03:35.255941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.255928 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a048a1c-93c6-4e4d-a511-6e9378eb65ab" containerName="revoke-keys" Oct 30 05:03:35.261765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.261738 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" Oct 30 05:03:35.282658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.282631 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:03:35.283676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.283567 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:03:35.285194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.283626 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:03:35.287481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.287459 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4"] Oct 30 05:03:35.292766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.292749 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:03:35.397991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.397951 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcc8e386-npzm4\" (UID: \"f0a6b587-e673-4794-8347-8477bf0024cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" Oct 30 05:03:35.398194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.398096 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcc8e386-npzm4\" (UID: \"f0a6b587-e673-4794-8347-8477bf0024cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" Oct 30 05:03:35.398266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.398185 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwthd\" (UniqueName: \"kubernetes.io/projected/f0a6b587-e673-4794-8347-8477bf0024cc-kube-api-access-xwthd\") pod \"maas-api-revoke-sub-fcc8e386-npzm4\" (UID: \"f0a6b587-e673-4794-8347-8477bf0024cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" Oct 30 05:03:35.499591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.499558 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcc8e386-npzm4\" (UID: \"f0a6b587-e673-4794-8347-8477bf0024cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" Oct 30 05:03:35.499775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.499674 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcc8e386-npzm4\" (UID: \"f0a6b587-e673-4794-8347-8477bf0024cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" Oct 30 05:03:35.499775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.499698 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwthd\" (UniqueName: \"kubernetes.io/projected/f0a6b587-e673-4794-8347-8477bf0024cc-kube-api-access-xwthd\") pod \"maas-api-revoke-sub-fcc8e386-npzm4\" (UID: \"f0a6b587-e673-4794-8347-8477bf0024cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" Oct 30 05:03:35.500241 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.500214 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcc8e386-npzm4\" (UID: \"f0a6b587-e673-4794-8347-8477bf0024cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" Oct 30 05:03:35.502114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.502089 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcc8e386-npzm4\" (UID: \"f0a6b587-e673-4794-8347-8477bf0024cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" Oct 30 05:03:35.521896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.521834 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwthd\" (UniqueName: \"kubernetes.io/projected/f0a6b587-e673-4794-8347-8477bf0024cc-kube-api-access-xwthd\") pod \"maas-api-revoke-sub-fcc8e386-npzm4\" (UID: \"f0a6b587-e673-4794-8347-8477bf0024cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" Oct 30 05:03:35.605975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.605947 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:03:35.611245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.611224 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" Oct 30 05:03:35.953026 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:35.952994 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4"] Oct 30 05:03:36.669446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:36.667615 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" event={"ID":"f0a6b587-e673-4794-8347-8477bf0024cc","Type":"ContainerStarted","Data":"302e10fb64357845c3ffc1f54b7551171cc4432f9d73932ebf5ac473425cd0c3"} Oct 30 05:03:36.669446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:36.667685 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" event={"ID":"f0a6b587-e673-4794-8347-8477bf0024cc","Type":"ContainerStarted","Data":"6b778e67d40685f88b32a75712a2476c61a4953a809f87676569dbc29011c4ce"} Oct 30 05:03:36.746381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:36.745701 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" podStartSLOduration=1.745677014 podStartE2EDuration="1.745677014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:36.711569836 +0000 UTC m=+1141.239336623" watchObservedRunningTime="2026-10-30 05:03:36.745677014 +0000 UTC m=+1141.273443805" Oct 30 05:03:37.781380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:37.776116 6587 generic.go:356] "Generic (PLEG): container finished" podID="f0a6b587-e673-4794-8347-8477bf0024cc" containerID="302e10fb64357845c3ffc1f54b7551171cc4432f9d73932ebf5ac473425cd0c3" exitCode=0 Oct 30 05:03:37.781380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:37.776337 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" event={"ID":"f0a6b587-e673-4794-8347-8477bf0024cc","Type":"ContainerDied","Data":"302e10fb64357845c3ffc1f54b7551171cc4432f9d73932ebf5ac473425cd0c3"} Oct 30 05:03:38.136577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:38.136519 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:38.329465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:38.321861 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:03:38.510472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:38.501587 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:38.594575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:38.594522 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:38.813392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:38.810099 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:03:39.210084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:39.209970 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:03:39.422738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:39.212619 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:03:39.510081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:39.509979 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:39.511686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:39.510781 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:03:39.511686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:39.511417 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:03:39.511686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:39.511614 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 05:03:39.540592 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:39.534497 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:39.618503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:39.605554 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:39.618503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:39.607342 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:03:39.705585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:39.705519 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:39.898400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:39.864738 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:40.180432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.180247 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:40.283472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.283425 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:40.292383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.285180 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": context deadline exceeded" Oct 30 05:03:40.387401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.339261 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m"] Oct 30 05:03:40.387401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.385028 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7d98dd-wsj4m"] Oct 30 05:03:40.387401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.385100 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:03:40.506573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.506446 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:03:40.513951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.513907 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:40.514532 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.514501 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:40.519388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.515341 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:03:40.593542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.540536 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:40.611384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.607604 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:40.688384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.613755 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:03:40.701160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.620057 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:40.875371 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:40.875310 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.182460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.159625 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.183115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.183065 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.183819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.183800 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:03:41.289062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.289017 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.548490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.530048 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.548490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.530622 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:03:41.579387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.573145 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.609221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.609169 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:03:41.685469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.679565 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" Oct 30 05:03:41.685469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.679996 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:03:41.707705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.707594 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.847444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.838448 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a6b587-e673-4794-8347-8477bf0024cc-kube-api-access-xwthd\" (UniqueName: \"kubernetes.io/projected/f0a6b587-e673-4794-8347-8477bf0024cc-kube-api-access-xwthd\") pod \"f0a6b587-e673-4794-8347-8477bf0024cc\" (UID: \"f0a6b587-e673-4794-8347-8477bf0024cc\") " Oct 30 05:03:41.847444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.838687 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-xks-ca\") pod \"f0a6b587-e673-4794-8347-8477bf0024cc\" (UID: \"f0a6b587-e673-4794-8347-8477bf0024cc\") " Oct 30 05:03:41.847444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.838763 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-openshift-ca\") pod \"f0a6b587-e673-4794-8347-8477bf0024cc\" (UID: \"f0a6b587-e673-4794-8347-8477bf0024cc\") " Oct 30 05:03:41.847444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.839100 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.881381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.876449 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:41.895716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.895660 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-openshift-ca" pod "f0a6b587-e673-4794-8347-8477bf0024cc" (UID: "f0a6b587-e673-4794-8347-8477bf0024cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:03:41.964018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.963975 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-xks-ca" pod "f0a6b587-e673-4794-8347-8477bf0024cc" (UID: "f0a6b587-e673-4794-8347-8477bf0024cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:03:41.994383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:41.989851 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae14d57-3dea-4327-a1de-28239d35fae6" path="/var/lib/kubelet/pods/dae14d57-3dea-4327-a1de-28239d35fae6/volumes" Oct 30 05:03:42.018002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.017900 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:42.018002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.017949 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a6b587-e673-4794-8347-8477bf0024cc-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:42.021724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.021695 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:03:42.180256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.180164 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a6b587-e673-4794-8347-8477bf0024cc-kube-api-access-xwthd" pod "f0a6b587-e673-4794-8347-8477bf0024cc" (UID: "f0a6b587-e673-4794-8347-8477bf0024cc"). InnerVolumeSpecName "kube-api-access-xwthd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:03:42.256250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.224556 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:42.311819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.311776 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwthd\" (UniqueName: \"kubernetes.io/projected/f0a6b587-e673-4794-8347-8477bf0024cc-kube-api-access-xwthd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:03:42.312115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.312082 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:03:42.646714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.645455 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:03:42.650448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.650407 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:03:42.683382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.661346 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:03:42.694449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.683562 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:42.779389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.763959 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:42.869694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.869645 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:42.886633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.886590 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:43.019442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:42.896801 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:43.214378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.209237 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" event={"ID":"f0a6b587-e673-4794-8347-8477bf0024cc","Type":"ContainerDied","Data":"6b778e67d40685f88b32a75712a2476c61a4953a809f87676569dbc29011c4ce"} Oct 30 05:03:43.214378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.209401 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b778e67d40685f88b32a75712a2476c61a4953a809f87676569dbc29011c4ce" Oct 30 05:03:43.246841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.246739 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4" Oct 30 05:03:43.289449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.281340 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:03:43.291481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.290579 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:43.317569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.317523 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:03:43.372472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.356438 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r"] Oct 30 05:03:43.400151 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.400118 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:03:43.603235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.603203 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:03:43.668761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.668705 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:03:43.737383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.712605 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:03:43.793253 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.793209 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:43.880182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.880077 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:03:44.020807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:43.958153 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5000d16-h458r"] Oct 30 05:03:44.021144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:44.021125 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:03:44.093549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:44.093496 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:44.320022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:44.319954 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:44.482379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:44.452847 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:03:44.539218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:44.539185 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr"] Oct 30 05:03:44.658900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:44.658813 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:03:44.797534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:44.797487 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:44.854015 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:44.853983 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:03:44.967419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:44.967306 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c980cf8-kjwnr"] Oct 30 05:03:45.022725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:45.022668 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:45.227435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:45.215219 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:03:45.456652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:45.456608 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:03:45.616400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:45.614294 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:45.760381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:45.759477 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:03:45.874151 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:45.874066 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1968b9e2-7abc-44c6-991a-a3f7e64853d6" path="/var/lib/kubelet/pods/1968b9e2-7abc-44c6-991a-a3f7e64853d6/volumes" Oct 30 05:03:45.882425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:45.876261 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3193d88-8b5c-4fad-9873-aca7e7ec72ab" path="/var/lib/kubelet/pods/c3193d88-8b5c-4fad-9873-aca7e7ec72ab/volumes" Oct 30 05:03:46.060866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:45.882025 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:46.061060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:46.024728 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:03:46.089422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:46.089390 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:03:46.108944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:46.108906 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm"] Oct 30 05:03:46.113776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:46.113746 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7486d1ca-7clsm"] Oct 30 05:03:47.584713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:47.584678 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5" path="/var/lib/kubelet/pods/cfc29a3f-d5e3-4b0b-bf3b-e7f83c5b32b5/volumes" Oct 30 05:03:51.031861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:51.031826 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz"] Oct 30 05:03:51.048462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:51.048410 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt"] Oct 30 05:03:51.056209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:51.056182 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31ddb39e-x7bcz"] Oct 30 05:03:51.061038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:51.061013 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a199ab49-bhzwt"] Oct 30 05:03:51.584192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:51.584154 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="030b0117-0d9a-4370-8a1c-40bd335543c7" path="/var/lib/kubelet/pods/030b0117-0d9a-4370-8a1c-40bd335543c7/volumes" Oct 30 05:03:51.584719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:51.584703 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25ac0aa-86c0-4e22-aff5-490a57f95397" path="/var/lib/kubelet/pods/b25ac0aa-86c0-4e22-aff5-490a57f95397/volumes" Oct 30 05:03:55.999299 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:03:55.999257 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:03:56.142784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:56.142747 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8"] Oct 30 05:03:56.179649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:56.170300 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1f5fb4-hk2l8"] Oct 30 05:03:57.586145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:57.586097 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c331d5-051b-4a00-ae7f-e9ad577dcaeb" path="/var/lib/kubelet/pods/c0c331d5-051b-4a00-ae7f-e9ad577dcaeb/volumes" Oct 30 05:03:58.545939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.545898 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s"] Oct 30 05:03:58.547064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.547037 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f0a6b587-e673-4794-8347-8477bf0024cc" containerName="revoke-keys" Oct 30 05:03:58.547152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.547067 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a6b587-e673-4794-8347-8477bf0024cc" containerName="revoke-keys" Oct 30 05:03:58.547310 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.547297 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a6b587-e673-4794-8347-8477bf0024cc" containerName="revoke-keys" Oct 30 05:03:58.552398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.552375 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" Oct 30 05:03:58.565974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.565946 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s"] Oct 30 05:03:58.568343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.568320 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:03:58.572224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.572203 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:03:58.572502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.572340 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:03:58.581964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.581945 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:03:58.658207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.658164 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d67de78b-66d7s\" (UID: \"e66b9449-cffe-4534-b23a-92ec73daef4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" Oct 30 05:03:58.658891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.658577 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtdxh\" (UniqueName: \"kubernetes.io/projected/e66b9449-cffe-4534-b23a-92ec73daef4c-kube-api-access-xtdxh\") pod \"maas-api-revoke-keys-e2e-ait-d67de78b-66d7s\" (UID: \"e66b9449-cffe-4534-b23a-92ec73daef4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" Oct 30 05:03:58.658891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.658720 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d67de78b-66d7s\" (UID: \"e66b9449-cffe-4534-b23a-92ec73daef4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" Oct 30 05:03:58.760095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.760058 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d67de78b-66d7s\" (UID: \"e66b9449-cffe-4534-b23a-92ec73daef4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" Oct 30 05:03:58.760325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.760304 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d67de78b-66d7s\" (UID: \"e66b9449-cffe-4534-b23a-92ec73daef4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" Oct 30 05:03:58.760500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.760482 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtdxh\" (UniqueName: \"kubernetes.io/projected/e66b9449-cffe-4534-b23a-92ec73daef4c-kube-api-access-xtdxh\") pod \"maas-api-revoke-keys-e2e-ait-d67de78b-66d7s\" (UID: \"e66b9449-cffe-4534-b23a-92ec73daef4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" Oct 30 05:03:58.760960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.760941 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d67de78b-66d7s\" (UID: \"e66b9449-cffe-4534-b23a-92ec73daef4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" Oct 30 05:03:58.762677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.762651 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d67de78b-66d7s\" (UID: \"e66b9449-cffe-4534-b23a-92ec73daef4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" Oct 30 05:03:58.780969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.780946 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtdxh\" (UniqueName: \"kubernetes.io/projected/e66b9449-cffe-4534-b23a-92ec73daef4c-kube-api-access-xtdxh\") pod \"maas-api-revoke-keys-e2e-ait-d67de78b-66d7s\" (UID: \"e66b9449-cffe-4534-b23a-92ec73daef4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" Oct 30 05:03:58.871570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:58.871539 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" Oct 30 05:03:59.186861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:03:59.185467 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s"] Oct 30 05:04:00.018666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:00.018631 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" event={"ID":"e66b9449-cffe-4534-b23a-92ec73daef4c","Type":"ContainerStarted","Data":"fab138b415cd5d6c9f171412b4d0cecddfc85b03561cf0e5a2f2c65425ecf698"} Oct 30 05:04:00.019094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:00.018695 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" event={"ID":"e66b9449-cffe-4534-b23a-92ec73daef4c","Type":"ContainerStarted","Data":"369c3719e8c8dbfb0b6335ed8a92e728c09fac8cc8d1e0d39ee41535f4ac7b16"} Oct 30 05:04:00.039394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:00.039347 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz"] Oct 30 05:04:00.053246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:00.053214 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015cf7e4-6pcrz"] Oct 30 05:04:00.073484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:00.073432 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" podStartSLOduration=2.073418431 podStartE2EDuration="2.073418431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:00.070062953 +0000 UTC m=+1164.597829739" watchObservedRunningTime="2026-10-30 05:04:00.073418431 +0000 UTC m=+1164.601185216" Oct 30 05:04:01.032421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:01.032345 6587 generic.go:356] "Generic (PLEG): container finished" podID="e66b9449-cffe-4534-b23a-92ec73daef4c" containerID="fab138b415cd5d6c9f171412b4d0cecddfc85b03561cf0e5a2f2c65425ecf698" exitCode=0 Oct 30 05:04:01.032884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:01.032465 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" event={"ID":"e66b9449-cffe-4534-b23a-92ec73daef4c","Type":"ContainerDied","Data":"fab138b415cd5d6c9f171412b4d0cecddfc85b03561cf0e5a2f2c65425ecf698"} Oct 30 05:04:01.034597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:01.034573 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6"] Oct 30 05:04:01.046414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:01.045474 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe807f85-54hq6"] Oct 30 05:04:01.584861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:01.584825 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e78fe40-2571-4741-a15a-739200d2d0fd" path="/var/lib/kubelet/pods/0e78fe40-2571-4741-a15a-739200d2d0fd/volumes" Oct 30 05:04:01.585502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:01.585483 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d517d5-ed40-4bcf-9125-5a012c5e94ee" path="/var/lib/kubelet/pods/44d517d5-ed40-4bcf-9125-5a012c5e94ee/volumes" Oct 30 05:04:02.270583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:02.270554 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" Oct 30 05:04:02.369578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:02.369536 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66b9449-cffe-4534-b23a-92ec73daef4c-kube-api-access-xtdxh\" (UniqueName: \"kubernetes.io/projected/e66b9449-cffe-4534-b23a-92ec73daef4c-kube-api-access-xtdxh\") pod \"e66b9449-cffe-4534-b23a-92ec73daef4c\" (UID: \"e66b9449-cffe-4534-b23a-92ec73daef4c\") " Oct 30 05:04:02.369759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:02.369717 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-openshift-ca\") pod \"e66b9449-cffe-4534-b23a-92ec73daef4c\" (UID: \"e66b9449-cffe-4534-b23a-92ec73daef4c\") " Oct 30 05:04:02.370072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:02.370044 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-openshift-ca" pod "e66b9449-cffe-4534-b23a-92ec73daef4c" (UID: "e66b9449-cffe-4534-b23a-92ec73daef4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:04:02.370213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:02.370195 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-xks-ca\") pod \"e66b9449-cffe-4534-b23a-92ec73daef4c\" (UID: \"e66b9449-cffe-4534-b23a-92ec73daef4c\") " Oct 30 05:04:02.371083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:02.371061 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:02.372479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:02.372446 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66b9449-cffe-4534-b23a-92ec73daef4c-kube-api-access-xtdxh" pod "e66b9449-cffe-4534-b23a-92ec73daef4c" (UID: "e66b9449-cffe-4534-b23a-92ec73daef4c"). InnerVolumeSpecName "kube-api-access-xtdxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:04:02.372723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:02.372690 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-xks-ca" pod "e66b9449-cffe-4534-b23a-92ec73daef4c" (UID: "e66b9449-cffe-4534-b23a-92ec73daef4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:04:02.472830 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:02.472796 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66b9449-cffe-4534-b23a-92ec73daef4c-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:02.472830 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:02.472825 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtdxh\" (UniqueName: \"kubernetes.io/projected/e66b9449-cffe-4534-b23a-92ec73daef4c-kube-api-access-xtdxh\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:03.058767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:03.058726 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" Oct 30 05:04:03.058971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:03.058770 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s" event={"ID":"e66b9449-cffe-4534-b23a-92ec73daef4c","Type":"ContainerDied","Data":"369c3719e8c8dbfb0b6335ed8a92e728c09fac8cc8d1e0d39ee41535f4ac7b16"} Oct 30 05:04:03.058971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:03.058830 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="369c3719e8c8dbfb0b6335ed8a92e728c09fac8cc8d1e0d39ee41535f4ac7b16" Oct 30 05:04:06.030121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.030080 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc"] Oct 30 05:04:06.054381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.045795 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-409d4ef0-zzptc"] Oct 30 05:04:06.110380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.101269 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq"] Oct 30 05:04:06.110380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.102414 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e66b9449-cffe-4534-b23a-92ec73daef4c" containerName="revoke-keys" Oct 30 05:04:06.110380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.102436 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66b9449-cffe-4534-b23a-92ec73daef4c" containerName="revoke-keys" Oct 30 05:04:06.124381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.117492 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66b9449-cffe-4534-b23a-92ec73daef4c" containerName="revoke-keys" Oct 30 05:04:06.124381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.123773 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" Oct 30 05:04:06.156392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.153303 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:04:06.156392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.154058 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:04:06.156392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.154149 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:04:06.156392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.154241 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:04:06.190378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.188554 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq"] Oct 30 05:04:06.310108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.310069 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq\" (UID: \"0be3253c-4570-425f-ac1d-fc27c92f8934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" Oct 30 05:04:06.310715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.310683 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vttdq\" (UniqueName: \"kubernetes.io/projected/0be3253c-4570-425f-ac1d-fc27c92f8934-kube-api-access-vttdq\") pod \"maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq\" (UID: \"0be3253c-4570-425f-ac1d-fc27c92f8934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" Oct 30 05:04:06.310901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.310878 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq\" (UID: \"0be3253c-4570-425f-ac1d-fc27c92f8934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" Oct 30 05:04:06.434376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.431490 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vttdq\" (UniqueName: \"kubernetes.io/projected/0be3253c-4570-425f-ac1d-fc27c92f8934-kube-api-access-vttdq\") pod \"maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq\" (UID: \"0be3253c-4570-425f-ac1d-fc27c92f8934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" Oct 30 05:04:06.434376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.431551 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq\" (UID: \"0be3253c-4570-425f-ac1d-fc27c92f8934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" Oct 30 05:04:06.434376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.431630 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq\" (UID: \"0be3253c-4570-425f-ac1d-fc27c92f8934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" Oct 30 05:04:06.475454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.475416 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq\" (UID: \"0be3253c-4570-425f-ac1d-fc27c92f8934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" Oct 30 05:04:06.481499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.481042 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq\" (UID: \"0be3253c-4570-425f-ac1d-fc27c92f8934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" Oct 30 05:04:06.503433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.503368 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vttdq\" (UniqueName: \"kubernetes.io/projected/0be3253c-4570-425f-ac1d-fc27c92f8934-kube-api-access-vttdq\") pod \"maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq\" (UID: \"0be3253c-4570-425f-ac1d-fc27c92f8934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" Oct 30 05:04:06.833384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:06.821793 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" Oct 30 05:04:07.677571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:07.677525 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:04:08.413445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:08.405600 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:08.452383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:08.424837 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ae2139-e029-4928-a061-9ab4fb7fde04" path="/var/lib/kubelet/pods/d3ae2139-e029-4928-a061-9ab4fb7fde04/volumes" Oct 30 05:04:08.475384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:08.457464 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq"] Oct 30 05:04:08.929396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:08.925429 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc"] Oct 30 05:04:08.929396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:08.741510 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:04:08.938443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:08.425026 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:08.946383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:08.847204 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:09.173042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:08.924971 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:04:09.173328 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:08.924695 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:04:09.207293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:09.049027 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:04:09.207610 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:09.086970 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:04:09.236139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:09.096389 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:04:09.266429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:09.122623 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-186d6b66-kz9tc"] Oct 30 05:04:09.677105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:09.658158 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:04:09.677105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:09.463921 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:09.698336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:09.698303 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:04:09.816632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:09.816581 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" podUID="82856c63-29d3-42d2-9078-3880374c9ebc" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 05:04:10.026569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.013471 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:10.026569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.013844 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:04:10.112656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.112584 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 05:04:10.205484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.205436 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:04:10.206063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.206031 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:04:10.383964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.210881 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:04:10.384522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.222128 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:10.385199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.234839 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:04:10.484616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.258459 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:04:10.485121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.310058 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:10.485571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.351117 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d02f424-d083-4820-90d5-abf6a9764688" path="/var/lib/kubelet/pods/4d02f424-d083-4820-90d5-abf6a9764688/volumes" Oct 30 05:04:10.500377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.486247 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" event={"ID":"0be3253c-4570-425f-ac1d-fc27c92f8934","Type":"ContainerStarted","Data":"9b1c67be48e8bf1b6ad553f22de7fc62a15444b30de831036f4cedb51619dcf9"} Oct 30 05:04:10.502425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.502401 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:04:10.503014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.502993 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:04:10.507521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.503212 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:04:10.507714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.507695 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:04:10.507869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.507854 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:04:10.507996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.507984 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:04:10.508127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.508114 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:04:10.641415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.639076 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:04:10.660075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.660051 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:04:10.802960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:10.786897 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": context deadline exceeded" Oct 30 05:04:11.026993 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:11.013684 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:11.026993 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:11.014131 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:11.322007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:11.282436 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:11.414102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:11.414050 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:11.424496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:11.424452 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:11.425415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:11.425380 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:04:11.497680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:11.497635 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:11.650099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:11.649993 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:11.704428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:11.678771 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:12.037293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.037186 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:12.050299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.042898 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd"] Oct 30 05:04:12.050571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.050544 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" event={"ID":"0be3253c-4570-425f-ac1d-fc27c92f8934","Type":"ContainerStarted","Data":"ee9fd681185ce3ff3072cc97772a74ebb35684841da6ead65b5b9a8f9f2ce8ee"} Oct 30 05:04:12.061404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.050713 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9458c9-s9bgd"] Oct 30 05:04:12.061667 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.042941 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:12.235816 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:04:12.235773 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be3253c_4570_425f_ac1d_fc27c92f8934.slice/crio-conmon-ee9fd681185ce3ff3072cc97772a74ebb35684841da6ead65b5b9a8f9f2ce8ee.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be3253c_4570_425f_ac1d_fc27c92f8934.slice/crio-ee9fd681185ce3ff3072cc97772a74ebb35684841da6ead65b5b9a8f9f2ce8ee.scope\": RecentStats: unable to find data in memory cache]" Oct 30 05:04:12.270229 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:04:12.270160 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be3253c_4570_425f_ac1d_fc27c92f8934.slice/crio-ee9fd681185ce3ff3072cc97772a74ebb35684841da6ead65b5b9a8f9f2ce8ee.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be3253c_4570_425f_ac1d_fc27c92f8934.slice/crio-conmon-ee9fd681185ce3ff3072cc97772a74ebb35684841da6ead65b5b9a8f9f2ce8ee.scope\": RecentStats: unable to find data in memory cache]" Oct 30 05:04:12.340383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.330110 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:12.416382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.395415 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:04:12.417262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.417218 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:12.443331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.443282 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:04:12.445907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.445464 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:12.511434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.499150 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:12.669105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.666823 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:04:12.716393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.716168 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:04:12.716573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.716493 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:04:12.813762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.811153 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:04:12.813762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.811637 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:04:12.815351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.814558 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:04:12.815799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.815777 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:04:12.816126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.816103 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:04:12.820300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:12.820271 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:04:13.052096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:13.052062 6587 generic.go:356] "Generic (PLEG): container finished" podID="0be3253c-4570-425f-ac1d-fc27c92f8934" containerID="ee9fd681185ce3ff3072cc97772a74ebb35684841da6ead65b5b9a8f9f2ce8ee" exitCode=0 Oct 30 05:04:13.052616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:13.052148 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" event={"ID":"0be3253c-4570-425f-ac1d-fc27c92f8934","Type":"ContainerDied","Data":"ee9fd681185ce3ff3072cc97772a74ebb35684841da6ead65b5b9a8f9f2ce8ee"} Oct 30 05:04:13.253452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:13.253421 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" Oct 30 05:04:13.337866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:13.336335 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-xks-ca\") pod \"0be3253c-4570-425f-ac1d-fc27c92f8934\" (UID: \"0be3253c-4570-425f-ac1d-fc27c92f8934\") " Oct 30 05:04:13.337866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:13.336434 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be3253c-4570-425f-ac1d-fc27c92f8934-kube-api-access-vttdq\" (UniqueName: \"kubernetes.io/projected/0be3253c-4570-425f-ac1d-fc27c92f8934-kube-api-access-vttdq\") pod \"0be3253c-4570-425f-ac1d-fc27c92f8934\" (UID: \"0be3253c-4570-425f-ac1d-fc27c92f8934\") " Oct 30 05:04:13.337866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:13.337016 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-openshift-ca\") pod \"0be3253c-4570-425f-ac1d-fc27c92f8934\" (UID: \"0be3253c-4570-425f-ac1d-fc27c92f8934\") " Oct 30 05:04:13.339762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:13.339143 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-openshift-ca" pod "0be3253c-4570-425f-ac1d-fc27c92f8934" (UID: "0be3253c-4570-425f-ac1d-fc27c92f8934"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:04:13.344401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:13.342156 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-xks-ca" pod "0be3253c-4570-425f-ac1d-fc27c92f8934" (UID: "0be3253c-4570-425f-ac1d-fc27c92f8934"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:04:13.344401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:13.342159 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be3253c-4570-425f-ac1d-fc27c92f8934-kube-api-access-vttdq" pod "0be3253c-4570-425f-ac1d-fc27c92f8934" (UID: "0be3253c-4570-425f-ac1d-fc27c92f8934"). InnerVolumeSpecName "kube-api-access-vttdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:04:13.440587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:13.440546 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:13.440587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:13.440588 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be3253c-4570-425f-ac1d-fc27c92f8934-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:13.440820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:13.440603 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vttdq\" (UniqueName: \"kubernetes.io/projected/0be3253c-4570-425f-ac1d-fc27c92f8934-kube-api-access-vttdq\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:13.584746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:13.584708 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c720829-65a8-4900-8b86-09ddb4c1608d" path="/var/lib/kubelet/pods/9c720829-65a8-4900-8b86-09ddb4c1608d/volumes" Oct 30 05:04:14.065494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:14.065450 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" event={"ID":"0be3253c-4570-425f-ac1d-fc27c92f8934","Type":"ContainerDied","Data":"9b1c67be48e8bf1b6ad553f22de7fc62a15444b30de831036f4cedb51619dcf9"} Oct 30 05:04:14.065964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:14.065489 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq" Oct 30 05:04:14.065964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:14.065527 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b1c67be48e8bf1b6ad553f22de7fc62a15444b30de831036f4cedb51619dcf9" Oct 30 05:04:17.460337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:17.460253 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s"] Oct 30 05:04:17.470734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:17.470704 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d67de78b-66d7s"] Oct 30 05:04:17.607694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:17.605917 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66b9449-cffe-4534-b23a-92ec73daef4c" path="/var/lib/kubelet/pods/e66b9449-cffe-4534-b23a-92ec73daef4c/volumes" Oct 30 05:04:23.065036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.062669 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk"] Oct 30 05:04:23.065834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.065806 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0be3253c-4570-425f-ac1d-fc27c92f8934" containerName="revoke-keys" Oct 30 05:04:23.065935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.065836 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be3253c-4570-425f-ac1d-fc27c92f8934" containerName="revoke-keys" Oct 30 05:04:23.066027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.066012 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be3253c-4570-425f-ac1d-fc27c92f8934" containerName="revoke-keys" Oct 30 05:04:23.073672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.073643 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" Oct 30 05:04:23.101493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.101460 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk"] Oct 30 05:04:23.136681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.136595 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5a926a9d\"" Oct 30 05:04:23.165380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.137159 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5a926a9d-dockercfg-f9jq9\"" Oct 30 05:04:23.173434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.173396 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs9xf\" (UniqueName: \"kubernetes.io/projected/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-kube-api-access-qs9xf\") pod \"payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk\" (UID: \"e9bec227-0d60-4ee3-b51d-95f4ac2722e2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" Oct 30 05:04:23.174010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.173981 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk\" (UID: \"e9bec227-0d60-4ee3-b51d-95f4ac2722e2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" Oct 30 05:04:23.280180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.280141 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk\" (UID: \"e9bec227-0d60-4ee3-b51d-95f4ac2722e2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" Oct 30 05:04:23.280568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.280528 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs9xf\" (UniqueName: \"kubernetes.io/projected/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-kube-api-access-qs9xf\") pod \"payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk\" (UID: \"e9bec227-0d60-4ee3-b51d-95f4ac2722e2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" Oct 30 05:04:23.294385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.282169 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk\" (UID: \"e9bec227-0d60-4ee3-b51d-95f4ac2722e2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" Oct 30 05:04:23.356624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.346503 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj"] Oct 30 05:04:23.356624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.355056 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs9xf\" (UniqueName: \"kubernetes.io/projected/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-kube-api-access-qs9xf\") pod \"payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk\" (UID: \"e9bec227-0d60-4ee3-b51d-95f4ac2722e2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" Oct 30 05:04:23.369056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.369027 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" Oct 30 05:04:23.372339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.372313 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj"] Oct 30 05:04:23.372514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.372481 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.383567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.382434 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.383567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.382482 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.383567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.382548 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.383567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.382642 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8thkk\" (UniqueName: \"kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-kube-api-access-8thkk\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.383567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.382708 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.383567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.382751 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a3df96-4e74-44bd-a810-f65702e88fd5-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.383567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.382870 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.383567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.382901 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.383567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.382932 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.494424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.494274 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.494424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.494328 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.494424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.494383 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.573014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.545563 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.573014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.545634 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.573014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.545716 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.573014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.545853 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8thkk\" (UniqueName: \"kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-kube-api-access-8thkk\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.573014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.545928 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.573014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.545991 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a3df96-4e74-44bd-a810-f65702e88fd5-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.573014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.547013 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a3df96-4e74-44bd-a810-f65702e88fd5-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.573014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.516154 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.573014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.547313 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.573014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.548264 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.573014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.554463 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.573014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.572938 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.583444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.582511 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8thkk\" (UniqueName: \"kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-kube-api-access-8thkk\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.599822 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.598349 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.603877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.603462 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.757265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.756908 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:23.894588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:23.894539 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk"] Oct 30 05:04:23.899199 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:04:23.899155 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bec227_0d60_4ee3_b51d_95f4ac2722e2.slice/crio-0af56d06f0c3939abb6c63677f5fb3ba2d5568aa106decd4be5375570c911dec WatchSource:0}: Error finding container 0af56d06f0c3939abb6c63677f5fb3ba2d5568aa106decd4be5375570c911dec: Status 404 returned error can't find the container with id 0af56d06f0c3939abb6c63677f5fb3ba2d5568aa106decd4be5375570c911dec Oct 30 05:04:24.103024 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:24.102975 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj"] Oct 30 05:04:24.120072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:24.120033 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:24.120217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:24.120129 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:24.120217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:24.120184 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:24.324645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:24.324490 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" event={"ID":"e9bec227-0d60-4ee3-b51d-95f4ac2722e2","Type":"ContainerStarted","Data":"a7f414985a247215aaaacd0c0b835329e6470d89841fddbc5ee01b1baf3d4550"} Oct 30 05:04:24.324645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:24.324597 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" event={"ID":"e9bec227-0d60-4ee3-b51d-95f4ac2722e2","Type":"ContainerStarted","Data":"0af56d06f0c3939abb6c63677f5fb3ba2d5568aa106decd4be5375570c911dec"} Oct 30 05:04:24.328568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:24.328536 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" event={"ID":"a5a3df96-4e74-44bd-a810-f65702e88fd5","Type":"ContainerStarted","Data":"64e21138c69a7d847d684cfe232ff61f1ce1b23a9949a9f23dc550ad8893f27e"} Oct 30 05:04:24.328705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:24.328589 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" event={"ID":"a5a3df96-4e74-44bd-a810-f65702e88fd5","Type":"ContainerStarted","Data":"4635f67bc4c377ccb72a988ce5e356a2e42bd66cd118648d5c5c51646af8f44a"} Oct 30 05:04:24.373855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:24.370014 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" podStartSLOduration=1.369995065 podStartE2EDuration="1.369995065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:24.366942787 +0000 UTC m=+1188.894709574" watchObservedRunningTime="2026-10-30 05:04:24.369995065 +0000 UTC m=+1188.897761854" Oct 30 05:04:24.760888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:24.760854 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:25.362416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:25.361645 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" Oct 30 05:04:25.777587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:25.777485 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 05:04:25.777587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:25.777569 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" podUID="a5a3df96-4e74-44bd-a810-f65702e88fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 05:04:26.003347 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:04:26.003303 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:04:26.433410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.429332 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" podStartSLOduration=3.429309414 podStartE2EDuration="3.429309414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:24.430709847 +0000 UTC m=+1188.958476634" watchObservedRunningTime="2026-10-30 05:04:26.429309414 +0000 UTC m=+1190.957076200" Oct 30 05:04:26.464390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.463591 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q"] Oct 30 05:04:26.482139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.481776 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.490022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.488882 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q"] Oct 30 05:04:26.568426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.567707 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b0fba292-openshift-default-dockercfg-ql9zz\"" Oct 30 05:04:26.628385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.626857 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-data\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.628385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.626959 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-certs\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.628385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.626990 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f096b7c-287e-4c31-b573-266e99480658-istio-podinfo\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.628385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.627022 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f096b7c-287e-4c31-b573-266e99480658-istiod-ca-cert\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.628385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.627062 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmpk8\" (UniqueName: \"kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-kube-api-access-fmpk8\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.628385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.627109 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-envoy\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.628385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.627155 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-istio-token\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.628385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.627688 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-credential-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.628385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.627805 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.732433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.731420 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-istio-token\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.732433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.731728 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-credential-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.732433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.731815 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.732433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.731879 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-data\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.732433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.731935 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-certs\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.732433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.731962 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f096b7c-287e-4c31-b573-266e99480658-istio-podinfo\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.732433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.731991 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f096b7c-287e-4c31-b573-266e99480658-istiod-ca-cert\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.732433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.732027 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmpk8\" (UniqueName: \"kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-kube-api-access-fmpk8\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.732433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.732070 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-envoy\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.733083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.732936 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-data\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.733586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.733559 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-credential-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.734012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.733991 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.739379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.738247 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f096b7c-287e-4c31-b573-266e99480658-istiod-ca-cert\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.739379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.738539 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-certs\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.763394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.761165 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 05:04:26.776789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.776759 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" podUID="a5a3df96-4e74-44bd-a810-f65702e88fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 05:04:26.777002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.764210 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f096b7c-287e-4c31-b573-266e99480658-istio-podinfo\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.777157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.766081 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-envoy\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.777301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.776682 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmpk8\" (UniqueName: \"kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-kube-api-access-fmpk8\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.777435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.761982 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-istio-token\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-qx79q\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:26.843384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:26.839178 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:27.252612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:27.252571 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q"] Oct 30 05:04:27.267099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:27.267058 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:27.267213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:27.267152 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:27.267213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:27.267193 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:27.390399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:27.389111 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" event={"ID":"0f096b7c-287e-4c31-b573-266e99480658","Type":"ContainerStarted","Data":"79610489dc9223ea438aa1127a0ab45b9cb89e4e51197b263fd15ee3fa1452f8"} Oct 30 05:04:27.773406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:27.764962 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 05:04:27.773406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:27.765033 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" podUID="a5a3df96-4e74-44bd-a810-f65702e88fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 05:04:28.475384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:28.472058 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" event={"ID":"0f096b7c-287e-4c31-b573-266e99480658","Type":"ContainerStarted","Data":"1a72f32d7aea1360ff2fcdd2aa2b4fe1f85ffa7d35df31774f99f5af4e3f3f45"} Oct 30 05:04:28.681324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:28.681154 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" podStartSLOduration=2.681107383 podStartE2EDuration="2.681107383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:28.659157751 +0000 UTC m=+1193.186924537" watchObservedRunningTime="2026-10-30 05:04:28.681107383 +0000 UTC m=+1193.208874171" Oct 30 05:04:28.784512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:28.761001 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 05:04:28.784512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:28.761057 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" podUID="a5a3df96-4e74-44bd-a810-f65702e88fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 05:04:28.848385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:28.847631 6587 patch_prober.go:29] interesting pod/e2e-ait-b0fba292-openshift-default-658566db69-qx79q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:04:28.848385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:28.847697 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" podUID="0f096b7c-287e-4c31-b573-266e99480658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 05:04:28.848385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:28.848009 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:29.703434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:29.703401 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq"] Oct 30 05:04:29.748623 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:29.746517 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:29.760878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:29.760844 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 05:04:29.779220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:29.779176 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" podUID="a5a3df96-4e74-44bd-a810-f65702e88fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 05:04:29.808320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:29.807483 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:29.819109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:29.819075 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0fba292-mzwtq"] Oct 30 05:04:29.824828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:29.823430 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:29.846128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:29.845707 6587 patch_prober.go:29] interesting pod/e2e-ait-b0fba292-openshift-default-658566db69-qx79q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:04:29.846128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:29.845763 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" podUID="0f096b7c-287e-4c31-b573-266e99480658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 05:04:30.760728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:30.760240 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 05:04:30.760728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:30.760301 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" podUID="a5a3df96-4e74-44bd-a810-f65702e88fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 05:04:30.851393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:30.850455 6587 patch_prober.go:29] interesting pod/e2e-ait-b0fba292-openshift-default-658566db69-qx79q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:04:30.851393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:30.850531 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" podUID="0f096b7c-287e-4c31-b573-266e99480658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 05:04:31.252695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:31.252635 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q"] Oct 30 05:04:31.689535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:31.635638 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be3253c-4570-425f-ac1d-fc27c92f8934" path="/var/lib/kubelet/pods/0be3253c-4570-425f-ac1d-fc27c92f8934/volumes" Oct 30 05:04:31.746413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:31.744256 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" podUID="0f096b7c-287e-4c31-b573-266e99480658" containerName="istio-proxy" containerID="cri-o://1a72f32d7aea1360ff2fcdd2aa2b4fe1f85ffa7d35df31774f99f5af4e3f3f45" gracePeriod=30 Oct 30 05:04:31.777163 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:31.777126 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 05:04:31.777494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:31.777461 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" podUID="a5a3df96-4e74-44bd-a810-f65702e88fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 05:04:32.238463 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.238426 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb"] Oct 30 05:04:32.298678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.298643 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb"] Oct 30 05:04:32.298850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.298781 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.340930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.317401 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-47b1fc31-openshift-default-dockercfg-sj52d\"" Oct 30 05:04:32.411706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.411662 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-certs\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.412690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.412660 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-token\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.412883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.412866 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-data\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.413033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.413019 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0be95d-900e-4aab-858a-2f825aff67ba-istiod-ca-cert\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.413224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.413205 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh7v4\" (UniqueName: \"kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-kube-api-access-sh7v4\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.498154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.498057 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-envoy\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.498464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.498435 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-socket\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.498745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.498723 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-podinfo\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.498996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.498970 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-credential-socket\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.610387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.607742 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-podinfo\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.610387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.607837 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-credential-socket\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.610387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.607921 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-certs\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.610387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.607952 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-token\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.610387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.607973 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-data\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.610387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.608012 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0be95d-900e-4aab-858a-2f825aff67ba-istiod-ca-cert\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.610387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.608061 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh7v4\" (UniqueName: \"kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-kube-api-access-sh7v4\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.610387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.608096 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-envoy\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.610387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.608119 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-socket\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.610387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.608760 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-credential-socket\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.610387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.608762 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-socket\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.624913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.624839 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-certs\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.651383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.649873 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-podinfo\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.651383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.650731 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0be95d-900e-4aab-858a-2f825aff67ba-istiod-ca-cert\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.651383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.650927 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-data\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.672381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.667626 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-envoy\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.690380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.689401 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh7v4\" (UniqueName: \"kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-kube-api-access-sh7v4\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.696452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.695883 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-token\") pod \"e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.842384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.840662 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:32.974638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.972779 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:32.983934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.983585 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:33.000388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:32.999810 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:04:33.476467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:33.473999 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb"] Oct 30 05:04:33.484993 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:04:33.484959 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0be95d_900e_4aab_858a_2f825aff67ba.slice/crio-ed8fc0e8114533b7499ec597729bfec8db15531e653e625f12dd593a9e3cea11 WatchSource:0}: Error finding container ed8fc0e8114533b7499ec597729bfec8db15531e653e625f12dd593a9e3cea11: Status 404 returned error can't find the container with id ed8fc0e8114533b7499ec597729bfec8db15531e653e625f12dd593a9e3cea11 Oct 30 05:04:33.488654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:33.488521 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:33.488654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:33.488586 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:33.488654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:33.488615 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:33.875869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:33.875047 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" event={"ID":"dc0be95d-900e-4aab-858a-2f825aff67ba","Type":"ContainerStarted","Data":"ef51cc532d696c0914f6b5676fdc667b8badfec1420503561c85813fe54cfe79"} Oct 30 05:04:33.875869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:33.875100 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" event={"ID":"dc0be95d-900e-4aab-858a-2f825aff67ba","Type":"ContainerStarted","Data":"ed8fc0e8114533b7499ec597729bfec8db15531e653e625f12dd593a9e3cea11"} Oct 30 05:04:33.922629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:33.921262 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" podStartSLOduration=1.9212446600000002 podStartE2EDuration="1.92124466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:33.919631871 +0000 UTC m=+1198.447398658" watchObservedRunningTime="2026-10-30 05:04:33.92124466 +0000 UTC m=+1198.449011449" Oct 30 05:04:34.842254 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:34.842216 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:34.848675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:34.848646 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:34.904855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:34.904816 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:34.906013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:34.905988 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:04:35.801314 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:35.801284 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:04:35.803175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:35.803153 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:04:35.841973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:35.841941 6587 scope.go:122] "RemoveContainer" containerID="4cc2fc67f9ab0a457e6d6ecb6a2bd7dfb4803da88b4a3c5806435235bc1e91d9" Oct 30 05:04:35.854401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:35.854380 6587 scope.go:122] "RemoveContainer" containerID="5751eb81aadc2d205b241ef9008f869e0646d227790cfe03b37b1085475e2bff" Oct 30 05:04:36.379242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:36.379207 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" Oct 30 05:04:37.467562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:37.467526 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7"] Oct 30 05:04:37.474876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:37.474846 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" Oct 30 05:04:37.523914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:37.523883 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7"] Oct 30 05:04:37.524559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:37.524535 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-47b1fc31\"" Oct 30 05:04:37.583939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:37.583908 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-47b1fc31-dockercfg-r57kb\"" Oct 30 05:04:37.663101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:37.663052 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdgx\" (UniqueName: \"kubernetes.io/projected/c363690c-d246-4213-bc45-fe83b8704ae2-kube-api-access-2jdgx\") pod \"payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7\" (UID: \"c363690c-d246-4213-bc45-fe83b8704ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" Oct 30 05:04:37.663421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:37.663392 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c363690c-d246-4213-bc45-fe83b8704ae2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7\" (UID: \"c363690c-d246-4213-bc45-fe83b8704ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" Oct 30 05:04:37.805028 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:37.804650 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c363690c-d246-4213-bc45-fe83b8704ae2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7\" (UID: \"c363690c-d246-4213-bc45-fe83b8704ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" Oct 30 05:04:37.805028 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:37.804903 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdgx\" (UniqueName: \"kubernetes.io/projected/c363690c-d246-4213-bc45-fe83b8704ae2-kube-api-access-2jdgx\") pod \"payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7\" (UID: \"c363690c-d246-4213-bc45-fe83b8704ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" Oct 30 05:04:37.807063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:37.806510 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c363690c-d246-4213-bc45-fe83b8704ae2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7\" (UID: \"c363690c-d246-4213-bc45-fe83b8704ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" Oct 30 05:04:37.851378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:37.848184 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdgx\" (UniqueName: \"kubernetes.io/projected/c363690c-d246-4213-bc45-fe83b8704ae2-kube-api-access-2jdgx\") pod \"payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7\" (UID: \"c363690c-d246-4213-bc45-fe83b8704ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" Oct 30 05:04:38.111213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:38.111133 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" Oct 30 05:04:38.456647 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:04:38.456612 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc363690c_d246_4213_bc45_fe83b8704ae2.slice/crio-7bc65533f39835e0956bf971c0beb5229d749a0ffefa0e5adc19451571ba4d82 WatchSource:0}: Error finding container 7bc65533f39835e0956bf971c0beb5229d749a0ffefa0e5adc19451571ba4d82: Status 404 returned error can't find the container with id 7bc65533f39835e0956bf971c0beb5229d749a0ffefa0e5adc19451571ba4d82 Oct 30 05:04:38.471719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:38.471690 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7"] Oct 30 05:04:38.959619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:38.959577 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" event={"ID":"c363690c-d246-4213-bc45-fe83b8704ae2","Type":"ContainerStarted","Data":"2ca4fde499fa42570f141c26d6e16d2e17a341c3504d6bec5fc6c66cc7de13f0"} Oct 30 05:04:38.959823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:38.959642 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" event={"ID":"c363690c-d246-4213-bc45-fe83b8704ae2","Type":"ContainerStarted","Data":"7bc65533f39835e0956bf971c0beb5229d749a0ffefa0e5adc19451571ba4d82"} Oct 30 05:04:38.959823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:38.959670 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" Oct 30 05:04:39.011142 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:39.011045 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" podStartSLOduration=2.011030771 podStartE2EDuration="2.011030771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:39.005426817 +0000 UTC m=+1203.533193605" watchObservedRunningTime="2026-10-30 05:04:39.011030771 +0000 UTC m=+1203.538797556" Oct 30 05:04:40.314699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.314669 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:40.457793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.457749 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f096b7c-287e-4c31-b573-266e99480658-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f096b7c-287e-4c31-b573-266e99480658-istio-podinfo\") pod \"0f096b7c-287e-4c31-b573-266e99480658\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " Oct 30 05:04:40.457982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.457821 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-istio-token\" (UniqueName: \"kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-istio-token\") pod \"0f096b7c-287e-4c31-b573-266e99480658\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " Oct 30 05:04:40.457982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.457868 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f096b7c-287e-4c31-b573-266e99480658-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f096b7c-287e-4c31-b573-266e99480658-istiod-ca-cert\") pod \"0f096b7c-287e-4c31-b573-266e99480658\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " Oct 30 05:04:40.457982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.457912 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-credential-socket\") pod \"0f096b7c-287e-4c31-b573-266e99480658\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " Oct 30 05:04:40.459172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.458138 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-kube-api-access-fmpk8\" (UniqueName: \"kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-kube-api-access-fmpk8\") pod \"0f096b7c-287e-4c31-b573-266e99480658\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " Oct 30 05:04:40.459172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.458285 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-data\") pod \"0f096b7c-287e-4c31-b573-266e99480658\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " Oct 30 05:04:40.459172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.458328 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-socket\") pod \"0f096b7c-287e-4c31-b573-266e99480658\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " Oct 30 05:04:40.459172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.458436 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-certs\") pod \"0f096b7c-287e-4c31-b573-266e99480658\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " Oct 30 05:04:40.459172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.458503 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-envoy\") pod \"0f096b7c-287e-4c31-b573-266e99480658\" (UID: \"0f096b7c-287e-4c31-b573-266e99480658\") " Oct 30 05:04:40.459172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.458775 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-certs" pod "0f096b7c-287e-4c31-b573-266e99480658" (UID: "0f096b7c-287e-4c31-b573-266e99480658"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:40.459172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.458769 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-socket" pod "0f096b7c-287e-4c31-b573-266e99480658" (UID: "0f096b7c-287e-4c31-b573-266e99480658"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:40.459172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.458856 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-credential-socket" pod "0f096b7c-287e-4c31-b573-266e99480658" (UID: "0f096b7c-287e-4c31-b573-266e99480658"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:40.459172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.458942 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-data" pod "0f096b7c-287e-4c31-b573-266e99480658" (UID: "0f096b7c-287e-4c31-b573-266e99480658"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:40.459833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.459297 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f096b7c-287e-4c31-b573-266e99480658-istiod-ca-cert" pod "0f096b7c-287e-4c31-b573-266e99480658" (UID: "0f096b7c-287e-4c31-b573-266e99480658"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:04:40.459833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.459515 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f096b7c-287e-4c31-b573-266e99480658-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.459833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.459538 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.459833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.459554 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.459833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.459566 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.459833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.459579 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.462717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.462234 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-kube-api-access-fmpk8" pod "0f096b7c-287e-4c31-b573-266e99480658" (UID: "0f096b7c-287e-4c31-b573-266e99480658"). InnerVolumeSpecName "kube-api-access-fmpk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:04:40.462717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.462312 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f096b7c-287e-4c31-b573-266e99480658-istio-podinfo" pod "0f096b7c-287e-4c31-b573-266e99480658" (UID: "0f096b7c-287e-4c31-b573-266e99480658"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:04:40.468424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.464166 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-istio-token" pod "0f096b7c-287e-4c31-b573-266e99480658" (UID: "0f096b7c-287e-4c31-b573-266e99480658"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:04:40.468424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.464273 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-envoy" pod "0f096b7c-287e-4c31-b573-266e99480658" (UID: "0f096b7c-287e-4c31-b573-266e99480658"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:40.561576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.561541 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f096b7c-287e-4c31-b573-266e99480658-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.561576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.561567 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.561576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.561579 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmpk8\" (UniqueName: \"kubernetes.io/projected/0f096b7c-287e-4c31-b573-266e99480658-kube-api-access-fmpk8\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.561869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.561589 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f096b7c-287e-4c31-b573-266e99480658-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:04:40.978670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.978567 6587 generic.go:356] "Generic (PLEG): container finished" podID="0f096b7c-287e-4c31-b573-266e99480658" containerID="1a72f32d7aea1360ff2fcdd2aa2b4fe1f85ffa7d35df31774f99f5af4e3f3f45" exitCode=0 Oct 30 05:04:40.978670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.978626 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" Oct 30 05:04:40.978670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.978645 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" event={"ID":"0f096b7c-287e-4c31-b573-266e99480658","Type":"ContainerDied","Data":"1a72f32d7aea1360ff2fcdd2aa2b4fe1f85ffa7d35df31774f99f5af4e3f3f45"} Oct 30 05:04:40.978943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.978705 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q" event={"ID":"0f096b7c-287e-4c31-b573-266e99480658","Type":"ContainerDied","Data":"79610489dc9223ea438aa1127a0ab45b9cb89e4e51197b263fd15ee3fa1452f8"} Oct 30 05:04:40.978943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.978728 6587 scope.go:122] "RemoveContainer" containerID="1a72f32d7aea1360ff2fcdd2aa2b4fe1f85ffa7d35df31774f99f5af4e3f3f45" Oct 30 05:04:40.991592 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.991571 6587 scope.go:122] "RemoveContainer" containerID="1a72f32d7aea1360ff2fcdd2aa2b4fe1f85ffa7d35df31774f99f5af4e3f3f45" Oct 30 05:04:40.991876 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:04:40.991857 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a72f32d7aea1360ff2fcdd2aa2b4fe1f85ffa7d35df31774f99f5af4e3f3f45\": container with ID starting with 1a72f32d7aea1360ff2fcdd2aa2b4fe1f85ffa7d35df31774f99f5af4e3f3f45 not found: ID does not exist" containerID="1a72f32d7aea1360ff2fcdd2aa2b4fe1f85ffa7d35df31774f99f5af4e3f3f45" Oct 30 05:04:40.991950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:40.991884 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a72f32d7aea1360ff2fcdd2aa2b4fe1f85ffa7d35df31774f99f5af4e3f3f45"} err="failed to get container status \"1a72f32d7aea1360ff2fcdd2aa2b4fe1f85ffa7d35df31774f99f5af4e3f3f45\": rpc error: code = NotFound desc = could not find container \"1a72f32d7aea1360ff2fcdd2aa2b4fe1f85ffa7d35df31774f99f5af4e3f3f45\": container with ID starting with 1a72f32d7aea1360ff2fcdd2aa2b4fe1f85ffa7d35df31774f99f5af4e3f3f45 not found: ID does not exist" Oct 30 05:04:41.059549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:41.059513 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q"] Oct 30 05:04:41.066826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:41.066799 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q"] Oct 30 05:04:41.582944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:41.582907 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f096b7c-287e-4c31-b573-266e99480658" path="/var/lib/kubelet/pods/0f096b7c-287e-4c31-b573-266e99480658/volumes" Oct 30 05:04:49.971809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:49.971778 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" Oct 30 05:04:51.589128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.589092 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7"] Oct 30 05:04:51.590048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.590021 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0f096b7c-287e-4c31-b573-266e99480658" containerName="istio-proxy" Oct 30 05:04:51.590048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.590048 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f096b7c-287e-4c31-b573-266e99480658" containerName="istio-proxy" Oct 30 05:04:51.590217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.590208 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f096b7c-287e-4c31-b573-266e99480658" containerName="istio-proxy" Oct 30 05:04:51.595806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.595786 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.625848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.625802 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7"] Oct 30 05:04:51.657253 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.657217 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-e15c99-openshift-default-dockercfg-ngf5s\"" Oct 30 05:04:51.724331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.724287 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-certs\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.724578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.724417 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istiod-ca-cert\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.724578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.724538 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-token\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.724713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.724631 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-credential-socket\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.724713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.724688 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skhn\" (UniqueName: \"kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-kube-api-access-8skhn\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.724909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.724882 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-envoy\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.725048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.724971 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-podinfo\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.725134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.725119 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-data\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.725205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.725158 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-socket\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.827132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.827087 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-certs\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.827349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.827157 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istiod-ca-cert\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.827349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.827205 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-token\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.827349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.827286 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-credential-socket\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.827349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.827341 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skhn\" (UniqueName: \"kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-kube-api-access-8skhn\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.827611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.827486 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-envoy\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.827611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.827562 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-podinfo\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.827761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.827721 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-data\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.827820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.827757 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-socket\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.842962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.842861 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-credential-socket\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.843230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.843209 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-certs\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.848494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.843876 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istiod-ca-cert\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.860727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.858573 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-data\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.869709 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.867023 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-socket\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.872469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.872015 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-envoy\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.875032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.874434 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-podinfo\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.882222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.881213 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skhn\" (UniqueName: \"kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-kube-api-access-8skhn\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.932413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.918908 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-token\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:51.932413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:51.927441 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:52.310046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:52.310005 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7"] Oct 30 05:04:52.321597 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:04:52.321564 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a8b4fd_b32c_4097_bdeb_80afe0c5541b.slice/crio-91d303bb96b27d261ef4b7bcfeeec14f38bd77fb127f89e0854ad888871ff53f WatchSource:0}: Error finding container 91d303bb96b27d261ef4b7bcfeeec14f38bd77fb127f89e0854ad888871ff53f: Status 404 returned error can't find the container with id 91d303bb96b27d261ef4b7bcfeeec14f38bd77fb127f89e0854ad888871ff53f Oct 30 05:04:52.325338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:52.325297 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:52.325476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:52.325454 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:52.325548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:52.325521 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:04:53.213315 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:53.213264 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" event={"ID":"50a8b4fd-b32c-4097-bdeb-80afe0c5541b","Type":"ContainerStarted","Data":"267dcf322ae60df01bc8acb9112bb2035f3d3a7546a56af2b4c2fb6617af97a1"} Oct 30 05:04:53.213824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:53.213338 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" event={"ID":"50a8b4fd-b32c-4097-bdeb-80afe0c5541b","Type":"ContainerStarted","Data":"91d303bb96b27d261ef4b7bcfeeec14f38bd77fb127f89e0854ad888871ff53f"} Oct 30 05:04:53.291696 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:53.291646 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" podStartSLOduration=2.291631603 podStartE2EDuration="2.291631603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:53.277659528 +0000 UTC m=+1217.805426313" watchObservedRunningTime="2026-10-30 05:04:53.291631603 +0000 UTC m=+1217.819398389" Oct 30 05:04:53.928469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:53.928428 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:53.935318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:53.935280 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:54.223762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:54.223683 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:54.224678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:04:54.224641 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:04:55.986984 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:04:55.986949 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:05:10.959687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:10.959651 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx"] Oct 30 05:05:10.968937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:10.968906 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" Oct 30 05:05:10.984805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:10.984774 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx"] Oct 30 05:05:10.994387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:10.993545 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:05:11.001456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.001419 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:05:11.001456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.001451 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:05:11.001675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.001528 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:05:11.081633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.081600 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg85m\" (UniqueName: \"kubernetes.io/projected/f8e626e6-4d6a-4568-ac7e-f100030c4f63-kube-api-access-vg85m\") pod \"maas-api-revoke-sub-76ae3bb0-tn6tx\" (UID: \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" Oct 30 05:05:11.081825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.081651 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76ae3bb0-tn6tx\" (UID: \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" Oct 30 05:05:11.082008 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.081983 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76ae3bb0-tn6tx\" (UID: \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" Oct 30 05:05:11.184145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.184104 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg85m\" (UniqueName: \"kubernetes.io/projected/f8e626e6-4d6a-4568-ac7e-f100030c4f63-kube-api-access-vg85m\") pod \"maas-api-revoke-sub-76ae3bb0-tn6tx\" (UID: \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" Oct 30 05:05:11.184318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.184254 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76ae3bb0-tn6tx\" (UID: \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" Oct 30 05:05:11.184530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.184514 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76ae3bb0-tn6tx\" (UID: \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" Oct 30 05:05:11.185067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.185038 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76ae3bb0-tn6tx\" (UID: \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" Oct 30 05:05:11.186799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.186781 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76ae3bb0-tn6tx\" (UID: \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" Oct 30 05:05:11.208804 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.208778 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg85m\" (UniqueName: \"kubernetes.io/projected/f8e626e6-4d6a-4568-ac7e-f100030c4f63-kube-api-access-vg85m\") pod \"maas-api-revoke-sub-76ae3bb0-tn6tx\" (UID: \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" Oct 30 05:05:11.289118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.289032 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" Oct 30 05:05:11.599273 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:05:11.599241 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e626e6_4d6a_4568_ac7e_f100030c4f63.slice/crio-50c16b43592bc27c77056ed83e6372fbf99fb57b9db36612d4d64435aaf984d3 WatchSource:0}: Error finding container 50c16b43592bc27c77056ed83e6372fbf99fb57b9db36612d4d64435aaf984d3: Status 404 returned error can't find the container with id 50c16b43592bc27c77056ed83e6372fbf99fb57b9db36612d4d64435aaf984d3 Oct 30 05:05:11.601490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:11.601466 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx"] Oct 30 05:05:12.449258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:12.449214 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" event={"ID":"f8e626e6-4d6a-4568-ac7e-f100030c4f63","Type":"ContainerStarted","Data":"8ad9e2837e5049ec536d82d8aa278b53264cc6b8c3d64cd5455b28634cda8d31"} Oct 30 05:05:12.449750 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:12.449295 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" event={"ID":"f8e626e6-4d6a-4568-ac7e-f100030c4f63","Type":"ContainerStarted","Data":"50c16b43592bc27c77056ed83e6372fbf99fb57b9db36612d4d64435aaf984d3"} Oct 30 05:05:12.503041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:12.502983 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" podStartSLOduration=2.5029662249999998 podStartE2EDuration="2.502966225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:12.501791196 +0000 UTC m=+1237.029557981" watchObservedRunningTime="2026-10-30 05:05:12.502966225 +0000 UTC m=+1237.030733012" Oct 30 05:05:12.885664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:12.885628 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d"] Oct 30 05:05:12.973380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:12.971496 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" Oct 30 05:05:13.001819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.001777 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d"] Oct 30 05:05:13.197493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.195606 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d\" (UID: \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" Oct 30 05:05:13.197493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.197029 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n5s2\" (UniqueName: \"kubernetes.io/projected/3e35a2dc-3d04-4126-9fcc-6791b6019fda-kube-api-access-8n5s2\") pod \"maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d\" (UID: \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" Oct 30 05:05:13.197493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.197189 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d\" (UID: \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" Oct 30 05:05:13.298583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.298550 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d\" (UID: \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" Oct 30 05:05:13.298775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.298606 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n5s2\" (UniqueName: \"kubernetes.io/projected/3e35a2dc-3d04-4126-9fcc-6791b6019fda-kube-api-access-8n5s2\") pod \"maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d\" (UID: \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" Oct 30 05:05:13.298775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.298711 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d\" (UID: \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" Oct 30 05:05:13.299294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.299272 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d\" (UID: \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" Oct 30 05:05:13.301068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.301042 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d\" (UID: \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" Oct 30 05:05:13.323742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.323715 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n5s2\" (UniqueName: \"kubernetes.io/projected/3e35a2dc-3d04-4126-9fcc-6791b6019fda-kube-api-access-8n5s2\") pod \"maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d\" (UID: \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" Oct 30 05:05:13.464416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.464318 6587 generic.go:356] "Generic (PLEG): container finished" podID="f8e626e6-4d6a-4568-ac7e-f100030c4f63" containerID="8ad9e2837e5049ec536d82d8aa278b53264cc6b8c3d64cd5455b28634cda8d31" exitCode=0 Oct 30 05:05:13.464884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.464415 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" event={"ID":"f8e626e6-4d6a-4568-ac7e-f100030c4f63","Type":"ContainerDied","Data":"8ad9e2837e5049ec536d82d8aa278b53264cc6b8c3d64cd5455b28634cda8d31"} Oct 30 05:05:13.600208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.600173 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" Oct 30 05:05:13.937877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:13.937842 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d"] Oct 30 05:05:13.939832 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:05:13.939804 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e35a2dc_3d04_4126_9fcc_6791b6019fda.slice/crio-f83e40f33e84b0f9a4474e8a99a1818dc9155c2a6fc8f616bd6526d954f09447 WatchSource:0}: Error finding container f83e40f33e84b0f9a4474e8a99a1818dc9155c2a6fc8f616bd6526d954f09447: Status 404 returned error can't find the container with id f83e40f33e84b0f9a4474e8a99a1818dc9155c2a6fc8f616bd6526d954f09447 Oct 30 05:05:14.524674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:14.524636 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" event={"ID":"3e35a2dc-3d04-4126-9fcc-6791b6019fda","Type":"ContainerStarted","Data":"f83e40f33e84b0f9a4474e8a99a1818dc9155c2a6fc8f616bd6526d954f09447"} Oct 30 05:05:14.950403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:14.950371 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" Oct 30 05:05:15.022464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.015331 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e626e6-4d6a-4568-ac7e-f100030c4f63-kube-api-access-vg85m\" (UniqueName: \"kubernetes.io/projected/f8e626e6-4d6a-4568-ac7e-f100030c4f63-kube-api-access-vg85m\") pod \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\" (UID: \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\") " Oct 30 05:05:15.022464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.015396 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-xks-ca\") pod \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\" (UID: \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\") " Oct 30 05:05:15.022464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.015634 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-openshift-ca\") pod \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\" (UID: \"f8e626e6-4d6a-4568-ac7e-f100030c4f63\") " Oct 30 05:05:15.022464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.022206 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-openshift-ca" pod "f8e626e6-4d6a-4568-ac7e-f100030c4f63" (UID: "f8e626e6-4d6a-4568-ac7e-f100030c4f63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:15.058223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.058178 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-xks-ca" pod "f8e626e6-4d6a-4568-ac7e-f100030c4f63" (UID: "f8e626e6-4d6a-4568-ac7e-f100030c4f63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:05:15.058417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.058324 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e626e6-4d6a-4568-ac7e-f100030c4f63-kube-api-access-vg85m" pod "f8e626e6-4d6a-4568-ac7e-f100030c4f63" (UID: "f8e626e6-4d6a-4568-ac7e-f100030c4f63"). InnerVolumeSpecName "kube-api-access-vg85m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:15.148395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.147690 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:15.148395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.147727 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg85m\" (UniqueName: \"kubernetes.io/projected/f8e626e6-4d6a-4568-ac7e-f100030c4f63-kube-api-access-vg85m\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:15.148395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.147747 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e626e6-4d6a-4568-ac7e-f100030c4f63-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:15.551951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.551887 6587 generic.go:356] "Generic (PLEG): container finished" podID="3e35a2dc-3d04-4126-9fcc-6791b6019fda" containerID="8ebbbb5bc08faf2ebde7477e8ba30e77e8837faef0481add17893de0272641ea" exitCode=0 Oct 30 05:05:15.552383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.551981 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" event={"ID":"3e35a2dc-3d04-4126-9fcc-6791b6019fda","Type":"ContainerDied","Data":"8ebbbb5bc08faf2ebde7477e8ba30e77e8837faef0481add17893de0272641ea"} Oct 30 05:05:15.553980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.553957 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" Oct 30 05:05:15.554115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.554005 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx" event={"ID":"f8e626e6-4d6a-4568-ac7e-f100030c4f63","Type":"ContainerDied","Data":"50c16b43592bc27c77056ed83e6372fbf99fb57b9db36612d4d64435aaf984d3"} Oct 30 05:05:15.554115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:15.554046 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50c16b43592bc27c77056ed83e6372fbf99fb57b9db36612d4d64435aaf984d3" Oct 30 05:05:16.057161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:16.057127 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv"] Oct 30 05:05:16.062651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:16.062616 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ac2012-plddv"] Oct 30 05:05:16.865674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:16.865518 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" Oct 30 05:05:17.003621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:17.003572 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-openshift-ca\") pod \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\" (UID: \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\") " Oct 30 05:05:17.003831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:17.003769 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e35a2dc-3d04-4126-9fcc-6791b6019fda-kube-api-access-8n5s2\" (UniqueName: \"kubernetes.io/projected/3e35a2dc-3d04-4126-9fcc-6791b6019fda-kube-api-access-8n5s2\") pod \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\" (UID: \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\") " Oct 30 05:05:17.004030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:17.003999 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-openshift-ca" pod "3e35a2dc-3d04-4126-9fcc-6791b6019fda" (UID: "3e35a2dc-3d04-4126-9fcc-6791b6019fda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:17.004117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:17.004037 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-xks-ca\") pod \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\" (UID: \"3e35a2dc-3d04-4126-9fcc-6791b6019fda\") " Oct 30 05:05:17.005271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:17.005234 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:17.006459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:17.006435 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-xks-ca" pod "3e35a2dc-3d04-4126-9fcc-6791b6019fda" (UID: "3e35a2dc-3d04-4126-9fcc-6791b6019fda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:05:17.013990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:17.013959 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e35a2dc-3d04-4126-9fcc-6791b6019fda-kube-api-access-8n5s2" pod "3e35a2dc-3d04-4126-9fcc-6791b6019fda" (UID: "3e35a2dc-3d04-4126-9fcc-6791b6019fda"). InnerVolumeSpecName "kube-api-access-8n5s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:17.107030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:17.106993 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n5s2\" (UniqueName: \"kubernetes.io/projected/3e35a2dc-3d04-4126-9fcc-6791b6019fda-kube-api-access-8n5s2\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:17.107030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:17.107025 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e35a2dc-3d04-4126-9fcc-6791b6019fda-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:17.582097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:17.581742 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" Oct 30 05:05:17.592886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:17.592853 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d39cd0-5c20-4f5c-acc3-6a63bebfcc07" path="/var/lib/kubelet/pods/79d39cd0-5c20-4f5c-acc3-6a63bebfcc07/volumes" Oct 30 05:05:17.594123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:17.594095 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d" event={"ID":"3e35a2dc-3d04-4126-9fcc-6791b6019fda","Type":"ContainerDied","Data":"f83e40f33e84b0f9a4474e8a99a1818dc9155c2a6fc8f616bd6526d954f09447"} Oct 30 05:05:17.594310 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:17.594294 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f83e40f33e84b0f9a4474e8a99a1818dc9155c2a6fc8f616bd6526d954f09447" Oct 30 05:05:18.920381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:18.919588 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s"] Oct 30 05:05:18.921106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:18.920945 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3e35a2dc-3d04-4126-9fcc-6791b6019fda" containerName="revoke-keys" Oct 30 05:05:18.921106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:18.920970 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e35a2dc-3d04-4126-9fcc-6791b6019fda" containerName="revoke-keys" Oct 30 05:05:18.921106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:18.920994 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f8e626e6-4d6a-4568-ac7e-f100030c4f63" containerName="revoke-keys" Oct 30 05:05:18.921106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:18.921003 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e626e6-4d6a-4568-ac7e-f100030c4f63" containerName="revoke-keys" Oct 30 05:05:18.921423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:18.921244 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e35a2dc-3d04-4126-9fcc-6791b6019fda" containerName="revoke-keys" Oct 30 05:05:18.921423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:18.921259 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e626e6-4d6a-4568-ac7e-f100030c4f63" containerName="revoke-keys" Oct 30 05:05:18.941908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:18.940670 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:18.978477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:18.978445 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s"] Oct 30 05:05:18.978983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:18.978946 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c613f5ea-openshift-default-dockercfg-dxwcw\"" Oct 30 05:05:19.150515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.150468 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.150945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.150923 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.167968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.167932 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.182637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.182540 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59851d6-7c9f-483a-b778-9f78aba34404-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.183091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.183067 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.183658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.183404 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxrrg\" (UniqueName: \"kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-kube-api-access-sxrrg\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.183658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.183550 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.183811 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.183747 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59851d6-7c9f-483a-b778-9f78aba34404-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.184043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.183926 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.290875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.290826 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.294230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.294197 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.295296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.295273 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.295864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.294782 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.296069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.296051 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.296198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.296184 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59851d6-7c9f-483a-b778-9f78aba34404-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.296372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.296339 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.297493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.297472 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxrrg\" (UniqueName: \"kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-kube-api-access-sxrrg\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.297677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.297657 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.297892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.297089 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.298062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.297902 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59851d6-7c9f-483a-b778-9f78aba34404-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.298317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.297382 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.298617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.298589 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59851d6-7c9f-483a-b778-9f78aba34404-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.298854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.298835 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.313657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.309195 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.313657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.310439 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59851d6-7c9f-483a-b778-9f78aba34404-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.347407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.344266 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.364395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.347656 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxrrg\" (UniqueName: \"kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-kube-api-access-sxrrg\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:19.590124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:19.590072 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:20.015707 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:05:20.015672 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59851d6_7c9f_483a_b778_9f78aba34404.slice/crio-9961c9d606cbf6ca753176a58fb7bbf7de6365607c18ccbd3fce892d2e242f6c WatchSource:0}: Error finding container 9961c9d606cbf6ca753176a58fb7bbf7de6365607c18ccbd3fce892d2e242f6c: Status 404 returned error can't find the container with id 9961c9d606cbf6ca753176a58fb7bbf7de6365607c18ccbd3fce892d2e242f6c Oct 30 05:05:20.018673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:20.018639 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:20.018762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:20.018713 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:20.018762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:20.018747 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:20.021698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:20.021668 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s"] Oct 30 05:05:20.620212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:20.620170 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" event={"ID":"c59851d6-7c9f-483a-b778-9f78aba34404","Type":"ContainerStarted","Data":"b28fa3a0544653a02386e6c3f05a5bbe807c13d1c69a7641da2bbb5fe9162c25"} Oct 30 05:05:20.620500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:20.620278 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" event={"ID":"c59851d6-7c9f-483a-b778-9f78aba34404","Type":"ContainerStarted","Data":"9961c9d606cbf6ca753176a58fb7bbf7de6365607c18ccbd3fce892d2e242f6c"} Oct 30 05:05:20.696405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:20.696295 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" podStartSLOduration=2.696278124 podStartE2EDuration="2.696278124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:20.694928934 +0000 UTC m=+1245.222695721" watchObservedRunningTime="2026-10-30 05:05:20.696278124 +0000 UTC m=+1245.224044910" Oct 30 05:05:21.591171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:21.591134 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:21.598102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:21.598074 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:21.632977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:21.632940 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:21.634028 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:21.634000 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:05:23.586031 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:23.585988 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk"] Oct 30 05:05:25.371344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.371269 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2"] Oct 30 05:05:25.375088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.375067 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" Oct 30 05:05:25.417400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.417308 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c613f5ea\"" Oct 30 05:05:25.432322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.432270 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c613f5ea-dockercfg-kxvhb\"" Oct 30 05:05:25.487382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.485972 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-plugins-config-volume\") pod \"payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2\" (UID: \"52b8904b-0f62-4abb-8ec3-b43c6cb628ff\") " pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" Oct 30 05:05:25.487382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.486252 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zntc\" (UniqueName: \"kubernetes.io/projected/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-kube-api-access-2zntc\") pod \"payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2\" (UID: \"52b8904b-0f62-4abb-8ec3-b43c6cb628ff\") " pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" Oct 30 05:05:25.551934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.551891 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2"] Oct 30 05:05:25.595285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.593738 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-plugins-config-volume\") pod \"payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2\" (UID: \"52b8904b-0f62-4abb-8ec3-b43c6cb628ff\") " pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" Oct 30 05:05:25.595285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.593989 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zntc\" (UniqueName: \"kubernetes.io/projected/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-kube-api-access-2zntc\") pod \"payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2\" (UID: \"52b8904b-0f62-4abb-8ec3-b43c6cb628ff\") " pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" Oct 30 05:05:25.595285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.595201 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-plugins-config-volume\") pod \"payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2\" (UID: \"52b8904b-0f62-4abb-8ec3-b43c6cb628ff\") " pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" Oct 30 05:05:25.645638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.642804 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zntc\" (UniqueName: \"kubernetes.io/projected/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-kube-api-access-2zntc\") pod \"payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2\" (UID: \"52b8904b-0f62-4abb-8ec3-b43c6cb628ff\") " pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" Oct 30 05:05:25.693007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.692959 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" Oct 30 05:05:25.701899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.701866 6587 generic.go:356] "Generic (PLEG): container finished" podID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerID="ef51cc532d696c0914f6b5676fdc667b8badfec1420503561c85813fe54cfe79" exitCode=0 Oct 30 05:05:25.702060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.702000 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" event={"ID":"dc0be95d-900e-4aab-858a-2f825aff67ba","Type":"ContainerDied","Data":"ef51cc532d696c0914f6b5676fdc667b8badfec1420503561c85813fe54cfe79"} Oct 30 05:05:25.702610 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.702584 6587 scope.go:122] "RemoveContainer" containerID="ef51cc532d696c0914f6b5676fdc667b8badfec1420503561c85813fe54cfe79" Oct 30 05:05:25.705788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.705749 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:25.705884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.705866 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:25.705944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:25.705912 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:26.077627 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:05:26.077588 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:05:26.173738 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:05:26.173677 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b8904b_0f62_4abb_8ec3_b43c6cb628ff.slice/crio-181cc5ab1770a72da3b4a40fefc94bdf8b0f4b7c5b6fb64988df03d53a4c8ec8 WatchSource:0}: Error finding container 181cc5ab1770a72da3b4a40fefc94bdf8b0f4b7c5b6fb64988df03d53a4c8ec8: Status 404 returned error can't find the container with id 181cc5ab1770a72da3b4a40fefc94bdf8b0f4b7c5b6fb64988df03d53a4c8ec8 Oct 30 05:05:26.179173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.179143 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2"] Oct 30 05:05:26.384053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.384021 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw"] Oct 30 05:05:26.391342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.391311 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" Oct 30 05:05:26.417911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.417879 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw"] Oct 30 05:05:26.419807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.419682 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:05:26.419807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.419706 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:05:26.420009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.419813 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:05:26.420190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.420167 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:05:26.521521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.521488 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtvbv\" (UniqueName: \"kubernetes.io/projected/33e62d64-3af2-41bc-8eeb-0f4439429c0c-kube-api-access-gtvbv\") pod \"maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw\" (UID: \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" Oct 30 05:05:26.521718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.521615 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw\" (UID: \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" Oct 30 05:05:26.521807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.521784 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw\" (UID: \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" Oct 30 05:05:26.623816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.623781 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtvbv\" (UniqueName: \"kubernetes.io/projected/33e62d64-3af2-41bc-8eeb-0f4439429c0c-kube-api-access-gtvbv\") pod \"maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw\" (UID: \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" Oct 30 05:05:26.624021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.623876 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw\" (UID: \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" Oct 30 05:05:26.624021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.623982 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw\" (UID: \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" Oct 30 05:05:26.624752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.624724 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw\" (UID: \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" Oct 30 05:05:26.626430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.626404 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw\" (UID: \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" Oct 30 05:05:26.659591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.659565 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtvbv\" (UniqueName: \"kubernetes.io/projected/33e62d64-3af2-41bc-8eeb-0f4439429c0c-kube-api-access-gtvbv\") pod \"maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw\" (UID: \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" Oct 30 05:05:26.708666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.708578 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" Oct 30 05:05:26.733431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.716808 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" event={"ID":"52b8904b-0f62-4abb-8ec3-b43c6cb628ff","Type":"ContainerStarted","Data":"32a648306b78587fb4e63e49b06c299afb03f083e585fa675932a19b168e0851"} Oct 30 05:05:26.733431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.716870 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" event={"ID":"52b8904b-0f62-4abb-8ec3-b43c6cb628ff","Type":"ContainerStarted","Data":"181cc5ab1770a72da3b4a40fefc94bdf8b0f4b7c5b6fb64988df03d53a4c8ec8"} Oct 30 05:05:26.733431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.718049 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" Oct 30 05:05:26.733431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.720811 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" event={"ID":"dc0be95d-900e-4aab-858a-2f825aff67ba","Type":"ContainerStarted","Data":"5a5b8b25cc04e94b3174469f625f4f4f8814ecfdd6329295b6fcfc374d8ecb9e"} Oct 30 05:05:26.773131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.773079 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" podStartSLOduration=1.773063439 podStartE2EDuration="1.773063439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:26.768504796 +0000 UTC m=+1251.296271582" watchObservedRunningTime="2026-10-30 05:05:26.773063439 +0000 UTC m=+1251.300830240" Oct 30 05:05:26.842140 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.842103 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:05:26.842337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.842315 6587 patch_prober.go:29] interesting pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:05:26.842437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:26.842401 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 05:05:27.054571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:27.054518 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw"] Oct 30 05:05:27.759136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:27.758498 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" event={"ID":"33e62d64-3af2-41bc-8eeb-0f4439429c0c","Type":"ContainerStarted","Data":"113268d3e88e0d3d19abb67dcecca8e4bb342c3f6c8ee7e0001ac1a6d03fc81e"} Oct 30 05:05:27.759136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:27.758549 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" event={"ID":"33e62d64-3af2-41bc-8eeb-0f4439429c0c","Type":"ContainerStarted","Data":"93f0f34a67c4d17547555914bc38256d5bf112bef5bcffbffabd255baefd61cf"} Oct 30 05:05:27.812720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:27.812663 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" podStartSLOduration=1.812649896 podStartE2EDuration="1.812649896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:27.807185625 +0000 UTC m=+1252.334952411" watchObservedRunningTime="2026-10-30 05:05:27.812649896 +0000 UTC m=+1252.340416683" Oct 30 05:05:27.841979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:27.841940 6587 patch_prober.go:29] interesting pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:05:27.842224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:27.842194 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 05:05:28.586868 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:28.586756 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" podUID="e9bec227-0d60-4ee3-b51d-95f4ac2722e2" containerName="payload-pre-processing" containerID="cri-o://a7f414985a247215aaaacd0c0b835329e6470d89841fddbc5ee01b1baf3d4550" gracePeriod=25 Oct 30 05:05:28.771851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:28.771816 6587 generic.go:356] "Generic (PLEG): container finished" podID="33e62d64-3af2-41bc-8eeb-0f4439429c0c" containerID="113268d3e88e0d3d19abb67dcecca8e4bb342c3f6c8ee7e0001ac1a6d03fc81e" exitCode=0 Oct 30 05:05:28.772254 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:28.771937 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" event={"ID":"33e62d64-3af2-41bc-8eeb-0f4439429c0c","Type":"ContainerDied","Data":"113268d3e88e0d3d19abb67dcecca8e4bb342c3f6c8ee7e0001ac1a6d03fc81e"} Oct 30 05:05:28.773743 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:28.773719 6587 generic.go:356] "Generic (PLEG): container finished" podID="e9bec227-0d60-4ee3-b51d-95f4ac2722e2" containerID="a7f414985a247215aaaacd0c0b835329e6470d89841fddbc5ee01b1baf3d4550" exitCode=0 Oct 30 05:05:28.773892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:28.773872 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" event={"ID":"e9bec227-0d60-4ee3-b51d-95f4ac2722e2","Type":"ContainerDied","Data":"a7f414985a247215aaaacd0c0b835329e6470d89841fddbc5ee01b1baf3d4550"} Oct 30 05:05:28.854481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:28.852739 6587 patch_prober.go:29] interesting pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:05:28.854481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:28.852796 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 05:05:29.001308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:28.999157 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" Oct 30 05:05:29.059375 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.059321 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86"] Oct 30 05:05:29.067866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.066453 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ab3227-v4f86"] Oct 30 05:05:29.184064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.183690 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-kube-api-access-qs9xf\" (UniqueName: \"kubernetes.io/projected/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-kube-api-access-qs9xf\") pod \"e9bec227-0d60-4ee3-b51d-95f4ac2722e2\" (UID: \"e9bec227-0d60-4ee3-b51d-95f4ac2722e2\") " Oct 30 05:05:29.184064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.183770 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-plugins-config-volume\") pod \"e9bec227-0d60-4ee3-b51d-95f4ac2722e2\" (UID: \"e9bec227-0d60-4ee3-b51d-95f4ac2722e2\") " Oct 30 05:05:29.184582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.184551 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-plugins-config-volume" pod "e9bec227-0d60-4ee3-b51d-95f4ac2722e2" (UID: "e9bec227-0d60-4ee3-b51d-95f4ac2722e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:29.186021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.185998 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-kube-api-access-qs9xf" pod "e9bec227-0d60-4ee3-b51d-95f4ac2722e2" (UID: "e9bec227-0d60-4ee3-b51d-95f4ac2722e2"). InnerVolumeSpecName "kube-api-access-qs9xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:29.285790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.285748 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs9xf\" (UniqueName: \"kubernetes.io/projected/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-kube-api-access-qs9xf\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:29.285790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.285782 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9bec227-0d60-4ee3-b51d-95f4ac2722e2-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:29.583233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.583201 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10" path="/var/lib/kubelet/pods/07e9ddc8-8e34-4c3f-be96-ec71ebbd0a10/volumes" Oct 30 05:05:29.847454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.841963 6587 patch_prober.go:29] interesting pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:05:29.847454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.842042 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 05:05:29.858989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.858170 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" Oct 30 05:05:29.863149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.863112 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk" event={"ID":"e9bec227-0d60-4ee3-b51d-95f4ac2722e2","Type":"ContainerDied","Data":"0af56d06f0c3939abb6c63677f5fb3ba2d5568aa106decd4be5375570c911dec"} Oct 30 05:05:29.863438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.863414 6587 scope.go:122] "RemoveContainer" containerID="a7f414985a247215aaaacd0c0b835329e6470d89841fddbc5ee01b1baf3d4550" Oct 30 05:05:29.979489 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.979446 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk"] Oct 30 05:05:29.985570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:29.985498 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk"] Oct 30 05:05:30.078439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.078404 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq"] Oct 30 05:05:30.085734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.085700 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f21d02-5m8tq"] Oct 30 05:05:30.168329 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.168300 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" Oct 30 05:05:30.258728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.258652 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-openshift-ca\") pod \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\" (UID: \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\") " Oct 30 05:05:30.259115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.259075 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-openshift-ca" pod "33e62d64-3af2-41bc-8eeb-0f4439429c0c" (UID: "33e62d64-3af2-41bc-8eeb-0f4439429c0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:30.259261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.259126 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-xks-ca\") pod \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\" (UID: \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\") " Oct 30 05:05:30.259261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.259175 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e62d64-3af2-41bc-8eeb-0f4439429c0c-kube-api-access-gtvbv\" (UniqueName: \"kubernetes.io/projected/33e62d64-3af2-41bc-8eeb-0f4439429c0c-kube-api-access-gtvbv\") pod \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\" (UID: \"33e62d64-3af2-41bc-8eeb-0f4439429c0c\") " Oct 30 05:05:30.260031 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.260008 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:30.262031 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.262004 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e62d64-3af2-41bc-8eeb-0f4439429c0c-kube-api-access-gtvbv" pod "33e62d64-3af2-41bc-8eeb-0f4439429c0c" (UID: "33e62d64-3af2-41bc-8eeb-0f4439429c0c"). InnerVolumeSpecName "kube-api-access-gtvbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:30.262136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.262106 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-xks-ca" pod "33e62d64-3af2-41bc-8eeb-0f4439429c0c" (UID: "33e62d64-3af2-41bc-8eeb-0f4439429c0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:05:30.361407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.361312 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e62d64-3af2-41bc-8eeb-0f4439429c0c-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:30.361407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.361343 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtvbv\" (UniqueName: \"kubernetes.io/projected/33e62d64-3af2-41bc-8eeb-0f4439429c0c-kube-api-access-gtvbv\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:30.849380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.845761 6587 patch_prober.go:29] interesting pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:05:30.849380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.845844 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 05:05:30.869729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.869700 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" Oct 30 05:05:30.870007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.869732 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw" event={"ID":"33e62d64-3af2-41bc-8eeb-0f4439429c0c","Type":"ContainerDied","Data":"93f0f34a67c4d17547555914bc38256d5bf112bef5bcffbffabd255baefd61cf"} Oct 30 05:05:30.870145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:30.870038 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93f0f34a67c4d17547555914bc38256d5bf112bef5bcffbffabd255baefd61cf" Oct 30 05:05:31.679387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:31.676733 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e462373-e3db-49a2-87b7-d860a46f0e5c" path="/var/lib/kubelet/pods/4e462373-e3db-49a2-87b7-d860a46f0e5c/volumes" Oct 30 05:05:31.679387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:31.677458 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9bec227-0d60-4ee3-b51d-95f4ac2722e2" path="/var/lib/kubelet/pods/e9bec227-0d60-4ee3-b51d-95f4ac2722e2/volumes" Oct 30 05:05:31.855379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:31.852527 6587 patch_prober.go:29] interesting pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:05:31.855379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:31.852593 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 05:05:32.305588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:32.305551 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d"] Oct 30 05:05:32.348379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:32.345115 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5a926a9d-6z44d"] Oct 30 05:05:33.005911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:33.005873 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:05:33.006974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:33.006951 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:05:33.008003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:33.007983 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:05:33.584487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:33.584451 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e35a2dc-3d04-4126-9fcc-6791b6019fda" path="/var/lib/kubelet/pods/3e35a2dc-3d04-4126-9fcc-6791b6019fda/volumes" Oct 30 05:05:35.897484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:35.897451 6587 scope.go:122] "RemoveContainer" containerID="f5fb5f644c43d6f48f92d2465edd6ccbc61db9cb9fab36df89ccd7173476d9f2" Oct 30 05:05:35.911963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:35.911941 6587 scope.go:122] "RemoveContainer" containerID="0bcb3675be5597cf9309a0494ae66c3acd442a3670e619cb0c38ac9824a84206" Oct 30 05:05:35.924889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:35.924867 6587 scope.go:122] "RemoveContainer" containerID="b0fcc11eb89db88c02280cd0abcc315f3428494d129a8f1da0b4280d66797b80" Oct 30 05:05:35.935976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:35.935945 6587 scope.go:122] "RemoveContainer" containerID="bc3b8ddb058bab276836403a807e2e0d058cd6dff021c8edeef4fd9cbb9895d9" Oct 30 05:05:35.950524 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:35.950501 6587 scope.go:122] "RemoveContainer" containerID="6982c48f69306b0f8c4dfe47ba6d13f46a81d78ef26773527d2d774d9a0c20ee" Oct 30 05:05:35.962513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:35.962487 6587 scope.go:122] "RemoveContainer" containerID="05615386e783cd9b03d1c2d807dcb63550edb2dacd88559cc646ea7c759616b7" Oct 30 05:05:35.979648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:35.979577 6587 scope.go:122] "RemoveContainer" containerID="56844ffbd10c7b51be4e0b7d28385ebdc57fd6d16af540c35ba247362c096bcc" Oct 30 05:05:36.055486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:36.052760 6587 scope.go:122] "RemoveContainer" containerID="10dbd51cdf070de7a0ffad4877d8890f295213ef0718043099d8c996ff725eed" Oct 30 05:05:36.084959 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:36.084410 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk"] Oct 30 05:05:36.129929 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:36.129891 6587 scope.go:122] "RemoveContainer" containerID="249de0e1fb087df68f590e3129d2c06e10c6036638e8bbf50f1d8a67559aa74f" Oct 30 05:05:36.158109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:36.157372 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d17404e8-twwnk"] Oct 30 05:05:36.242435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:36.236310 6587 scope.go:122] "RemoveContainer" containerID="68f221e75c2fb0660e10278c5c67b92e733246c948b5fb5abb7f20043365c5c9" Oct 30 05:05:36.288639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:36.288589 6587 scope.go:122] "RemoveContainer" containerID="5fc207671721f7f4a5971dcb317eaf6c38f33edc6faccbce02d8c446e645c120" Oct 30 05:05:36.951734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:36.951699 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7"] Oct 30 05:05:37.641969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:37.641928 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b7e47a-91c4-4e73-a1b0-02d5353b3a62" path="/var/lib/kubelet/pods/b8b7e47a-91c4-4e73-a1b0-02d5353b3a62/volumes" Oct 30 05:05:38.887678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:38.887642 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" Oct 30 05:05:39.626595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:39.626529 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:39.650941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:39.650809 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:39.686714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:39.686672 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:39.724495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:39.724442 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:39.740577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:39.740550 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4"] Oct 30 05:05:39.795393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:39.790312 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-404168a7-297x4"] Oct 30 05:05:39.828936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:39.828892 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:39.843391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:39.838804 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:05:40.120387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.116456 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:40.193392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.184604 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:40.214417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.202975 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:40.444450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.425486 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:05:40.489383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.488685 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:40.762492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.751443 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:05:40.762492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.751959 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:05:40.762492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.752345 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:40.762492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.752818 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:05:40.814969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.781330 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:40.814969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.781782 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:05:40.897933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.841635 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:40.897933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.842142 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:05:40.897933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.861148 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:40.897933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:40.866628 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:05:41.122397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.094826 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:41.152334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.117669 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:41.177001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.176970 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:05:41.207860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.185307 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:41.209956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.209905 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:05:41.534490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.526998 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:05:41.566208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.565739 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:41.600261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.574011 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:05:41.601401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.601217 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:05:41.771734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.771695 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:41.813439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.783687 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:05:41.874485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.874441 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:41.896235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.883634 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:41.985665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:41.985618 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" podUID="c363690c-d246-4213-bc45-fe83b8704ae2" containerName="payload-pre-processing" containerID="cri-o://2ca4fde499fa42570f141c26d6e16d2e17a341c3504d6bec5fc6c66cc7de13f0" gracePeriod=25 Oct 30 05:05:42.206102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:42.206049 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:42.552444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:42.551967 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:42.604972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:42.604928 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:42.607132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:42.607103 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6e16f83-b311-4ae9-8aba-d87af6fd0eca" path="/var/lib/kubelet/pods/e6e16f83-b311-4ae9-8aba-d87af6fd0eca/volumes" Oct 30 05:05:42.609557 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:05:42.609321 6587 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.008s" Oct 30 05:05:42.609969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:42.609953 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:05:42.610452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:42.610435 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:05:42.611774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:42.611756 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:05:42.739234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:42.739200 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:05:42.800392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:42.797418 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:42.859796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:42.859711 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:05:42.892550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:42.892506 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:42.966273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:42.905730 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:42.966273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:42.910436 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:05:43.078980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.078898 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:05:43.101381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.085527 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:05:43.101381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.085582 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:05:43.124431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.115582 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:05:43.364882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.364846 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" Oct 30 05:05:43.473856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.473770 6587 generic.go:356] "Generic (PLEG): container finished" podID="c363690c-d246-4213-bc45-fe83b8704ae2" containerID="2ca4fde499fa42570f141c26d6e16d2e17a341c3504d6bec5fc6c66cc7de13f0" exitCode=0 Oct 30 05:05:43.474109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.473817 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" Oct 30 05:05:43.474198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.473835 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" event={"ID":"c363690c-d246-4213-bc45-fe83b8704ae2","Type":"ContainerDied","Data":"2ca4fde499fa42570f141c26d6e16d2e17a341c3504d6bec5fc6c66cc7de13f0"} Oct 30 05:05:43.474292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.474267 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7" event={"ID":"c363690c-d246-4213-bc45-fe83b8704ae2","Type":"ContainerDied","Data":"7bc65533f39835e0956bf971c0beb5229d749a0ffefa0e5adc19451571ba4d82"} Oct 30 05:05:43.474347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.474297 6587 scope.go:122] "RemoveContainer" containerID="2ca4fde499fa42570f141c26d6e16d2e17a341c3504d6bec5fc6c66cc7de13f0" Oct 30 05:05:43.484416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.481890 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c363690c-d246-4213-bc45-fe83b8704ae2-kube-api-access-2jdgx\" (UniqueName: \"kubernetes.io/projected/c363690c-d246-4213-bc45-fe83b8704ae2-kube-api-access-2jdgx\") pod \"c363690c-d246-4213-bc45-fe83b8704ae2\" (UID: \"c363690c-d246-4213-bc45-fe83b8704ae2\") " Oct 30 05:05:43.484416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.482056 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c363690c-d246-4213-bc45-fe83b8704ae2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c363690c-d246-4213-bc45-fe83b8704ae2-plugins-config-volume\") pod \"c363690c-d246-4213-bc45-fe83b8704ae2\" (UID: \"c363690c-d246-4213-bc45-fe83b8704ae2\") " Oct 30 05:05:43.487998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.487524 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c363690c-d246-4213-bc45-fe83b8704ae2-plugins-config-volume" pod "c363690c-d246-4213-bc45-fe83b8704ae2" (UID: "c363690c-d246-4213-bc45-fe83b8704ae2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:43.491372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.490289 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c363690c-d246-4213-bc45-fe83b8704ae2-kube-api-access-2jdgx" pod "c363690c-d246-4213-bc45-fe83b8704ae2" (UID: "c363690c-d246-4213-bc45-fe83b8704ae2"). InnerVolumeSpecName "kube-api-access-2jdgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:43.497415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.497383 6587 scope.go:122] "RemoveContainer" containerID="2ca4fde499fa42570f141c26d6e16d2e17a341c3504d6bec5fc6c66cc7de13f0" Oct 30 05:05:43.497983 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:05:43.497956 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca4fde499fa42570f141c26d6e16d2e17a341c3504d6bec5fc6c66cc7de13f0\": container with ID starting with 2ca4fde499fa42570f141c26d6e16d2e17a341c3504d6bec5fc6c66cc7de13f0 not found: ID does not exist" containerID="2ca4fde499fa42570f141c26d6e16d2e17a341c3504d6bec5fc6c66cc7de13f0" Oct 30 05:05:43.498079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.497993 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca4fde499fa42570f141c26d6e16d2e17a341c3504d6bec5fc6c66cc7de13f0"} err="failed to get container status \"2ca4fde499fa42570f141c26d6e16d2e17a341c3504d6bec5fc6c66cc7de13f0\": rpc error: code = NotFound desc = could not find container \"2ca4fde499fa42570f141c26d6e16d2e17a341c3504d6bec5fc6c66cc7de13f0\": container with ID starting with 2ca4fde499fa42570f141c26d6e16d2e17a341c3504d6bec5fc6c66cc7de13f0 not found: ID does not exist" Oct 30 05:05:43.588445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.588411 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jdgx\" (UniqueName: \"kubernetes.io/projected/c363690c-d246-4213-bc45-fe83b8704ae2-kube-api-access-2jdgx\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:43.588445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.588440 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c363690c-d246-4213-bc45-fe83b8704ae2-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:43.839381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.832400 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7"] Oct 30 05:05:43.867372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:43.867317 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7"] Oct 30 05:05:45.583209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:45.583177 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c363690c-d246-4213-bc45-fe83b8704ae2" path="/var/lib/kubelet/pods/c363690c-d246-4213-bc45-fe83b8704ae2/volumes" Oct 30 05:05:50.584413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:50.582244 6587 generic.go:356] "Generic (PLEG): container finished" podID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerID="267dcf322ae60df01bc8acb9112bb2035f3d3a7546a56af2b4c2fb6617af97a1" exitCode=0 Oct 30 05:05:50.584413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:50.582655 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" event={"ID":"50a8b4fd-b32c-4097-bdeb-80afe0c5541b","Type":"ContainerDied","Data":"267dcf322ae60df01bc8acb9112bb2035f3d3a7546a56af2b4c2fb6617af97a1"} Oct 30 05:05:50.584413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:50.583194 6587 scope.go:122] "RemoveContainer" containerID="267dcf322ae60df01bc8acb9112bb2035f3d3a7546a56af2b4c2fb6617af97a1" Oct 30 05:05:50.585584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:50.585140 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:50.585584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:50.585209 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:50.585584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:50.585240 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:51.595753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:51.595711 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" event={"ID":"50a8b4fd-b32c-4097-bdeb-80afe0c5541b","Type":"ContainerStarted","Data":"227cbc21a0b522db94825bb9cb7c2a70cb860a54e774733c6726924aa11dd51b"} Oct 30 05:05:51.937515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:51.931983 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:05:51.937515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:51.932118 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:05:51.937515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:51.932176 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:05:52.137387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:52.135120 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs"] Oct 30 05:05:52.162406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:52.152677 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0405e0-lk6gs"] Oct 30 05:05:52.381417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:52.381375 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw"] Oct 30 05:05:52.395424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:52.395378 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-47b1fc31-6gdvw"] Oct 30 05:05:52.639448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:52.614259 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb"] Oct 30 05:05:52.639448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:52.614545 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerName="istio-proxy" containerID="cri-o://5a5b8b25cc04e94b3174469f625f4f4f8814ecfdd6329295b6fcfc374d8ecb9e" gracePeriod=30 Oct 30 05:05:52.951438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:52.950781 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:05:52.951438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:52.950839 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:05:53.583640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:53.583597 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145a9be0-0ea7-48ec-aab8-c186cb5af459" path="/var/lib/kubelet/pods/145a9be0-0ea7-48ec-aab8-c186cb5af459/volumes" Oct 30 05:05:53.584232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:53.584206 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e62d64-3af2-41bc-8eeb-0f4439429c0c" path="/var/lib/kubelet/pods/33e62d64-3af2-41bc-8eeb-0f4439429c0c/volumes" Oct 30 05:05:53.938439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:53.937528 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:05:53.938439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:53.937633 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:05:54.231453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.225478 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:05:54.558594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.554845 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j"] Oct 30 05:05:54.558594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.555621 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c363690c-d246-4213-bc45-fe83b8704ae2" containerName="payload-pre-processing" Oct 30 05:05:54.558594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.555637 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c363690c-d246-4213-bc45-fe83b8704ae2" containerName="payload-pre-processing" Oct 30 05:05:54.558594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.555664 6587 cpu_manager.go:414] "containerMap: removing container" podUID="33e62d64-3af2-41bc-8eeb-0f4439429c0c" containerName="revoke-keys" Oct 30 05:05:54.558594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.555669 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e62d64-3af2-41bc-8eeb-0f4439429c0c" containerName="revoke-keys" Oct 30 05:05:54.558594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.555686 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e9bec227-0d60-4ee3-b51d-95f4ac2722e2" containerName="payload-pre-processing" Oct 30 05:05:54.558594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.555692 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9bec227-0d60-4ee3-b51d-95f4ac2722e2" containerName="payload-pre-processing" Oct 30 05:05:54.558594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.555802 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c363690c-d246-4213-bc45-fe83b8704ae2" containerName="payload-pre-processing" Oct 30 05:05:54.558594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.555812 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e62d64-3af2-41bc-8eeb-0f4439429c0c" containerName="revoke-keys" Oct 30 05:05:54.558594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.555822 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9bec227-0d60-4ee3-b51d-95f4ac2722e2" containerName="payload-pre-processing" Oct 30 05:05:54.558594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.558592 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.580736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.580706 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j"] Oct 30 05:05:54.588998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.588942 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ff4760-openshift-default-dockercfg-5kn59\"" Oct 30 05:05:54.602126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.602096 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istiod-ca-cert\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.602294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.602138 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-certs\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.602294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.602249 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-podinfo\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.602541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.602512 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-socket\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.602656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.602637 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-credential-socket\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.602745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.602721 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djzbj\" (UniqueName: \"kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-kube-api-access-djzbj\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.602829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.602812 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-envoy\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.602901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.602885 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-data\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.602965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.602915 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-token\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.704922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.704876 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-podinfo\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.706242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.706206 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-socket\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.706454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.706326 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-credential-socket\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.706454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.706385 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djzbj\" (UniqueName: \"kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-kube-api-access-djzbj\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.706454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.706446 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-envoy\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.706643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.706522 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-data\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.706643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.706550 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-token\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.706754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.706637 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istiod-ca-cert\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.706754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.706674 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-certs\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.707125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.707102 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-certs\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.708188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.708159 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-data\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.708591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.708567 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-socket\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.709106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.709083 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istiod-ca-cert\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.709345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.709325 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-credential-socket\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.763762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.763722 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-envoy\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.771578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.771481 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-podinfo\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.782342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.778684 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-token\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.782342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.781178 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djzbj\" (UniqueName: \"kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-kube-api-access-djzbj\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.889874 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.889768 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:54.945565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.945039 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:05:54.945565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:54.945121 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:05:55.201673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:55.201587 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh"] Oct 30 05:05:55.204428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:55.204396 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b84178-86zsh"] Oct 30 05:05:55.296415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:55.292746 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j"] Oct 30 05:05:55.303433 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:05:55.303351 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0a3d86_39e5_4015_baa3_6b09d8713f69.slice/crio-68848be2d85d3770f59575cc2fffe9c86768bd4037ad01c6435184f77e6f5546 WatchSource:0}: Error finding container 68848be2d85d3770f59575cc2fffe9c86768bd4037ad01c6435184f77e6f5546: Status 404 returned error can't find the container with id 68848be2d85d3770f59575cc2fffe9c86768bd4037ad01c6435184f77e6f5546 Oct 30 05:05:55.306271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:55.306233 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:55.306402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:55.306325 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:55.306402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:55.306389 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:05:55.586616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:55.586577 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58f88cf-af86-49b9-8e2e-fa14124410a9" path="/var/lib/kubelet/pods/e58f88cf-af86-49b9-8e2e-fa14124410a9/volumes" Oct 30 05:05:55.802948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:55.802909 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" event={"ID":"fc0a3d86-39e5-4015-baa3-6b09d8713f69","Type":"ContainerStarted","Data":"4da638fc4aa0cb8c8c051537dfbe05629172376895c3f0a3de7307e43948bb47"} Oct 30 05:05:55.803117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:55.802973 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" event={"ID":"fc0a3d86-39e5-4015-baa3-6b09d8713f69","Type":"ContainerStarted","Data":"68848be2d85d3770f59575cc2fffe9c86768bd4037ad01c6435184f77e6f5546"} Oct 30 05:05:55.887348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:55.887241 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" podStartSLOduration=1.887220164 podStartE2EDuration="1.887220164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:55.878115623 +0000 UTC m=+1280.405882413" watchObservedRunningTime="2026-10-30 05:05:55.887220164 +0000 UTC m=+1280.414986953" Oct 30 05:05:55.890450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:55.890423 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:55.930381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:55.928550 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:05:55.930381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:55.928611 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:05:55.988639 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:05:55.988603 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:05:56.897007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:56.896977 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:56.930377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:56.929615 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:05:56.930377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:56.929672 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:05:57.825302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:57.825262 6587 generic.go:356] "Generic (PLEG): container finished" podID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerID="5a5b8b25cc04e94b3174469f625f4f4f8814ecfdd6329295b6fcfc374d8ecb9e" exitCode=0 Oct 30 05:05:57.825711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:57.825339 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" event={"ID":"dc0be95d-900e-4aab-858a-2f825aff67ba","Type":"ContainerDied","Data":"5a5b8b25cc04e94b3174469f625f4f4f8814ecfdd6329295b6fcfc374d8ecb9e"} Oct 30 05:05:57.825711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:57.825424 6587 scope.go:122] "RemoveContainer" containerID="ef51cc532d696c0914f6b5676fdc667b8badfec1420503561c85813fe54cfe79" Oct 30 05:05:57.825793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:57.825724 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:57.828095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:57.827298 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:05:57.935405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:57.935332 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:05:57.938053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:57.937860 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:05:58.022617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.022581 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:05:58.206884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.206849 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-podinfo\") pod \"dc0be95d-900e-4aab-858a-2f825aff67ba\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " Oct 30 05:05:58.207086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.206910 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-token\" (UniqueName: \"kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-token\") pod \"dc0be95d-900e-4aab-858a-2f825aff67ba\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " Oct 30 05:05:58.207086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.206948 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-certs\") pod \"dc0be95d-900e-4aab-858a-2f825aff67ba\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " Oct 30 05:05:58.207086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.207038 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-socket\") pod \"dc0be95d-900e-4aab-858a-2f825aff67ba\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " Oct 30 05:05:58.207282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.207088 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-credential-socket\") pod \"dc0be95d-900e-4aab-858a-2f825aff67ba\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " Oct 30 05:05:58.207282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.207144 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-kube-api-access-sh7v4\" (UniqueName: \"kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-kube-api-access-sh7v4\") pod \"dc0be95d-900e-4aab-858a-2f825aff67ba\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " Oct 30 05:05:58.207282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.207235 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-data\") pod \"dc0be95d-900e-4aab-858a-2f825aff67ba\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " Oct 30 05:05:58.207282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.207270 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc0be95d-900e-4aab-858a-2f825aff67ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0be95d-900e-4aab-858a-2f825aff67ba-istiod-ca-cert\") pod \"dc0be95d-900e-4aab-858a-2f825aff67ba\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " Oct 30 05:05:58.207520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.207461 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-envoy\") pod \"dc0be95d-900e-4aab-858a-2f825aff67ba\" (UID: \"dc0be95d-900e-4aab-858a-2f825aff67ba\") " Oct 30 05:05:58.207611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.207582 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-credential-socket" pod "dc0be95d-900e-4aab-858a-2f825aff67ba" (UID: "dc0be95d-900e-4aab-858a-2f825aff67ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:58.207777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.207760 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-certs" pod "dc0be95d-900e-4aab-858a-2f825aff67ba" (UID: "dc0be95d-900e-4aab-858a-2f825aff67ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:58.207926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.207909 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-socket" pod "dc0be95d-900e-4aab-858a-2f825aff67ba" (UID: "dc0be95d-900e-4aab-858a-2f825aff67ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:58.208347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.208328 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:58.208457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.208350 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:58.208457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.208379 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:58.208615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.208595 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-data" pod "dc0be95d-900e-4aab-858a-2f825aff67ba" (UID: "dc0be95d-900e-4aab-858a-2f825aff67ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:58.209097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.209063 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0be95d-900e-4aab-858a-2f825aff67ba-istiod-ca-cert" pod "dc0be95d-900e-4aab-858a-2f825aff67ba" (UID: "dc0be95d-900e-4aab-858a-2f825aff67ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:58.211305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.210963 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-kube-api-access-sh7v4" pod "dc0be95d-900e-4aab-858a-2f825aff67ba" (UID: "dc0be95d-900e-4aab-858a-2f825aff67ba"). InnerVolumeSpecName "kube-api-access-sh7v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:58.218543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.218512 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-envoy" pod "dc0be95d-900e-4aab-858a-2f825aff67ba" (UID: "dc0be95d-900e-4aab-858a-2f825aff67ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:58.219459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.219438 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-token" pod "dc0be95d-900e-4aab-858a-2f825aff67ba" (UID: "dc0be95d-900e-4aab-858a-2f825aff67ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:58.220822 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.220794 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-podinfo" pod "dc0be95d-900e-4aab-858a-2f825aff67ba" (UID: "dc0be95d-900e-4aab-858a-2f825aff67ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:05:58.309717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.309680 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh7v4\" (UniqueName: \"kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-kube-api-access-sh7v4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:58.309717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.309710 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:58.309717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.309721 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0be95d-900e-4aab-858a-2f825aff67ba-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:58.309938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.309731 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:58.309938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.309739 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:58.309938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.309747 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0be95d-900e-4aab-858a-2f825aff67ba-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:05:58.836373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.836323 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" event={"ID":"dc0be95d-900e-4aab-858a-2f825aff67ba","Type":"ContainerDied","Data":"ed8fc0e8114533b7499ec597729bfec8db15531e653e625f12dd593a9e3cea11"} Oct 30 05:05:58.837062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.836390 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb" Oct 30 05:05:58.837062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.836406 6587 scope.go:122] "RemoveContainer" containerID="5a5b8b25cc04e94b3174469f625f4f4f8814ecfdd6329295b6fcfc374d8ecb9e" Oct 30 05:05:58.925865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.925827 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb"] Oct 30 05:05:58.938270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:58.938239 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb"] Oct 30 05:05:59.583684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:05:59.583650 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" path="/var/lib/kubelet/pods/dc0be95d-900e-4aab-858a-2f825aff67ba/volumes" Oct 30 05:06:08.030868 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:08.030827 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4"] Oct 30 05:06:08.044605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:08.044564 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b1030e5-r8xl4"] Oct 30 05:06:09.583511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:09.583479 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b3f500a-2154-4497-bb3c-d61fddaac62a" path="/var/lib/kubelet/pods/7b3f500a-2154-4497-bb3c-d61fddaac62a/volumes" Oct 30 05:06:17.048901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:17.048869 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl"] Oct 30 05:06:17.060928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:17.060897 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803c8e40-z26bl"] Oct 30 05:06:17.584933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:17.584900 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17915466-615b-467d-9483-211240c24e24" path="/var/lib/kubelet/pods/17915466-615b-467d-9483-211240c24e24/volumes" Oct 30 05:06:19.033039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:19.033007 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw"] Oct 30 05:06:19.046233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:19.046207 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-334b30c4-7bszw"] Oct 30 05:06:19.590331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:19.590295 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63492924-2c05-4dda-86d2-b0fb7b03b0f8" path="/var/lib/kubelet/pods/63492924-2c05-4dda-86d2-b0fb7b03b0f8/volumes" Oct 30 05:06:22.674495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:22.674458 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:06:22.679423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:22.674522 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:06:23.227127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:23.227032 6587 generic.go:356] "Generic (PLEG): container finished" podID="c59851d6-7c9f-483a-b778-9f78aba34404" containerID="b28fa3a0544653a02386e6c3f05a5bbe807c13d1c69a7641da2bbb5fe9162c25" exitCode=0 Oct 30 05:06:23.227283 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:23.227113 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" event={"ID":"c59851d6-7c9f-483a-b778-9f78aba34404","Type":"ContainerDied","Data":"b28fa3a0544653a02386e6c3f05a5bbe807c13d1c69a7641da2bbb5fe9162c25"} Oct 30 05:06:23.227735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:23.227716 6587 scope.go:122] "RemoveContainer" containerID="b28fa3a0544653a02386e6c3f05a5bbe807c13d1c69a7641da2bbb5fe9162c25" Oct 30 05:06:23.229892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:23.229856 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:06:23.229983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:23.229951 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:06:23.230022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:23.229997 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:06:24.264995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:24.258301 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" event={"ID":"c59851d6-7c9f-483a-b778-9f78aba34404","Type":"ContainerStarted","Data":"72f37956350a0373d3bfdad901e799af45c50c818a25e5464ff212d8ba96679d"} Oct 30 05:06:24.590253 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:24.590223 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:06:24.590454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:24.590433 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:06:24.590515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:24.590485 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 05:06:25.591342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:25.591305 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:06:25.591774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:25.591375 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 05:06:26.030435 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:06:26.028563 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:06:26.590946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:26.590911 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:06:26.591113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:26.590980 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 05:06:27.590590 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:27.590554 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:06:27.590989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:27.590607 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 05:06:28.591270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:28.591233 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:06:28.591694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:28.591294 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 05:06:29.591179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:29.591141 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:06:29.591375 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:29.591200 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 05:06:30.595070 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:30.595037 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:06:30.595558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:30.595545 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:06:30.596412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:30.596344 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:06:35.298279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.298242 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n"] Oct 30 05:06:35.299026 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.299007 6587 cpu_manager.go:414] "containerMap: removing container" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerName="istio-proxy" Oct 30 05:06:35.299117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.299027 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerName="istio-proxy" Oct 30 05:06:35.299117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.299054 6587 cpu_manager.go:414] "containerMap: removing container" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerName="istio-proxy" Oct 30 05:06:35.299117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.299059 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerName="istio-proxy" Oct 30 05:06:35.299291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.299181 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerName="istio-proxy" Oct 30 05:06:35.299291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.299194 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc0be95d-900e-4aab-858a-2f825aff67ba" containerName="istio-proxy" Oct 30 05:06:35.302167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.302146 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.320332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.320293 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-93bad9-openshift-default-dockercfg-hnpf6\"" Oct 30 05:06:35.339505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.339468 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n"] Oct 30 05:06:35.380919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.380860 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.380919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.380926 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.381143 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.381103 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl5hb\" (UniqueName: \"kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-kube-api-access-cl5hb\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.381191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.381163 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.381341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.381314 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.381519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.381496 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.381606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.381548 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.381606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.381594 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.381736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.381717 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.493596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.493554 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.494330 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.494301 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.494911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.494887 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.495477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.495458 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.495698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.495680 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.509380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.496475 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.510257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.510230 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl5hb\" (UniqueName: \"kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-kube-api-access-cl5hb\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.510828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.510804 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.511108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.511087 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.512210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.496036 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.512342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.511618 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.512479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.510044 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.512584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.495342 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.512677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.511705 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.524301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.524275 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.524742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.524701 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.562432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.543838 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.565500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.565469 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl5hb\" (UniqueName: \"kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-kube-api-access-cl5hb\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:35.656413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.656382 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-93bad9-openshift-default-dockercfg-hnpf6\"" Oct 30 05:06:35.657788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:35.657766 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:36.095880 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:06:36.095842 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33684ba3_e578_4f9f_bafd_7f9faefb39ad.slice/crio-b23a79f08e75994c4f75e16a565f5cfc328d9356f08f3e0ca04e784b155c3882 WatchSource:0}: Error finding container b23a79f08e75994c4f75e16a565f5cfc328d9356f08f3e0ca04e784b155c3882: Status 404 returned error can't find the container with id b23a79f08e75994c4f75e16a565f5cfc328d9356f08f3e0ca04e784b155c3882 Oct 30 05:06:36.098994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:36.098964 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n"] Oct 30 05:06:36.099246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:36.099208 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:06:36.099306 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:36.099293 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:06:36.099376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:36.099333 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:06:36.405840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:36.405751 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" event={"ID":"33684ba3-e578-4f9f-bafd-7f9faefb39ad","Type":"ContainerStarted","Data":"dbea4f0a03d664e8fedc1643d39031ffb12a762d11a6b57d5017dd93102c5568"} Oct 30 05:06:36.405840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:36.405818 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" event={"ID":"33684ba3-e578-4f9f-bafd-7f9faefb39ad","Type":"ContainerStarted","Data":"b23a79f08e75994c4f75e16a565f5cfc328d9356f08f3e0ca04e784b155c3882"} Oct 30 05:06:36.440258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:36.440214 6587 scope.go:122] "RemoveContainer" containerID="b55844fa44b0385bedcf08ed92b5f920d169e74a4eeea405eee43d05f949fa26" Oct 30 05:06:36.460658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:36.460422 6587 scope.go:122] "RemoveContainer" containerID="f3693471d94de13538f393febf244e26f74c6262f00e1104a449e6cb1adc70bb" Oct 30 05:06:36.462583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:36.462538 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" podStartSLOduration=1.462523947 podStartE2EDuration="1.462523947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:36.460633547 +0000 UTC m=+1320.988400333" watchObservedRunningTime="2026-10-30 05:06:36.462523947 +0000 UTC m=+1320.990290734" Oct 30 05:06:36.476265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:36.476241 6587 scope.go:122] "RemoveContainer" containerID="87ff3071793e2aa212a68a0ff5bd597b9dacf14cfe843f55c56dcafe5d3b5bda" Oct 30 05:06:36.503679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:36.503594 6587 scope.go:122] "RemoveContainer" containerID="8ed6a2ab49b7ad6c39859815ddb219ec141064d67baedbf9273a75315f2047a8" Oct 30 05:06:36.517968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:36.517946 6587 scope.go:122] "RemoveContainer" containerID="864b2d147928af8ce8508ab4b30947966c1f773207226f433c035647ba07eee2" Oct 30 05:06:36.667935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:36.666857 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:37.665319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:37.665291 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:38.033763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:38.033639 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm"] Oct 30 05:06:38.038298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:38.038266 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa08835e-4sxqm"] Oct 30 05:06:38.425924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:38.425863 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:38.426960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:38.426935 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:06:39.588855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:39.588808 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dc6a393-2c3c-41be-b898-bcc9b87b5390" path="/var/lib/kubelet/pods/3dc6a393-2c3c-41be-b898-bcc9b87b5390/volumes" Oct 30 05:06:41.190184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:41.190153 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk"] Oct 30 05:06:41.195574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:41.195552 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" Oct 30 05:06:41.214967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:41.214939 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk"] Oct 30 05:06:41.295741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:41.295705 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-93bad9\"" Oct 30 05:06:41.303273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:41.297760 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-93bad9-dockercfg-xnp5l\"" Oct 30 05:06:41.402128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:41.401974 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk\" (UID: \"25ba8b43-1249-48f5-91b1-25bdb0f82dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" Oct 30 05:06:41.402480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:41.402426 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6p5w\" (UniqueName: \"kubernetes.io/projected/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-kube-api-access-b6p5w\") pod \"payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk\" (UID: \"25ba8b43-1249-48f5-91b1-25bdb0f82dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" Oct 30 05:06:41.553382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:41.552094 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk\" (UID: \"25ba8b43-1249-48f5-91b1-25bdb0f82dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" Oct 30 05:06:41.553382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:41.552169 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6p5w\" (UniqueName: \"kubernetes.io/projected/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-kube-api-access-b6p5w\") pod \"payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk\" (UID: \"25ba8b43-1249-48f5-91b1-25bdb0f82dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" Oct 30 05:06:41.560388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:41.554474 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk\" (UID: \"25ba8b43-1249-48f5-91b1-25bdb0f82dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" Oct 30 05:06:41.584994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:41.584957 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6p5w\" (UniqueName: \"kubernetes.io/projected/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-kube-api-access-b6p5w\") pod \"payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk\" (UID: \"25ba8b43-1249-48f5-91b1-25bdb0f82dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" Oct 30 05:06:41.811183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:41.811099 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" Oct 30 05:06:42.037091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.037054 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj"] Oct 30 05:06:42.050867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.050832 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45446341-jtmpj"] Oct 30 05:06:42.134317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.134286 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk"] Oct 30 05:06:42.173089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.172630 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm"] Oct 30 05:06:42.181043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.176979 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.201604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.201579 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm"] Oct 30 05:06:42.207201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.207173 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-528d67-openshift-default-dockercfg-hchfc\"" Oct 30 05:06:42.283435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.274563 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-socket\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.283435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.274790 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-credential-socket\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.283435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.274907 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-token\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.283435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.274944 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-data\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.283435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.274983 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-certs\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.283435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.275019 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-podinfo\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.283435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.275044 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07d2e44-c567-42b0-a939-3d2f33b2a078-istiod-ca-cert\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.283435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.275122 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkmb9\" (UniqueName: \"kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-kube-api-access-jkmb9\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.283435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.275157 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-envoy\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.422430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.390658 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-token\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.422430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.390723 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-data\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.422430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.390766 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-certs\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.422430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.390808 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-podinfo\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.422430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.390832 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07d2e44-c567-42b0-a939-3d2f33b2a078-istiod-ca-cert\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.422430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.390928 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkmb9\" (UniqueName: \"kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-kube-api-access-jkmb9\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.422430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.390966 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-envoy\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.422430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.391001 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-socket\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.422430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.391297 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-credential-socket\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.422430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.391953 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-credential-socket\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.422430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.392577 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-data\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.422430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.392808 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-certs\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.422430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.420986 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07d2e44-c567-42b0-a939-3d2f33b2a078-istiod-ca-cert\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.501035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.437844 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-socket\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.501035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.468333 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-podinfo\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.546945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.546682 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-envoy\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.565239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.565201 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkmb9\" (UniqueName: \"kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-kube-api-access-jkmb9\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.595683 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.595595 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" event={"ID":"25ba8b43-1249-48f5-91b1-25bdb0f82dd5","Type":"ContainerStarted","Data":"b26e2e16bd281cbc878817edb620e974488917caa85bc9a608af18fc5bb2e308"} Oct 30 05:06:42.617250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.610871 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-token\") pod \"e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:42.828799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:42.828762 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:43.044535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.044486 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb"] Oct 30 05:06:43.052952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.052843 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e424d0-l68bb"] Oct 30 05:06:43.188119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.188083 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm"] Oct 30 05:06:43.188278 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:06:43.188196 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07d2e44_c567_42b0_a939_3d2f33b2a078.slice/crio-7201d0e8eb2fdbcab57e980cd4be9f09bf5af497ffb7578e03d5e88ff44a1356 WatchSource:0}: Error finding container 7201d0e8eb2fdbcab57e980cd4be9f09bf5af497ffb7578e03d5e88ff44a1356: Status 404 returned error can't find the container with id 7201d0e8eb2fdbcab57e980cd4be9f09bf5af497ffb7578e03d5e88ff44a1356 Oct 30 05:06:43.196739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.196705 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:06:43.196836 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.196772 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:06:43.196836 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.196799 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:06:43.584909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.584870 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231fbddd-367a-493a-ab21-983bfb881d2b" path="/var/lib/kubelet/pods/231fbddd-367a-493a-ab21-983bfb881d2b/volumes" Oct 30 05:06:43.585720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.585692 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d81aeb2-a9f4-4061-ad64-4fc317582efb" path="/var/lib/kubelet/pods/5d81aeb2-a9f4-4061-ad64-4fc317582efb/volumes" Oct 30 05:06:43.607428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.607393 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" event={"ID":"25ba8b43-1249-48f5-91b1-25bdb0f82dd5","Type":"ContainerStarted","Data":"725ba193af45deb48e58b9bb1195662d681c0cb43f781c3320077e60eebde23e"} Oct 30 05:06:43.607697 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.607477 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" Oct 30 05:06:43.610285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.610252 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" event={"ID":"a07d2e44-c567-42b0-a939-3d2f33b2a078","Type":"ContainerStarted","Data":"16d1d7930e931dca3ed9ec2af156e8c6c7389331ea66571f92579cfc639912b5"} Oct 30 05:06:43.610438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.610313 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" event={"ID":"a07d2e44-c567-42b0-a939-3d2f33b2a078","Type":"ContainerStarted","Data":"7201d0e8eb2fdbcab57e980cd4be9f09bf5af497ffb7578e03d5e88ff44a1356"} Oct 30 05:06:43.657037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.656974 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" podStartSLOduration=2.656954699 podStartE2EDuration="2.656954699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:43.654038413 +0000 UTC m=+1328.181805199" watchObservedRunningTime="2026-10-30 05:06:43.656954699 +0000 UTC m=+1328.184721486" Oct 30 05:06:43.709063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.709007 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" podStartSLOduration=1.7089927 podStartE2EDuration="1.7089927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:43.705262657 +0000 UTC m=+1328.233029469" watchObservedRunningTime="2026-10-30 05:06:43.7089927 +0000 UTC m=+1328.236759486" Oct 30 05:06:43.836620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:43.836480 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:44.836171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:44.836130 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:45.636419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:45.636389 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:45.637211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:45.637181 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:06:48.165266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.165234 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj"] Oct 30 05:06:48.170052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.170032 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" Oct 30 05:06:48.188391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.188368 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj"] Oct 30 05:06:48.196871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.196846 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-528d67-dockercfg-qhbx9\"" Oct 30 05:06:48.197045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.196913 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-528d67\"" Oct 30 05:06:48.354383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.330149 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7bf576-f225-4be6-9f22-99fc919b5576-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj\" (UID: \"1a7bf576-f225-4be6-9f22-99fc919b5576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" Oct 30 05:06:48.355454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.355415 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxx89\" (UniqueName: \"kubernetes.io/projected/1a7bf576-f225-4be6-9f22-99fc919b5576-kube-api-access-gxx89\") pod \"payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj\" (UID: \"1a7bf576-f225-4be6-9f22-99fc919b5576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" Oct 30 05:06:48.386417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.385590 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk"] Oct 30 05:06:48.475045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.466034 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk"] Oct 30 05:06:48.475045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.466165 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" Oct 30 05:06:48.491379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.487675 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxx89\" (UniqueName: \"kubernetes.io/projected/1a7bf576-f225-4be6-9f22-99fc919b5576-kube-api-access-gxx89\") pod \"payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj\" (UID: \"1a7bf576-f225-4be6-9f22-99fc919b5576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" Oct 30 05:06:48.491379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.487821 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7bf576-f225-4be6-9f22-99fc919b5576-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj\" (UID: \"1a7bf576-f225-4be6-9f22-99fc919b5576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" Oct 30 05:06:48.491379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.487940 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fspd9\" (UniqueName: \"kubernetes.io/projected/f53cf9b1-71c2-418b-947c-9c2771e68784-kube-api-access-fspd9\") pod \"payload-processing-e2e-api-a-528d67-5998c8d466-84qzk\" (UID: \"f53cf9b1-71c2-418b-947c-9c2771e68784\") " pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" Oct 30 05:06:48.491379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.488029 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53cf9b1-71c2-418b-947c-9c2771e68784-plugins-config-volume\") pod \"payload-processing-e2e-api-a-528d67-5998c8d466-84qzk\" (UID: \"f53cf9b1-71c2-418b-947c-9c2771e68784\") " pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" Oct 30 05:06:48.491379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.489250 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7bf576-f225-4be6-9f22-99fc919b5576-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj\" (UID: \"1a7bf576-f225-4be6-9f22-99fc919b5576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" Oct 30 05:06:48.573128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.573095 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxx89\" (UniqueName: \"kubernetes.io/projected/1a7bf576-f225-4be6-9f22-99fc919b5576-kube-api-access-gxx89\") pod \"payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj\" (UID: \"1a7bf576-f225-4be6-9f22-99fc919b5576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" Oct 30 05:06:48.591218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.591175 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53cf9b1-71c2-418b-947c-9c2771e68784-plugins-config-volume\") pod \"payload-processing-e2e-api-a-528d67-5998c8d466-84qzk\" (UID: \"f53cf9b1-71c2-418b-947c-9c2771e68784\") " pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" Oct 30 05:06:48.591822 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.591795 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fspd9\" (UniqueName: \"kubernetes.io/projected/f53cf9b1-71c2-418b-947c-9c2771e68784-kube-api-access-fspd9\") pod \"payload-processing-e2e-api-a-528d67-5998c8d466-84qzk\" (UID: \"f53cf9b1-71c2-418b-947c-9c2771e68784\") " pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" Oct 30 05:06:48.593004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.592979 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53cf9b1-71c2-418b-947c-9c2771e68784-plugins-config-volume\") pod \"payload-processing-e2e-api-a-528d67-5998c8d466-84qzk\" (UID: \"f53cf9b1-71c2-418b-947c-9c2771e68784\") " pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" Oct 30 05:06:48.622554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.620112 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fspd9\" (UniqueName: \"kubernetes.io/projected/f53cf9b1-71c2-418b-947c-9c2771e68784-kube-api-access-fspd9\") pod \"payload-processing-e2e-api-a-528d67-5998c8d466-84qzk\" (UID: \"f53cf9b1-71c2-418b-947c-9c2771e68784\") " pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" Oct 30 05:06:48.786330 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.786254 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" Oct 30 05:06:48.873566 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:48.873532 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" Oct 30 05:06:49.148420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:49.142134 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj"] Oct 30 05:06:49.200257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:49.200230 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk"] Oct 30 05:06:49.201113 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:06:49.201080 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53cf9b1_71c2_418b_947c_9c2771e68784.slice/crio-b355cc2f7f598a994addb0e0601a6027669ad70e6231f9a7a91c2f70206de28f WatchSource:0}: Error finding container b355cc2f7f598a994addb0e0601a6027669ad70e6231f9a7a91c2f70206de28f: Status 404 returned error can't find the container with id b355cc2f7f598a994addb0e0601a6027669ad70e6231f9a7a91c2f70206de28f Oct 30 05:06:49.697212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:49.697177 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" event={"ID":"1a7bf576-f225-4be6-9f22-99fc919b5576","Type":"ContainerStarted","Data":"5bd799c68e5529b9fe88a8e30f67f8b8f45f4c5e2fa271f4151327c039b06621"} Oct 30 05:06:49.697411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:49.697233 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" event={"ID":"1a7bf576-f225-4be6-9f22-99fc919b5576","Type":"ContainerStarted","Data":"95c324ed1a7c137e6407db7484bec5abfd12d541a0c7194b9d60a7498381063c"} Oct 30 05:06:49.697411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:49.697280 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" Oct 30 05:06:49.699171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:49.699147 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" event={"ID":"f53cf9b1-71c2-418b-947c-9c2771e68784","Type":"ContainerStarted","Data":"e3399684f5d420e5e1486cf1b0122c0c394beb30a8f627608f56ae5094796806"} Oct 30 05:06:49.699327 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:49.699185 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" event={"ID":"f53cf9b1-71c2-418b-947c-9c2771e68784","Type":"ContainerStarted","Data":"b355cc2f7f598a994addb0e0601a6027669ad70e6231f9a7a91c2f70206de28f"} Oct 30 05:06:49.699327 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:49.699220 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" Oct 30 05:06:49.745928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:49.745881 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" podStartSLOduration=1.745863575 podStartE2EDuration="1.745863575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:49.74309396 +0000 UTC m=+1334.270860747" watchObservedRunningTime="2026-10-30 05:06:49.745863575 +0000 UTC m=+1334.273630377" Oct 30 05:06:49.785257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:49.785204 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" podStartSLOduration=1.7851904090000001 podStartE2EDuration="1.785190409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:49.784756075 +0000 UTC m=+1334.312522867" watchObservedRunningTime="2026-10-30 05:06:49.785190409 +0000 UTC m=+1334.312957195" Oct 30 05:06:54.648094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:54.648030 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" Oct 30 05:06:56.062400 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:06:56.062025 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:06:58.395844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:58.395802 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx"] Oct 30 05:06:58.400636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:58.400598 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" Oct 30 05:06:58.449839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:58.449793 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ef844616-dockercfg-ljh29\"" Oct 30 05:06:58.454947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:58.454913 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx"] Oct 30 05:06:58.460668 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:58.460650 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ef844616\"" Oct 30 05:06:58.559378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:58.553779 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad008a25-4aea-4e83-99b1-1a1e7d03a877-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx\" (UID: \"ad008a25-4aea-4e83-99b1-1a1e7d03a877\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" Oct 30 05:06:58.559378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:58.554095 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8n8m\" (UniqueName: \"kubernetes.io/projected/ad008a25-4aea-4e83-99b1-1a1e7d03a877-kube-api-access-g8n8m\") pod \"payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx\" (UID: \"ad008a25-4aea-4e83-99b1-1a1e7d03a877\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" Oct 30 05:06:58.663662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:58.663568 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad008a25-4aea-4e83-99b1-1a1e7d03a877-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx\" (UID: \"ad008a25-4aea-4e83-99b1-1a1e7d03a877\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" Oct 30 05:06:58.665940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:58.665899 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8n8m\" (UniqueName: \"kubernetes.io/projected/ad008a25-4aea-4e83-99b1-1a1e7d03a877-kube-api-access-g8n8m\") pod \"payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx\" (UID: \"ad008a25-4aea-4e83-99b1-1a1e7d03a877\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" Oct 30 05:06:58.735954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:58.665530 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad008a25-4aea-4e83-99b1-1a1e7d03a877-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx\" (UID: \"ad008a25-4aea-4e83-99b1-1a1e7d03a877\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" Oct 30 05:06:58.972467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:58.967325 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8n8m\" (UniqueName: \"kubernetes.io/projected/ad008a25-4aea-4e83-99b1-1a1e7d03a877-kube-api-access-g8n8m\") pod \"payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx\" (UID: \"ad008a25-4aea-4e83-99b1-1a1e7d03a877\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" Oct 30 05:06:59.059842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:59.059791 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" Oct 30 05:06:59.397820 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:06:59.397785 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad008a25_4aea_4e83_99b1_1a1e7d03a877.slice/crio-c9637723dc6ae5285d9da1774c6a78b837e21887cf2ac82c0bd590b82f08e345 WatchSource:0}: Error finding container c9637723dc6ae5285d9da1774c6a78b837e21887cf2ac82c0bd590b82f08e345: Status 404 returned error can't find the container with id c9637723dc6ae5285d9da1774c6a78b837e21887cf2ac82c0bd590b82f08e345 Oct 30 05:06:59.406243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:59.406214 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx"] Oct 30 05:06:59.989832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:59.989749 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" event={"ID":"ad008a25-4aea-4e83-99b1-1a1e7d03a877","Type":"ContainerStarted","Data":"7fb2b498dbb5e061c2b4a9f95374699d657d10a7fa650c56fc2d38938c593443"} Oct 30 05:06:59.989832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:59.989805 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" event={"ID":"ad008a25-4aea-4e83-99b1-1a1e7d03a877","Type":"ContainerStarted","Data":"c9637723dc6ae5285d9da1774c6a78b837e21887cf2ac82c0bd590b82f08e345"} Oct 30 05:06:59.990123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:06:59.990106 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" Oct 30 05:07:00.039244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:00.039175 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" podStartSLOduration=2.039156862 podStartE2EDuration="2.039156862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:00.0375853 +0000 UTC m=+1344.565352085" watchObservedRunningTime="2026-10-30 05:07:00.039156862 +0000 UTC m=+1344.566923650" Oct 30 05:07:00.775792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:00.770885 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" Oct 30 05:07:00.777634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:00.777576 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" Oct 30 05:07:02.202544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:02.202478 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2"] Oct 30 05:07:02.260545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:02.236095 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcd1a95c-tsvg2"] Oct 30 05:07:03.587813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:03.587779 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecfea321-d7af-4ee0-83f2-1aee2c643b01" path="/var/lib/kubelet/pods/ecfea321-d7af-4ee0-83f2-1aee2c643b01/volumes" Oct 30 05:07:06.053264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.052859 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl"] Oct 30 05:07:06.058551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.058525 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" Oct 30 05:07:06.074903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.074874 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl"] Oct 30 05:07:06.083292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.083266 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ac96f3\"" Oct 30 05:07:06.114533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.114510 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch"] Oct 30 05:07:06.119086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.119062 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" Oct 30 05:07:06.130449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.130419 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch"] Oct 30 05:07:06.137699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.137674 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ac96f3-dockercfg-b7kw4\"" Oct 30 05:07:06.141970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.141940 6587 generic.go:356] "Generic (PLEG): container finished" podID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerID="4da638fc4aa0cb8c8c051537dfbe05629172376895c3f0a3de7307e43948bb47" exitCode=0 Oct 30 05:07:06.142319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.142200 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" event={"ID":"fc0a3d86-39e5-4015-baa3-6b09d8713f69","Type":"ContainerDied","Data":"4da638fc4aa0cb8c8c051537dfbe05629172376895c3f0a3de7307e43948bb47"} Oct 30 05:07:06.143147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.142985 6587 scope.go:122] "RemoveContainer" containerID="4da638fc4aa0cb8c8c051537dfbe05629172376895c3f0a3de7307e43948bb47" Oct 30 05:07:06.146431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.145280 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:07:06.146431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.145405 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:07:06.146431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.145456 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:07:06.147790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.147767 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnfs9\" (UniqueName: \"kubernetes.io/projected/07ecdfee-0acb-45f5-b05b-0c5da48c4260-kube-api-access-lnfs9\") pod \"payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl\" (UID: \"07ecdfee-0acb-45f5-b05b-0c5da48c4260\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" Oct 30 05:07:06.148064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.148035 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msr9\" (UniqueName: \"kubernetes.io/projected/30889d31-1669-4b3a-885f-c0ae680dcacf-kube-api-access-9msr9\") pod \"payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch\" (UID: \"30889d31-1669-4b3a-885f-c0ae680dcacf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" Oct 30 05:07:06.148160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.148077 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30889d31-1669-4b3a-885f-c0ae680dcacf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch\" (UID: \"30889d31-1669-4b3a-885f-c0ae680dcacf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" Oct 30 05:07:06.148221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.148178 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ecdfee-0acb-45f5-b05b-0c5da48c4260-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl\" (UID: \"07ecdfee-0acb-45f5-b05b-0c5da48c4260\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" Oct 30 05:07:06.297389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.262130 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfs9\" (UniqueName: \"kubernetes.io/projected/07ecdfee-0acb-45f5-b05b-0c5da48c4260-kube-api-access-lnfs9\") pod \"payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl\" (UID: \"07ecdfee-0acb-45f5-b05b-0c5da48c4260\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" Oct 30 05:07:06.297389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.262376 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msr9\" (UniqueName: \"kubernetes.io/projected/30889d31-1669-4b3a-885f-c0ae680dcacf-kube-api-access-9msr9\") pod \"payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch\" (UID: \"30889d31-1669-4b3a-885f-c0ae680dcacf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" Oct 30 05:07:06.297389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.262425 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30889d31-1669-4b3a-885f-c0ae680dcacf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch\" (UID: \"30889d31-1669-4b3a-885f-c0ae680dcacf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" Oct 30 05:07:06.297389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.262547 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ecdfee-0acb-45f5-b05b-0c5da48c4260-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl\" (UID: \"07ecdfee-0acb-45f5-b05b-0c5da48c4260\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" Oct 30 05:07:06.297389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.278791 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30889d31-1669-4b3a-885f-c0ae680dcacf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch\" (UID: \"30889d31-1669-4b3a-885f-c0ae680dcacf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" Oct 30 05:07:06.297789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.297556 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ecdfee-0acb-45f5-b05b-0c5da48c4260-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl\" (UID: \"07ecdfee-0acb-45f5-b05b-0c5da48c4260\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" Oct 30 05:07:06.515320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.515238 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfs9\" (UniqueName: \"kubernetes.io/projected/07ecdfee-0acb-45f5-b05b-0c5da48c4260-kube-api-access-lnfs9\") pod \"payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl\" (UID: \"07ecdfee-0acb-45f5-b05b-0c5da48c4260\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" Oct 30 05:07:06.519216 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.519179 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msr9\" (UniqueName: \"kubernetes.io/projected/30889d31-1669-4b3a-885f-c0ae680dcacf-kube-api-access-9msr9\") pod \"payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch\" (UID: \"30889d31-1669-4b3a-885f-c0ae680dcacf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" Oct 30 05:07:06.675027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.674982 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" Oct 30 05:07:06.730960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:06.730716 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" Oct 30 05:07:07.042339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:07.042302 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl"] Oct 30 05:07:07.162405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:07.155876 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch"] Oct 30 05:07:07.172094 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:07:07.172014 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30889d31_1669_4b3a_885f_c0ae680dcacf.slice/crio-4a7982c6ea864cfc5f777c7005d90deda9a27306602317d864929c369b2845e2 WatchSource:0}: Error finding container 4a7982c6ea864cfc5f777c7005d90deda9a27306602317d864929c369b2845e2: Status 404 returned error can't find the container with id 4a7982c6ea864cfc5f777c7005d90deda9a27306602317d864929c369b2845e2 Oct 30 05:07:07.179168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:07.179140 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" event={"ID":"07ecdfee-0acb-45f5-b05b-0c5da48c4260","Type":"ContainerStarted","Data":"2d8a6c285985ab3f8f8e04af3a6fcacbbaec209dd68962be7bf2bfb4fdb26c59"} Oct 30 05:07:07.182022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:07.181976 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" event={"ID":"fc0a3d86-39e5-4015-baa3-6b09d8713f69","Type":"ContainerStarted","Data":"049b1b26089e05a2bdf2ea9a3f9cb5cb4c8d24be4a26316b3c6b09bf7b7e134a"} Oct 30 05:07:07.891839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:07.891806 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:07:07.892185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:07.892163 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 05:07:07.892333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:07.892308 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 05:07:08.055546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:08.055512 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm"] Oct 30 05:07:08.063101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:08.063065 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad250be-vbvfm"] Oct 30 05:07:08.208748 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:08.208650 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" event={"ID":"07ecdfee-0acb-45f5-b05b-0c5da48c4260","Type":"ContainerStarted","Data":"4871878ef909b3425c5c353ad956f6fc515542490381158a74bbfb7249804f6b"} Oct 30 05:07:08.210070 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:08.210046 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" Oct 30 05:07:08.241382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:08.237414 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" event={"ID":"30889d31-1669-4b3a-885f-c0ae680dcacf","Type":"ContainerStarted","Data":"9c09efd7c4eaba7ec276a8ef90cf1adc4ff1fca0c941c9bbb3a98e9b4b2f4b5e"} Oct 30 05:07:08.241382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:08.237551 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" event={"ID":"30889d31-1669-4b3a-885f-c0ae680dcacf","Type":"ContainerStarted","Data":"4a7982c6ea864cfc5f777c7005d90deda9a27306602317d864929c369b2845e2"} Oct 30 05:07:08.241382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:08.239241 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" Oct 30 05:07:08.291159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:08.291067 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" podStartSLOduration=2.291046657 podStartE2EDuration="2.291046657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:08.266440037 +0000 UTC m=+1352.794206825" watchObservedRunningTime="2026-10-30 05:07:08.291046657 +0000 UTC m=+1352.818813446" Oct 30 05:07:08.320391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:08.314889 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" podStartSLOduration=2.3148707330000002 podStartE2EDuration="2.314870733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:08.31325272 +0000 UTC m=+1352.841019507" watchObservedRunningTime="2026-10-30 05:07:08.314870733 +0000 UTC m=+1352.842637520" Oct 30 05:07:08.890566 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:08.890532 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 05:07:08.890755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:08.890598 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 05:07:09.584515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:09.584484 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c092142b-a383-485a-9c1a-cfd750e2f9e8" path="/var/lib/kubelet/pods/c092142b-a383-485a-9c1a-cfd750e2f9e8/volumes" Oct 30 05:07:09.890674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:09.890585 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 05:07:09.890674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:09.890653 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 05:07:10.890185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:10.890154 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 05:07:10.890626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:10.890214 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 05:07:11.007659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:11.007618 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" Oct 30 05:07:11.890373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:11.890328 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 05:07:11.890873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:11.890441 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 05:07:12.830384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:12.828464 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:07:12.896378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:12.891120 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 05:07:12.896378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:12.891203 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 05:07:13.895548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:13.895515 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:07:13.896860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:13.896835 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:07:15.032881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:15.032845 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr"] Oct 30 05:07:15.046636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:15.046603 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-091ef8a9-7cprr"] Oct 30 05:07:15.584214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:15.584180 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd17d06-25da-4cbc-9e7a-a744f882a585" path="/var/lib/kubelet/pods/4bd17d06-25da-4cbc-9e7a-a744f882a585/volumes" Oct 30 05:07:19.274812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:19.274721 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" Oct 30 05:07:19.275210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:19.275089 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" Oct 30 05:07:20.035902 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:20.035861 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b"] Oct 30 05:07:20.046578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:20.046544 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5117d40-9t74b"] Oct 30 05:07:21.583945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:21.583910 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a063d68-3337-4c89-9630-c48676fae9b6" path="/var/lib/kubelet/pods/1a063d68-3337-4c89-9630-c48676fae9b6/volumes" Oct 30 05:07:24.411521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.411475 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng"] Oct 30 05:07:24.423285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.423251 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" Oct 30 05:07:24.434429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.434390 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng"] Oct 30 05:07:24.440796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.440770 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:07:24.441075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.441058 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:07:24.507624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.507587 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:07:24.507794 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.507735 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:07:24.523160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.523122 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng\" (UID: \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" Oct 30 05:07:24.523319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.523179 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng\" (UID: \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" Oct 30 05:07:24.523439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.523413 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5mp\" (UniqueName: \"kubernetes.io/projected/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-kube-api-access-dk5mp\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng\" (UID: \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" Oct 30 05:07:24.625545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.624949 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng\" (UID: \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" Oct 30 05:07:24.625545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.625027 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng\" (UID: \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" Oct 30 05:07:24.625545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.625227 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5mp\" (UniqueName: \"kubernetes.io/projected/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-kube-api-access-dk5mp\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng\" (UID: \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" Oct 30 05:07:24.627074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.627047 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng\" (UID: \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" Oct 30 05:07:24.629261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.629234 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng\" (UID: \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" Oct 30 05:07:24.652023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.651992 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5mp\" (UniqueName: \"kubernetes.io/projected/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-kube-api-access-dk5mp\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng\" (UID: \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" Oct 30 05:07:24.755559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:24.755474 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" Oct 30 05:07:25.057431 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:07:25.057347 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e986bb_e7e5_49c6_ad8c_d9e6fe615bf1.slice/crio-b339e38087a16e5f53bc08caa8669d72023a927363ef826e43d4b75ad7e5fb2f WatchSource:0}: Error finding container b339e38087a16e5f53bc08caa8669d72023a927363ef826e43d4b75ad7e5fb2f: Status 404 returned error can't find the container with id b339e38087a16e5f53bc08caa8669d72023a927363ef826e43d4b75ad7e5fb2f Oct 30 05:07:25.060344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:25.060314 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng"] Oct 30 05:07:25.497909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:25.497865 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" event={"ID":"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1","Type":"ContainerStarted","Data":"cd17e506d6545e4aa6295e4bf0157f90b23814be0b69c4d16755a2cbebc15d08"} Oct 30 05:07:25.498313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:25.497930 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" event={"ID":"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1","Type":"ContainerStarted","Data":"b339e38087a16e5f53bc08caa8669d72023a927363ef826e43d4b75ad7e5fb2f"} Oct 30 05:07:25.545047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:25.544986 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" podStartSLOduration=1.544968324 podStartE2EDuration="1.544968324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:25.540324893 +0000 UTC m=+1370.068091679" watchObservedRunningTime="2026-10-30 05:07:25.544968324 +0000 UTC m=+1370.072735112" Oct 30 05:07:25.995910 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:07:25.995873 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:07:26.508112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:26.508074 6587 generic.go:356] "Generic (PLEG): container finished" podID="d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1" containerID="cd17e506d6545e4aa6295e4bf0157f90b23814be0b69c4d16755a2cbebc15d08" exitCode=0 Oct 30 05:07:26.508546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:26.508179 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" event={"ID":"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1","Type":"ContainerDied","Data":"cd17e506d6545e4aa6295e4bf0157f90b23814be0b69c4d16755a2cbebc15d08"} Oct 30 05:07:27.060461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:27.060423 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c"] Oct 30 05:07:27.071535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:27.071499 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a649b-5xq7c"] Oct 30 05:07:27.588859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:27.587568 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="032e3aad-0862-4063-b39c-0eb858bf7584" path="/var/lib/kubelet/pods/032e3aad-0862-4063-b39c-0eb858bf7584/volumes" Oct 30 05:07:27.720563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:27.720531 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" Oct 30 05:07:27.894318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:27.894282 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-openshift-ca\") pod \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\" (UID: \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\") " Oct 30 05:07:27.894534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:27.894331 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-kube-api-access-dk5mp\" (UniqueName: \"kubernetes.io/projected/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-kube-api-access-dk5mp\") pod \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\" (UID: \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\") " Oct 30 05:07:27.894534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:27.894391 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-xks-ca\") pod \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\" (UID: \"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1\") " Oct 30 05:07:27.894801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:27.894775 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-openshift-ca" pod "d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1" (UID: "d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:07:27.895408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:27.895330 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:07:27.896858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:27.896828 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-xks-ca" pod "d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1" (UID: "d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:07:27.896973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:27.896890 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-kube-api-access-dk5mp" pod "d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1" (UID: "d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1"). InnerVolumeSpecName "kube-api-access-dk5mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:07:27.997049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:27.997012 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk5mp\" (UniqueName: \"kubernetes.io/projected/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-kube-api-access-dk5mp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:07:27.997049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:27.997051 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:07:28.529292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:28.529252 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" Oct 30 05:07:28.529292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:28.529272 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng" event={"ID":"d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1","Type":"ContainerDied","Data":"b339e38087a16e5f53bc08caa8669d72023a927363ef826e43d4b75ad7e5fb2f"} Oct 30 05:07:28.529758 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:28.529347 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b339e38087a16e5f53bc08caa8669d72023a927363ef826e43d4b75ad7e5fb2f" Oct 30 05:07:32.038425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:32.038370 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf"] Oct 30 05:07:32.051519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:32.051491 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8e1afcc-cq5cf"] Oct 30 05:07:33.585301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:33.585254 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="214e7e6b-359a-4c41-8a15-7fa47ae39c78" path="/var/lib/kubelet/pods/214e7e6b-359a-4c41-8a15-7fa47ae39c78/volumes" Oct 30 05:07:34.900441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:34.900402 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2"] Oct 30 05:07:36.675815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:36.675784 6587 scope.go:122] "RemoveContainer" containerID="6a9ccd435ca7a9a95ec26bf4b87dc06d70f6eb6b42662e19905a90e18912e578" Oct 30 05:07:36.689440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:36.689414 6587 scope.go:122] "RemoveContainer" containerID="a22a26a03bd54dcaa78b0c0696688594f342c8f31647e229eba287de83628091" Oct 30 05:07:36.700959 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:36.700924 6587 scope.go:122] "RemoveContainer" containerID="dc5c2960751ff27c51e46521f8a3855b32e2af2cbaa6b56e9bf0ab631adb0b64" Oct 30 05:07:36.712213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:36.712183 6587 scope.go:122] "RemoveContainer" containerID="db0ed870d46a158004a9f962edf724429cdf28df31586e5435b63cfb7a11ac5c" Oct 30 05:07:36.724141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:36.724124 6587 scope.go:122] "RemoveContainer" containerID="9ce4d5bf55227fac29003397ee9787ccbb1e05ce8c096dc1401c96e314a0bb46" Oct 30 05:07:36.734427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:36.734381 6587 scope.go:122] "RemoveContainer" containerID="2d4b9f0ade9b8bc113b153ae371a36f9976a8b62c0c8dcbca1d9c2cd125fadfa" Oct 30 05:07:39.901214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:39.901167 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" podUID="52b8904b-0f62-4abb-8ec3-b43c6cb628ff" containerName="payload-processing" containerID="cri-o://32a648306b78587fb4e63e49b06c299afb03f083e585fa675932a19b168e0851" gracePeriod=25 Oct 30 05:07:41.039190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:41.039139 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5"] Oct 30 05:07:41.050495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:41.050455 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c614e530-kbsc5"] Oct 30 05:07:41.584662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:41.584630 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf5b2b3-385d-4cbc-9693-e5acb0bb561a" path="/var/lib/kubelet/pods/eaf5b2b3-385d-4cbc-9693-e5acb0bb561a/volumes" Oct 30 05:07:43.033401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:43.033323 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq"] Oct 30 05:07:43.062079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:43.062043 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f33b40d8-7cjpq"] Oct 30 05:07:43.584893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:43.584859 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a498169d-c8ad-49b6-b836-e6898140eb8e" path="/var/lib/kubelet/pods/a498169d-c8ad-49b6-b836-e6898140eb8e/volumes" Oct 30 05:07:46.460594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.460560 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c"] Oct 30 05:07:46.461913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.461887 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1" containerName="revoke-keys" Oct 30 05:07:46.462009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.461915 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1" containerName="revoke-keys" Oct 30 05:07:46.462115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.462103 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1" containerName="revoke-keys" Oct 30 05:07:46.472135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.472100 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" Oct 30 05:07:46.483195 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.483162 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c"] Oct 30 05:07:46.489581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.489142 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:07:46.503002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.502961 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:07:46.503180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.503031 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:07:46.503180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.503145 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:07:46.549977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.549932 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ce054d4-kk55c\" (UID: \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" Oct 30 05:07:46.550312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.550291 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w6df\" (UniqueName: \"kubernetes.io/projected/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-kube-api-access-2w6df\") pod \"maas-api-revoke-sub-5ce054d4-kk55c\" (UID: \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" Oct 30 05:07:46.550410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.550351 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ce054d4-kk55c\" (UID: \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" Oct 30 05:07:46.651713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.651675 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ce054d4-kk55c\" (UID: \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" Oct 30 05:07:46.651915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.651853 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ce054d4-kk55c\" (UID: \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" Oct 30 05:07:46.652216 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.652194 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6df\" (UniqueName: \"kubernetes.io/projected/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-kube-api-access-2w6df\") pod \"maas-api-revoke-sub-5ce054d4-kk55c\" (UID: \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" Oct 30 05:07:46.652426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.652401 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ce054d4-kk55c\" (UID: \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" Oct 30 05:07:46.654418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.654381 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ce054d4-kk55c\" (UID: \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" Oct 30 05:07:46.673680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.673657 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w6df\" (UniqueName: \"kubernetes.io/projected/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-kube-api-access-2w6df\") pod \"maas-api-revoke-sub-5ce054d4-kk55c\" (UID: \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" Oct 30 05:07:46.796929 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:46.796855 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" Oct 30 05:07:47.151644 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:07:47.136062 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc981fbe5_95c5_4a9c_94da_cfc66d8665fc.slice/crio-6fca78da1bb779c616b118e9ed3772d460101f0d3393c4177b9d453ba5a8d63b WatchSource:0}: Error finding container 6fca78da1bb779c616b118e9ed3772d460101f0d3393c4177b9d453ba5a8d63b: Status 404 returned error can't find the container with id 6fca78da1bb779c616b118e9ed3772d460101f0d3393c4177b9d453ba5a8d63b Oct 30 05:07:47.151644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:47.141410 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c"] Oct 30 05:07:47.795133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:47.795042 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" event={"ID":"c981fbe5-95c5-4a9c-94da-cfc66d8665fc","Type":"ContainerStarted","Data":"de94da50167894e51e4c1019c2f3fb5dd65172075374fa55bede2a777db994f0"} Oct 30 05:07:47.795133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:47.795105 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" event={"ID":"c981fbe5-95c5-4a9c-94da-cfc66d8665fc","Type":"ContainerStarted","Data":"6fca78da1bb779c616b118e9ed3772d460101f0d3393c4177b9d453ba5a8d63b"} Oct 30 05:07:47.844441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:47.844379 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" podStartSLOduration=1.844347926 podStartE2EDuration="1.844347926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:47.839432642 +0000 UTC m=+1392.367199432" watchObservedRunningTime="2026-10-30 05:07:47.844347926 +0000 UTC m=+1392.372114711" Oct 30 05:07:48.775424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:48.775353 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" podUID="52b8904b-0f62-4abb-8ec3-b43c6cb628ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:07:48.804983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:48.804949 6587 generic.go:356] "Generic (PLEG): container finished" podID="c981fbe5-95c5-4a9c-94da-cfc66d8665fc" containerID="de94da50167894e51e4c1019c2f3fb5dd65172075374fa55bede2a777db994f0" exitCode=0 Oct 30 05:07:48.805401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:48.805031 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" event={"ID":"c981fbe5-95c5-4a9c-94da-cfc66d8665fc","Type":"ContainerDied","Data":"de94da50167894e51e4c1019c2f3fb5dd65172075374fa55bede2a777db994f0"} Oct 30 05:07:50.272664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:50.272622 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" Oct 30 05:07:50.388765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:50.388650 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-xks-ca\") pod \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\" (UID: \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\") " Oct 30 05:07:50.388947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:50.388773 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-kube-api-access-2w6df\" (UniqueName: \"kubernetes.io/projected/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-kube-api-access-2w6df\") pod \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\" (UID: \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\") " Oct 30 05:07:50.388947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:50.388841 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-openshift-ca\") pod \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\" (UID: \"c981fbe5-95c5-4a9c-94da-cfc66d8665fc\") " Oct 30 05:07:50.389999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:50.389964 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-openshift-ca" pod "c981fbe5-95c5-4a9c-94da-cfc66d8665fc" (UID: "c981fbe5-95c5-4a9c-94da-cfc66d8665fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:07:50.392126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:50.392047 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:07:50.392126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:50.392101 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-xks-ca" pod "c981fbe5-95c5-4a9c-94da-cfc66d8665fc" (UID: "c981fbe5-95c5-4a9c-94da-cfc66d8665fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:07:50.392257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:50.392135 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-kube-api-access-2w6df" pod "c981fbe5-95c5-4a9c-94da-cfc66d8665fc" (UID: "c981fbe5-95c5-4a9c-94da-cfc66d8665fc"). InnerVolumeSpecName "kube-api-access-2w6df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:07:50.494640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:50.494593 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:07:50.494640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:50.494645 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w6df\" (UniqueName: \"kubernetes.io/projected/c981fbe5-95c5-4a9c-94da-cfc66d8665fc-kube-api-access-2w6df\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:07:50.894413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:50.893982 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" event={"ID":"c981fbe5-95c5-4a9c-94da-cfc66d8665fc","Type":"ContainerDied","Data":"6fca78da1bb779c616b118e9ed3772d460101f0d3393c4177b9d453ba5a8d63b"} Oct 30 05:07:50.894413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:50.894045 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fca78da1bb779c616b118e9ed3772d460101f0d3393c4177b9d453ba5a8d63b" Oct 30 05:07:50.894413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:50.894217 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c" Oct 30 05:07:53.802544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.802185 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:07:53.804988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.804952 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx"] Oct 30 05:07:53.806325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.806257 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c981fbe5-95c5-4a9c-94da-cfc66d8665fc" containerName="revoke-keys" Oct 30 05:07:53.806325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.806290 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c981fbe5-95c5-4a9c-94da-cfc66d8665fc" containerName="revoke-keys" Oct 30 05:07:53.806576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.806541 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c981fbe5-95c5-4a9c-94da-cfc66d8665fc" containerName="revoke-keys" Oct 30 05:07:53.814585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.814565 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx"] Oct 30 05:07:53.814850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.814792 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:53.892962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.892926 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-543d08-openshift-default-dockercfg-ntbnx\"" Oct 30 05:07:53.895294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.895258 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-socket\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:53.895520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.895499 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-istio-token\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:53.895673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.895657 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-credential-socket\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:53.895973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.895955 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484fe4bc-1006-4745-ac99-408b72f728d8-istiod-ca-cert\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:53.896112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.896097 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrp7s\" (UniqueName: \"kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-kube-api-access-mrp7s\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:53.896430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.896396 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-certs\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:53.910375 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.908127 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-envoy\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:53.910375 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.908170 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484fe4bc-1006-4745-ac99-408b72f728d8-istio-podinfo\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:53.910375 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:53.908285 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-data\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.033033 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-istio-token\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.033099 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-credential-socket\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.033270 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484fe4bc-1006-4745-ac99-408b72f728d8-istiod-ca-cert\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.033325 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrp7s\" (UniqueName: \"kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-kube-api-access-mrp7s\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.033411 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-certs\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.033553 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-envoy\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.033580 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484fe4bc-1006-4745-ac99-408b72f728d8-istio-podinfo\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.033689 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-data\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.033799 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-socket\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.034948 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-credential-socket\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.035627 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484fe4bc-1006-4745-ac99-408b72f728d8-istiod-ca-cert\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.036172 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-certs\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.046832 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-socket\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.145169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.047430 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-data\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.233767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.233540 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-envoy\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.237545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.235218 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484fe4bc-1006-4745-ac99-408b72f728d8-istio-podinfo\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.302197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.302158 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrp7s\" (UniqueName: \"kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-kube-api-access-mrp7s\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.321064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.310001 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-istio-token\") pod \"e2e-api-a-543d08-openshift-default-f45f9c965-597sx\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.371383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.370723 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:54.555039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.555000 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g"] Oct 30 05:07:54.566682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.566638 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:54.587237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.587208 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-645aef-openshift-default-dockercfg-tqhn4\"" Oct 30 05:07:54.973113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.973073 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g"] Oct 30 05:07:54.976508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.973999 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-socket\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:54.976895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.976870 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szslv\" (UniqueName: \"kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-kube-api-access-szslv\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:54.977318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.977120 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-envoy\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:54.977530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.977507 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-credential-socket\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:54.977806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.977785 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istiod-ca-cert\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:54.978235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.978212 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-token\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:54.981557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.981530 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-certs\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:54.981836 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.981816 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-podinfo\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:54.982177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:54.982156 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-data\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.192712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.192656 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:07:55.196708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.196650 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-certs\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.199505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.199475 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-podinfo\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.199791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.199774 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-data\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.199979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.199964 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-socket\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.200145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.200131 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szslv\" (UniqueName: \"kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-kube-api-access-szslv\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.200273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.200260 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-envoy\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.200420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.200396 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-credential-socket\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.200551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.200536 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istiod-ca-cert\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.200803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.200787 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-token\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.201877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.201853 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-certs\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.210174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.210132 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:07:55.219384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.211011 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-data\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.219384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.211241 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-socket\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.219384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.211276 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:07:55.219384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.218896 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-credential-socket\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.250427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.249545 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istiod-ca-cert\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.284383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.281962 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:07:55.493851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.492272 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-token\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.497227 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.497185 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szslv\" (UniqueName: \"kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-kube-api-access-szslv\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.500749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.500675 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-envoy\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.503603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.503551 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-podinfo\") pod \"e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.504284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.504256 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:55.813458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.813339 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:07:55.813773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.813674 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:07:55.814261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.813885 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:07:55.815920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:55.815634 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx"] Oct 30 05:07:56.106688 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:07:56.106537 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:07:56.302046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:56.302011 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g"] Oct 30 05:07:56.306431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:56.306391 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:07:56.307868 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:56.307282 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:07:56.307868 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:56.307323 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:07:56.545922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:56.545879 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" event={"ID":"8a0adf45-f6fc-41c4-9c8b-142f595fd03f","Type":"ContainerStarted","Data":"d8f3cf73fabced40c34aaebc2469c0bf39cd032e7bb7fed0cb776967246d4f27"} Oct 30 05:07:56.549761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:56.549727 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" event={"ID":"484fe4bc-1006-4745-ac99-408b72f728d8","Type":"ContainerStarted","Data":"fe9d7b90177dff739bb0d249895d600784cb509bda8762551a3ad665e6a01077"} Oct 30 05:07:56.553466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:56.553431 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" event={"ID":"484fe4bc-1006-4745-ac99-408b72f728d8","Type":"ContainerStarted","Data":"1bd57d11874092dead9aa355cc047203050c072e779723f17cd15b6284192353"} Oct 30 05:07:56.638616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:56.636178 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" podStartSLOduration=3.636158258 podStartE2EDuration="3.636158258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:56.635004667 +0000 UTC m=+1401.162771469" watchObservedRunningTime="2026-10-30 05:07:56.636158258 +0000 UTC m=+1401.163925045" Oct 30 05:07:57.110247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:57.110066 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm"] Oct 30 05:07:57.130381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:57.114680 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5399c41-qdbvm"] Oct 30 05:07:57.388439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:57.385766 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:57.395655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:57.395624 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:57.697832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:57.692651 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e878d6-d55f-4ca9-a3d8-66e6d8379172" path="/var/lib/kubelet/pods/90e878d6-d55f-4ca9-a3d8-66e6d8379172/volumes" Oct 30 05:07:57.697832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:57.693215 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" event={"ID":"8a0adf45-f6fc-41c4-9c8b-142f595fd03f","Type":"ContainerStarted","Data":"98dd0487492a898bea84251bab5e1fb2938a4491fc2a5c450e55dc5296df4501"} Oct 30 05:07:57.697832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:57.693297 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:57.697832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:57.693403 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:07:57.786615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:57.785560 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" podStartSLOduration=3.785545262 podStartE2EDuration="3.785545262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:57.773606332 +0000 UTC m=+1402.301373119" watchObservedRunningTime="2026-10-30 05:07:57.785545262 +0000 UTC m=+1402.313312048" Oct 30 05:07:58.504576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:58.504547 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:58.510936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:58.510911 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:58.765071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:58.764983 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:07:58.782863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:58.782823 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" podUID="52b8904b-0f62-4abb-8ec3-b43c6cb628ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:07:58.807384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:07:58.806601 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:08:00.272867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.272840 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" Oct 30 05:08:00.348753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.348709 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-kube-api-access-2zntc\" (UniqueName: \"kubernetes.io/projected/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-kube-api-access-2zntc\") pod \"52b8904b-0f62-4abb-8ec3-b43c6cb628ff\" (UID: \"52b8904b-0f62-4abb-8ec3-b43c6cb628ff\") " Oct 30 05:08:00.348970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.348953 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-plugins-config-volume\") pod \"52b8904b-0f62-4abb-8ec3-b43c6cb628ff\" (UID: \"52b8904b-0f62-4abb-8ec3-b43c6cb628ff\") " Oct 30 05:08:00.350316 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.350282 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-plugins-config-volume" pod "52b8904b-0f62-4abb-8ec3-b43c6cb628ff" (UID: "52b8904b-0f62-4abb-8ec3-b43c6cb628ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:00.355976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.355944 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-kube-api-access-2zntc" pod "52b8904b-0f62-4abb-8ec3-b43c6cb628ff" (UID: "52b8904b-0f62-4abb-8ec3-b43c6cb628ff"). InnerVolumeSpecName "kube-api-access-2zntc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:00.452242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.452197 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zntc\" (UniqueName: \"kubernetes.io/projected/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-kube-api-access-2zntc\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:00.452242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.452238 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b8904b-0f62-4abb-8ec3-b43c6cb628ff-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:00.803594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.803492 6587 generic.go:356] "Generic (PLEG): container finished" podID="52b8904b-0f62-4abb-8ec3-b43c6cb628ff" containerID="32a648306b78587fb4e63e49b06c299afb03f083e585fa675932a19b168e0851" exitCode=0 Oct 30 05:08:00.804188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.804149 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" Oct 30 05:08:00.804650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.804611 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" event={"ID":"52b8904b-0f62-4abb-8ec3-b43c6cb628ff","Type":"ContainerDied","Data":"32a648306b78587fb4e63e49b06c299afb03f083e585fa675932a19b168e0851"} Oct 30 05:08:00.804759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.804679 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2" event={"ID":"52b8904b-0f62-4abb-8ec3-b43c6cb628ff","Type":"ContainerDied","Data":"181cc5ab1770a72da3b4a40fefc94bdf8b0f4b7c5b6fb64988df03d53a4c8ec8"} Oct 30 05:08:00.804759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.804711 6587 scope.go:122] "RemoveContainer" containerID="32a648306b78587fb4e63e49b06c299afb03f083e585fa675932a19b168e0851" Oct 30 05:08:00.871391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.870516 6587 scope.go:122] "RemoveContainer" containerID="32a648306b78587fb4e63e49b06c299afb03f083e585fa675932a19b168e0851" Oct 30 05:08:00.884835 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:08:00.884725 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a648306b78587fb4e63e49b06c299afb03f083e585fa675932a19b168e0851\": container with ID starting with 32a648306b78587fb4e63e49b06c299afb03f083e585fa675932a19b168e0851 not found: ID does not exist" containerID="32a648306b78587fb4e63e49b06c299afb03f083e585fa675932a19b168e0851" Oct 30 05:08:00.884835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.884757 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a648306b78587fb4e63e49b06c299afb03f083e585fa675932a19b168e0851"} err="failed to get container status \"32a648306b78587fb4e63e49b06c299afb03f083e585fa675932a19b168e0851\": rpc error: code = NotFound desc = could not find container \"32a648306b78587fb4e63e49b06c299afb03f083e585fa675932a19b168e0851\": container with ID starting with 32a648306b78587fb4e63e49b06c299afb03f083e585fa675932a19b168e0851 not found: ID does not exist" Oct 30 05:08:00.906718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.906679 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2"] Oct 30 05:08:00.920017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:00.919973 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2"] Oct 30 05:08:01.684382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:01.679458 6587 patch_prober.go:29] interesting pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:08:01.684382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:01.679544 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:08:01.807925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:01.803797 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b8904b-0f62-4abb-8ec3-b43c6cb628ff" path="/var/lib/kubelet/pods/52b8904b-0f62-4abb-8ec3-b43c6cb628ff/volumes" Oct 30 05:08:02.043108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:02.043029 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl"] Oct 30 05:08:02.054838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:02.054808 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d52ecbf-wjzcl"] Oct 30 05:08:03.491941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:03.491899 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk"] Oct 30 05:08:03.492864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:03.492839 6587 cpu_manager.go:414] "containerMap: removing container" podUID="52b8904b-0f62-4abb-8ec3-b43c6cb628ff" containerName="payload-processing" Oct 30 05:08:03.492864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:03.492864 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b8904b-0f62-4abb-8ec3-b43c6cb628ff" containerName="payload-processing" Oct 30 05:08:03.493055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:03.493045 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b8904b-0f62-4abb-8ec3-b43c6cb628ff" containerName="payload-processing" Oct 30 05:08:03.819431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:03.819399 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" Oct 30 05:08:03.935462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:03.913792 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-645aef-dockercfg-x2x4x\"" Oct 30 05:08:03.935462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:03.913914 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-645aef\"" Oct 30 05:08:03.945795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:03.945758 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd420f62-adff-401e-8c12-6bf7eab929e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk\" (UID: \"cd420f62-adff-401e-8c12-6bf7eab929e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" Oct 30 05:08:03.946102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:03.946077 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl49h\" (UniqueName: \"kubernetes.io/projected/cd420f62-adff-401e-8c12-6bf7eab929e6-kube-api-access-jl49h\") pod \"payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk\" (UID: \"cd420f62-adff-401e-8c12-6bf7eab929e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" Oct 30 05:08:03.985559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:03.985408 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d0d0ca-0c75-4e62-b22f-4eeba8b557b7" path="/var/lib/kubelet/pods/80d0d0ca-0c75-4e62-b22f-4eeba8b557b7/volumes" Oct 30 05:08:03.986304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:03.986281 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk"] Oct 30 05:08:03.986542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:03.986523 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs"] Oct 30 05:08:04.057058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.051187 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl49h\" (UniqueName: \"kubernetes.io/projected/cd420f62-adff-401e-8c12-6bf7eab929e6-kube-api-access-jl49h\") pod \"payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk\" (UID: \"cd420f62-adff-401e-8c12-6bf7eab929e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" Oct 30 05:08:04.057243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.057149 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd420f62-adff-401e-8c12-6bf7eab929e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk\" (UID: \"cd420f62-adff-401e-8c12-6bf7eab929e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" Oct 30 05:08:04.066398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.064827 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd420f62-adff-401e-8c12-6bf7eab929e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk\" (UID: \"cd420f62-adff-401e-8c12-6bf7eab929e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" Oct 30 05:08:04.241726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.241620 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl49h\" (UniqueName: \"kubernetes.io/projected/cd420f62-adff-401e-8c12-6bf7eab929e6-kube-api-access-jl49h\") pod \"payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk\" (UID: \"cd420f62-adff-401e-8c12-6bf7eab929e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" Oct 30 05:08:04.334475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.334436 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs"] Oct 30 05:08:04.335212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.335187 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" Oct 30 05:08:04.350276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.350242 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" Oct 30 05:08:04.362184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.362155 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm49c\" (UniqueName: \"kubernetes.io/projected/67e16cca-13be-49d8-95a0-22b479aea9bd-kube-api-access-fm49c\") pod \"payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs\" (UID: \"67e16cca-13be-49d8-95a0-22b479aea9bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" Oct 30 05:08:04.362641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.362611 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e16cca-13be-49d8-95a0-22b479aea9bd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs\" (UID: \"67e16cca-13be-49d8-95a0-22b479aea9bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" Oct 30 05:08:04.464473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.464436 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e16cca-13be-49d8-95a0-22b479aea9bd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs\" (UID: \"67e16cca-13be-49d8-95a0-22b479aea9bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" Oct 30 05:08:04.464714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.464692 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm49c\" (UniqueName: \"kubernetes.io/projected/67e16cca-13be-49d8-95a0-22b479aea9bd-kube-api-access-fm49c\") pod \"payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs\" (UID: \"67e16cca-13be-49d8-95a0-22b479aea9bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" Oct 30 05:08:04.465594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.465565 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e16cca-13be-49d8-95a0-22b479aea9bd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs\" (UID: \"67e16cca-13be-49d8-95a0-22b479aea9bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" Oct 30 05:08:04.487152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.487120 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm49c\" (UniqueName: \"kubernetes.io/projected/67e16cca-13be-49d8-95a0-22b479aea9bd-kube-api-access-fm49c\") pod \"payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs\" (UID: \"67e16cca-13be-49d8-95a0-22b479aea9bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" Oct 30 05:08:04.617969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:04.617931 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" Oct 30 05:08:05.869452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:05.869411 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk"] Oct 30 05:08:05.870294 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:08:05.870260 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd420f62_adff_401e_8c12_6bf7eab929e6.slice/crio-c28c873742ef3dac42051335667f23195167dd17569fe92f1bb104c32d684084 WatchSource:0}: Error finding container c28c873742ef3dac42051335667f23195167dd17569fe92f1bb104c32d684084: Status 404 returned error can't find the container with id c28c873742ef3dac42051335667f23195167dd17569fe92f1bb104c32d684084 Oct 30 05:08:06.008381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:06.008323 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs"] Oct 30 05:08:06.039857 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:06.039819 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" event={"ID":"cd420f62-adff-401e-8c12-6bf7eab929e6","Type":"ContainerStarted","Data":"c28c873742ef3dac42051335667f23195167dd17569fe92f1bb104c32d684084"} Oct 30 05:08:07.050859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:07.050818 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" event={"ID":"67e16cca-13be-49d8-95a0-22b479aea9bd","Type":"ContainerStarted","Data":"c981afade5c0c5896712bc7330f7e706df3000fb38301ec970da31aace415a1d"} Oct 30 05:08:07.051296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:07.050914 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" event={"ID":"67e16cca-13be-49d8-95a0-22b479aea9bd","Type":"ContainerStarted","Data":"6a0e1481254cb65d612b5a18f60e15a9260225e7a7559c2ffd5a618401e95f21"} Oct 30 05:08:07.051296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:07.050950 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" Oct 30 05:08:07.052906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:07.052874 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" event={"ID":"cd420f62-adff-401e-8c12-6bf7eab929e6","Type":"ContainerStarted","Data":"9d58b27b686009c3fa686cdd2909b4819f4f5f82f8fe1ec6aee345b5fe7ec2ee"} Oct 30 05:08:07.053043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:07.052969 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" Oct 30 05:08:07.095609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:07.095557 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" podStartSLOduration=4.095542756 podStartE2EDuration="4.095542756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:07.091687222 +0000 UTC m=+1411.619454009" watchObservedRunningTime="2026-10-30 05:08:07.095542756 +0000 UTC m=+1411.623309542" Oct 30 05:08:07.135789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:07.135729 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" podStartSLOduration=4.135711549 podStartE2EDuration="4.135711549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:07.133017849 +0000 UTC m=+1411.660784635" watchObservedRunningTime="2026-10-30 05:08:07.135711549 +0000 UTC m=+1411.663478338" Oct 30 05:08:09.037190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:09.037149 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g"] Oct 30 05:08:09.048975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:09.048935 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5e5e447-sng2g"] Oct 30 05:08:09.469526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:09.469484 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:08:09.469722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:09.469565 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:08:09.587055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:09.587017 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d78db49-762e-45df-a7fc-eadb81752d98" path="/var/lib/kubelet/pods/7d78db49-762e-45df-a7fc-eadb81752d98/volumes" Oct 30 05:08:13.164826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.164788 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt"] Oct 30 05:08:13.217991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.217951 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt"] Oct 30 05:08:13.218181 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.218086 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.235371 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.234642 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-288e01-openshift-default-dockercfg-h6j69\"" Oct 30 05:08:13.329243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.316424 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-socket\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.329243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.316490 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-envoy\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.329243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.316547 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-istio-token\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.329243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.316575 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn47c\" (UniqueName: \"kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-kube-api-access-kn47c\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.329243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.316776 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-data\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.329243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.316902 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793b0190-5368-47f1-ae26-e6f736e9e878-istiod-ca-cert\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.329243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.317039 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-certs\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.329243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.317098 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-credential-socket\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.329243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.317168 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793b0190-5368-47f1-ae26-e6f736e9e878-istio-podinfo\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.428433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.427854 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-istio-token\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.428433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.427907 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn47c\" (UniqueName: \"kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-kube-api-access-kn47c\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.428433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.428080 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-data\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.428433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.428203 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793b0190-5368-47f1-ae26-e6f736e9e878-istiod-ca-cert\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.428433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.428321 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-certs\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.428433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.428389 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-credential-socket\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.428902 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.428449 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793b0190-5368-47f1-ae26-e6f736e9e878-istio-podinfo\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.428902 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.428509 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-socket\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.428902 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.428544 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-envoy\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.434382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.429480 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-data\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.434382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.431552 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-credential-socket\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.434382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.432213 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793b0190-5368-47f1-ae26-e6f736e9e878-istiod-ca-cert\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.434382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.432486 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-certs\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.434382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.433107 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-socket\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.453192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.446874 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-envoy\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.453192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.446874 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793b0190-5368-47f1-ae26-e6f736e9e878-istio-podinfo\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.469223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.466558 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn47c\" (UniqueName: \"kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-kube-api-access-kn47c\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.474389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.473947 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-istio-token\") pod \"e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:13.559449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:13.559400 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:14.049892 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:08:14.049857 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793b0190_5368_47f1_ae26_e6f736e9e878.slice/crio-0ceb167894b27ee48ed475edf607d35d647a34fd400e02fd5b7f80f30bf53b6f WatchSource:0}: Error finding container 0ceb167894b27ee48ed475edf607d35d647a34fd400e02fd5b7f80f30bf53b6f: Status 404 returned error can't find the container with id 0ceb167894b27ee48ed475edf607d35d647a34fd400e02fd5b7f80f30bf53b6f Oct 30 05:08:14.051269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:14.051241 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt"] Oct 30 05:08:14.052845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:14.052800 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:08:14.052960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:14.052875 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:08:14.052960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:14.052905 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:08:14.226655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:14.226611 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" event={"ID":"793b0190-5368-47f1-ae26-e6f736e9e878","Type":"ContainerStarted","Data":"37e7d3b9792a464344d3b75dfd735121bd8ef138dfed4e4e7db6d480e2c5e16c"} Oct 30 05:08:14.227069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:14.226685 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" event={"ID":"793b0190-5368-47f1-ae26-e6f736e9e878","Type":"ContainerStarted","Data":"0ceb167894b27ee48ed475edf607d35d647a34fd400e02fd5b7f80f30bf53b6f"} Oct 30 05:08:14.276278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:14.276219 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" podStartSLOduration=1.276206042 podStartE2EDuration="1.276206042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:14.273607825 +0000 UTC m=+1418.801374611" watchObservedRunningTime="2026-10-30 05:08:14.276206042 +0000 UTC m=+1418.803972828" Oct 30 05:08:14.655052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:14.560159 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:15.240958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.240861 6587 generic.go:356] "Generic (PLEG): container finished" podID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerID="dbea4f0a03d664e8fedc1643d39031ffb12a762d11a6b57d5017dd93102c5568" exitCode=0 Oct 30 05:08:15.240958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.240940 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" event={"ID":"33684ba3-e578-4f9f-bafd-7f9faefb39ad","Type":"ContainerDied","Data":"dbea4f0a03d664e8fedc1643d39031ffb12a762d11a6b57d5017dd93102c5568"} Oct 30 05:08:15.241577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.241557 6587 scope.go:122] "RemoveContainer" containerID="dbea4f0a03d664e8fedc1643d39031ffb12a762d11a6b57d5017dd93102c5568" Oct 30 05:08:15.243334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.243307 6587 generic.go:356] "Generic (PLEG): container finished" podID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerID="16d1d7930e931dca3ed9ec2af156e8c6c7389331ea66571f92579cfc639912b5" exitCode=0 Oct 30 05:08:15.243511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.243391 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" event={"ID":"a07d2e44-c567-42b0-a939-3d2f33b2a078","Type":"ContainerDied","Data":"16d1d7930e931dca3ed9ec2af156e8c6c7389331ea66571f92579cfc639912b5"} Oct 30 05:08:15.243636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.243611 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:08:15.243711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.243678 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:08:15.243765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.243709 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:08:15.244000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.243980 6587 scope.go:122] "RemoveContainer" containerID="16d1d7930e931dca3ed9ec2af156e8c6c7389331ea66571f92579cfc639912b5" Oct 30 05:08:15.245633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.245605 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:08:15.245740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.245669 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:08:15.245740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.245699 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:08:15.567234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.567198 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:15.636989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:15.636833 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:08:16.288460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:16.288078 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" event={"ID":"a07d2e44-c567-42b0-a939-3d2f33b2a078","Type":"ContainerStarted","Data":"ecdc88c635db71add9b411c6f8c4e091aa7fa08fd538f400e9eb489a1aaf5763"} Oct 30 05:08:16.290824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:16.290792 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" event={"ID":"33684ba3-e578-4f9f-bafd-7f9faefb39ad","Type":"ContainerStarted","Data":"4981368cf600221dad417e44ddbb9d6350f43e2f928833ddf1c045533d75f6f7"} Oct 30 05:08:16.291016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:16.291003 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:16.292233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:16.292213 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:08:16.735499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:16.730580 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:08:16.735499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:16.733658 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:08:16.735499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:16.733721 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 05:08:16.836703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:16.836672 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:08:16.892194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:16.887556 6587 patch_prober.go:29] interesting pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:08:16.892521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:16.892490 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:08:17.669382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:17.668670 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:08:17.669382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:17.668742 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 05:08:17.837813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:17.837771 6587 patch_prober.go:29] interesting pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:08:17.838128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:17.838097 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:08:17.961438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:17.959455 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt"] Oct 30 05:08:17.983420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:17.983385 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt"] Oct 30 05:08:17.983607 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:17.983516 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" Oct 30 05:08:18.000671 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.000641 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:08:18.000840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.000729 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:08:18.000840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.000814 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:08:18.014550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.014524 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:08:18.107990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.107709 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91a43fea-xlkkt\" (UID: \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" Oct 30 05:08:18.107990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.107907 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddz57\" (UniqueName: \"kubernetes.io/projected/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-kube-api-access-ddz57\") pod \"maas-api-revoke-sub-91a43fea-xlkkt\" (UID: \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" Oct 30 05:08:18.108200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.108138 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91a43fea-xlkkt\" (UID: \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" Oct 30 05:08:18.164382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.163904 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" Oct 30 05:08:18.174435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.165643 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" Oct 30 05:08:18.209465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.209429 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddz57\" (UniqueName: \"kubernetes.io/projected/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-kube-api-access-ddz57\") pod \"maas-api-revoke-sub-91a43fea-xlkkt\" (UID: \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" Oct 30 05:08:18.209699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.209677 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91a43fea-xlkkt\" (UID: \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" Oct 30 05:08:18.209842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.209817 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91a43fea-xlkkt\" (UID: \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" Oct 30 05:08:18.210605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.210543 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91a43fea-xlkkt\" (UID: \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" Oct 30 05:08:18.212434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.212382 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91a43fea-xlkkt\" (UID: \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" Oct 30 05:08:18.265381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.251626 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddz57\" (UniqueName: \"kubernetes.io/projected/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-kube-api-access-ddz57\") pod \"maas-api-revoke-sub-91a43fea-xlkkt\" (UID: \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" Oct 30 05:08:18.304311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.304091 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" Oct 30 05:08:18.689876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.689826 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:08:18.690251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.689914 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 05:08:18.854581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.847792 6587 patch_prober.go:29] interesting pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:08:18.854581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.847888 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:08:18.988044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:18.988012 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt"] Oct 30 05:08:18.990187 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:08:18.990153 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5641a03d_3b4b_4372_8dd8_6d508a0e4ec5.slice/crio-7fbb34e3aad613d7ff6567293435dfaee1396eeed878177ba910d01ff071370c WatchSource:0}: Error finding container 7fbb34e3aad613d7ff6567293435dfaee1396eeed878177ba910d01ff071370c: Status 404 returned error can't find the container with id 7fbb34e3aad613d7ff6567293435dfaee1396eeed878177ba910d01ff071370c Oct 30 05:08:19.075118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:19.075054 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j"] Oct 30 05:08:19.081653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:19.081614 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49933366-l826j"] Oct 30 05:08:19.503886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:19.503845 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" event={"ID":"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5","Type":"ContainerStarted","Data":"7fbb34e3aad613d7ff6567293435dfaee1396eeed878177ba910d01ff071370c"} Oct 30 05:08:19.693459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:19.675703 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:08:19.693459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:19.675760 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 05:08:19.846044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:19.846005 6587 patch_prober.go:29] interesting pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:08:19.846844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:19.846808 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:08:20.354385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.298538 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f66128-c09c-4032-9a79-3149b07e4812" path="/var/lib/kubelet/pods/69f66128-c09c-4032-9a79-3149b07e4812/volumes" Oct 30 05:08:20.354385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.299058 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p"] Oct 30 05:08:20.354385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.306268 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" Oct 30 05:08:20.354385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.325699 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-288e01\"" Oct 30 05:08:20.378921 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.359632 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-288e01-dockercfg-4f5xl\"" Oct 30 05:08:20.397768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.397594 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p"] Oct 30 05:08:20.406771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.406737 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf8h2\" (UniqueName: \"kubernetes.io/projected/0e8198e2-ee22-44d5-9ea8-a9d40e635880-kube-api-access-nf8h2\") pod \"payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p\" (UID: \"0e8198e2-ee22-44d5-9ea8-a9d40e635880\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" Oct 30 05:08:20.406921 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.406898 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8198e2-ee22-44d5-9ea8-a9d40e635880-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p\" (UID: \"0e8198e2-ee22-44d5-9ea8-a9d40e635880\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" Oct 30 05:08:20.509537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.509503 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8198e2-ee22-44d5-9ea8-a9d40e635880-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p\" (UID: \"0e8198e2-ee22-44d5-9ea8-a9d40e635880\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" Oct 30 05:08:20.509806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.509793 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf8h2\" (UniqueName: \"kubernetes.io/projected/0e8198e2-ee22-44d5-9ea8-a9d40e635880-kube-api-access-nf8h2\") pod \"payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p\" (UID: \"0e8198e2-ee22-44d5-9ea8-a9d40e635880\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" Oct 30 05:08:20.510717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.510693 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8198e2-ee22-44d5-9ea8-a9d40e635880-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p\" (UID: \"0e8198e2-ee22-44d5-9ea8-a9d40e635880\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" Oct 30 05:08:20.535629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.535592 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" event={"ID":"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5","Type":"ContainerStarted","Data":"a6c8015c03515e355df19b0bba0cf66040d8c1dcfd942e5e13643d7639b888af"} Oct 30 05:08:20.576285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.576249 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf8h2\" (UniqueName: \"kubernetes.io/projected/0e8198e2-ee22-44d5-9ea8-a9d40e635880-kube-api-access-nf8h2\") pod \"payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p\" (UID: \"0e8198e2-ee22-44d5-9ea8-a9d40e635880\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" Oct 30 05:08:20.594943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.594880 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" podStartSLOduration=3.594859812 podStartE2EDuration="3.594859812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:20.576966285 +0000 UTC m=+1425.104733073" watchObservedRunningTime="2026-10-30 05:08:20.594859812 +0000 UTC m=+1425.122626599" Oct 30 05:08:20.666439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.665167 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:08:20.666439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.665229 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 05:08:20.688646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.688620 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" Oct 30 05:08:20.850387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.847530 6587 patch_prober.go:29] interesting pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:08:20.850387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:20.847591 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:08:21.492979 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:08:21.492939 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8198e2_ee22_44d5_9ea8_a9d40e635880.slice/crio-48ffc8f95337ef56542a06fee8f5268c03d408940c9a9070f938b29959789ba2 WatchSource:0}: Error finding container 48ffc8f95337ef56542a06fee8f5268c03d408940c9a9070f938b29959789ba2: Status 404 returned error can't find the container with id 48ffc8f95337ef56542a06fee8f5268c03d408940c9a9070f938b29959789ba2 Oct 30 05:08:21.495269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:21.495243 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p"] Oct 30 05:08:21.596872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:21.596822 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" event={"ID":"0e8198e2-ee22-44d5-9ea8-a9d40e635880","Type":"ContainerStarted","Data":"48ffc8f95337ef56542a06fee8f5268c03d408940c9a9070f938b29959789ba2"} Oct 30 05:08:21.598615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:21.598549 6587 generic.go:356] "Generic (PLEG): container finished" podID="5641a03d-3b4b-4372-8dd8-6d508a0e4ec5" containerID="a6c8015c03515e355df19b0bba0cf66040d8c1dcfd942e5e13643d7639b888af" exitCode=0 Oct 30 05:08:21.598846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:21.598745 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" event={"ID":"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5","Type":"ContainerDied","Data":"a6c8015c03515e355df19b0bba0cf66040d8c1dcfd942e5e13643d7639b888af"} Oct 30 05:08:21.678180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:21.678143 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:08:21.678494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:21.678456 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 05:08:21.856386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:21.855988 6587 patch_prober.go:29] interesting pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:08:21.856386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:21.856046 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:08:22.673381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:22.666913 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 05:08:22.673381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:22.666972 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 05:08:22.696381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:22.682957 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" event={"ID":"0e8198e2-ee22-44d5-9ea8-a9d40e635880","Type":"ContainerStarted","Data":"d755a0688d848cf7683d7f331c4280de6ba9bb67c08eda47db2cf5087451b695"} Oct 30 05:08:22.710611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:22.710581 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" Oct 30 05:08:22.759759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:22.746492 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" podStartSLOduration=3.746472008 podStartE2EDuration="3.746472008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:22.745505659 +0000 UTC m=+1427.273272446" watchObservedRunningTime="2026-10-30 05:08:22.746472008 +0000 UTC m=+1427.274238795" Oct 30 05:08:22.830761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:22.829460 6587 patch_prober.go:29] interesting pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:08:22.830761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:22.829518 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:08:23.267588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.267122 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" Oct 30 05:08:23.376603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.371160 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq"] Oct 30 05:08:23.376603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.371469 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" podUID="896a9789-3a1d-4ec0-abfb-39131100ba0d" containerName="istio-proxy" containerID="cri-o://9c2072d7b1782a9bd14cd49f90cfe60b44d032e7654e22d2a265304002344def" gracePeriod=30 Oct 30 05:08:23.394398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.393734 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn"] Oct 30 05:08:23.394398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.394155 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" podUID="363a5464-b736-4f6c-800c-38b13778716c" containerName="istio-proxy" containerID="cri-o://07eca600f75f9e641e2a95b9105e55bc88852c65779bf795b05125c5c8898a42" gracePeriod=30 Oct 30 05:08:23.405660 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.405626 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk"] Oct 30 05:08:23.405911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.405883 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" podUID="1dabb599-1beb-451d-b03d-87ed3942f5ff" containerName="istio-proxy" containerID="cri-o://5e5453bebf90d6e50081d246ea1513b63fb55c71dddf9011ed4a3c419c26a4bc" gracePeriod=30 Oct 30 05:08:23.434114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.411189 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-kube-api-access-ddz57\" (UniqueName: \"kubernetes.io/projected/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-kube-api-access-ddz57\") pod \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\" (UID: \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\") " Oct 30 05:08:23.434114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.411245 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-openshift-ca\") pod \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\" (UID: \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\") " Oct 30 05:08:23.434114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.411505 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-xks-ca\") pod \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\" (UID: \"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5\") " Oct 30 05:08:23.434114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.413795 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-openshift-ca" pod "5641a03d-3b4b-4372-8dd8-6d508a0e4ec5" (UID: "5641a03d-3b4b-4372-8dd8-6d508a0e4ec5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:23.434114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.415179 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:23.434114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.415211 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l"] Oct 30 05:08:23.434114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.415618 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" podUID="407c363e-e3da-40c7-adb6-0d5e8880fd55" containerName="istio-proxy" containerID="cri-o://09e74d2b7299a5b9e37967a45f9742092e61eb988905b0a44966a8f4dabce27f" gracePeriod=30 Oct 30 05:08:23.434114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.427576 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-kube-api-access-ddz57" pod "5641a03d-3b4b-4372-8dd8-6d508a0e4ec5" (UID: "5641a03d-3b4b-4372-8dd8-6d508a0e4ec5"). InnerVolumeSpecName "kube-api-access-ddz57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:23.434114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.429736 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:08:23.443586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.443509 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-xks-ca" pod "5641a03d-3b4b-4372-8dd8-6d508a0e4ec5" (UID: "5641a03d-3b4b-4372-8dd8-6d508a0e4ec5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:08:23.561378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.560942 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:23.561378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.560978 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddz57\" (UniqueName: \"kubernetes.io/projected/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5-kube-api-access-ddz57\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:23.665256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.665221 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:08:23.695755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.695726 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" Oct 30 05:08:23.695984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.695787 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt" event={"ID":"5641a03d-3b4b-4372-8dd8-6d508a0e4ec5","Type":"ContainerDied","Data":"7fbb34e3aad613d7ff6567293435dfaee1396eeed878177ba910d01ff071370c"} Oct 30 05:08:23.696137 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.696006 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fbb34e3aad613d7ff6567293435dfaee1396eeed878177ba910d01ff071370c" Oct 30 05:08:23.697441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.697420 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:08:23.866782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.865755 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:08:23.870503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.870242 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:08:23.871674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:23.871477 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:08:25.991143 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:08:25.991109 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:08:26.031352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:26.031312 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf"] Oct 30 05:08:26.044841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:26.044796 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf2229c-kzbtf"] Oct 30 05:08:27.584745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:27.584713 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb90856-8a71-4042-a36b-d0d3d6529ad2" path="/var/lib/kubelet/pods/afb90856-8a71-4042-a36b-d0d3d6529ad2/volumes" Oct 30 05:08:28.133265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.133225 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft"] Oct 30 05:08:28.134650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.134625 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5641a03d-3b4b-4372-8dd8-6d508a0e4ec5" containerName="revoke-keys" Oct 30 05:08:28.134744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.134651 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5641a03d-3b4b-4372-8dd8-6d508a0e4ec5" containerName="revoke-keys" Oct 30 05:08:28.134883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.134870 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5641a03d-3b4b-4372-8dd8-6d508a0e4ec5" containerName="revoke-keys" Oct 30 05:08:28.138685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.138661 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" Oct 30 05:08:28.159826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.159796 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft"] Oct 30 05:08:28.164191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.164165 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4225df\"" Oct 30 05:08:28.164191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.164185 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4225df-dockercfg-5jfhp\"" Oct 30 05:08:28.405465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.404307 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m7jq\" (UniqueName: \"kubernetes.io/projected/e4567cee-33c8-4e3d-a858-6b7f1082b779-kube-api-access-8m7jq\") pod \"payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft\" (UID: \"e4567cee-33c8-4e3d-a858-6b7f1082b779\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" Oct 30 05:08:28.405465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.404808 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4567cee-33c8-4e3d-a858-6b7f1082b779-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft\" (UID: \"e4567cee-33c8-4e3d-a858-6b7f1082b779\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" Oct 30 05:08:28.446426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.440542 6587 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:08:28.446641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.446494 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" podUID="1dabb599-1beb-451d-b03d-87ed3942f5ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:08:28.513459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.507951 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4567cee-33c8-4e3d-a858-6b7f1082b779-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft\" (UID: \"e4567cee-33c8-4e3d-a858-6b7f1082b779\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" Oct 30 05:08:28.513459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.508093 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m7jq\" (UniqueName: \"kubernetes.io/projected/e4567cee-33c8-4e3d-a858-6b7f1082b779-kube-api-access-8m7jq\") pod \"payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft\" (UID: \"e4567cee-33c8-4e3d-a858-6b7f1082b779\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" Oct 30 05:08:28.538413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.536951 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4567cee-33c8-4e3d-a858-6b7f1082b779-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft\" (UID: \"e4567cee-33c8-4e3d-a858-6b7f1082b779\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" Oct 30 05:08:28.560428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.554478 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m7jq\" (UniqueName: \"kubernetes.io/projected/e4567cee-33c8-4e3d-a858-6b7f1082b779-kube-api-access-8m7jq\") pod \"payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft\" (UID: \"e4567cee-33c8-4e3d-a858-6b7f1082b779\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" Oct 30 05:08:28.771598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.771566 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" Oct 30 05:08:28.796455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.796420 6587 generic.go:356] "Generic (PLEG): container finished" podID="896a9789-3a1d-4ec0-abfb-39131100ba0d" containerID="9c2072d7b1782a9bd14cd49f90cfe60b44d032e7654e22d2a265304002344def" exitCode=0 Oct 30 05:08:28.796708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.796677 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" event={"ID":"896a9789-3a1d-4ec0-abfb-39131100ba0d","Type":"ContainerDied","Data":"9c2072d7b1782a9bd14cd49f90cfe60b44d032e7654e22d2a265304002344def"} Oct 30 05:08:28.799208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.799173 6587 generic.go:356] "Generic (PLEG): container finished" podID="407c363e-e3da-40c7-adb6-0d5e8880fd55" containerID="09e74d2b7299a5b9e37967a45f9742092e61eb988905b0a44966a8f4dabce27f" exitCode=0 Oct 30 05:08:28.799335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.799246 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" event={"ID":"407c363e-e3da-40c7-adb6-0d5e8880fd55","Type":"ContainerDied","Data":"09e74d2b7299a5b9e37967a45f9742092e61eb988905b0a44966a8f4dabce27f"} Oct 30 05:08:28.801732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.801707 6587 generic.go:356] "Generic (PLEG): container finished" podID="1dabb599-1beb-451d-b03d-87ed3942f5ff" containerID="5e5453bebf90d6e50081d246ea1513b63fb55c71dddf9011ed4a3c419c26a4bc" exitCode=0 Oct 30 05:08:28.801840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.801777 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" event={"ID":"1dabb599-1beb-451d-b03d-87ed3942f5ff","Type":"ContainerDied","Data":"5e5453bebf90d6e50081d246ea1513b63fb55c71dddf9011ed4a3c419c26a4bc"} Oct 30 05:08:28.804009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.803982 6587 generic.go:356] "Generic (PLEG): container finished" podID="363a5464-b736-4f6c-800c-38b13778716c" containerID="07eca600f75f9e641e2a95b9105e55bc88852c65779bf795b05125c5c8898a42" exitCode=0 Oct 30 05:08:28.804102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.804057 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" event={"ID":"363a5464-b736-4f6c-800c-38b13778716c","Type":"ContainerDied","Data":"07eca600f75f9e641e2a95b9105e55bc88852c65779bf795b05125c5c8898a42"} Oct 30 05:08:28.940197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:28.940165 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 05:08:29.018231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.018194 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/363a5464-b736-4f6c-800c-38b13778716c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363a5464-b736-4f6c-800c-38b13778716c-istio-podinfo\") pod \"363a5464-b736-4f6c-800c-38b13778716c\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " Oct 30 05:08:29.018481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.018455 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-data\") pod \"363a5464-b736-4f6c-800c-38b13778716c\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " Oct 30 05:08:29.018590 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.018543 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-kube-api-access-nlqb7\" (UniqueName: \"kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-kube-api-access-nlqb7\") pod \"363a5464-b736-4f6c-800c-38b13778716c\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " Oct 30 05:08:29.018590 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.018581 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363a5464-b736-4f6c-800c-38b13778716c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363a5464-b736-4f6c-800c-38b13778716c-istiod-ca-cert\") pod \"363a5464-b736-4f6c-800c-38b13778716c\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " Oct 30 05:08:29.018714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.018696 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-socket\") pod \"363a5464-b736-4f6c-800c-38b13778716c\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " Oct 30 05:08:29.018847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.018827 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-credential-socket\") pod \"363a5464-b736-4f6c-800c-38b13778716c\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " Oct 30 05:08:29.018916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.018875 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-certs\") pod \"363a5464-b736-4f6c-800c-38b13778716c\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " Oct 30 05:08:29.018916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.018910 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-envoy\") pod \"363a5464-b736-4f6c-800c-38b13778716c\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " Oct 30 05:08:29.019164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.019140 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-istio-token\" (UniqueName: \"kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-istio-token\") pod \"363a5464-b736-4f6c-800c-38b13778716c\" (UID: \"363a5464-b736-4f6c-800c-38b13778716c\") " Oct 30 05:08:29.020588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.019687 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-socket" pod "363a5464-b736-4f6c-800c-38b13778716c" (UID: "363a5464-b736-4f6c-800c-38b13778716c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.020588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.020190 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.020588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.020447 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-credential-socket" pod "363a5464-b736-4f6c-800c-38b13778716c" (UID: "363a5464-b736-4f6c-800c-38b13778716c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.020872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.020761 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-certs" pod "363a5464-b736-4f6c-800c-38b13778716c" (UID: "363a5464-b736-4f6c-800c-38b13778716c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.021103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.021069 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363a5464-b736-4f6c-800c-38b13778716c-istiod-ca-cert" pod "363a5464-b736-4f6c-800c-38b13778716c" (UID: "363a5464-b736-4f6c-800c-38b13778716c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:29.022242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.021468 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-data" pod "363a5464-b736-4f6c-800c-38b13778716c" (UID: "363a5464-b736-4f6c-800c-38b13778716c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.028806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.028440 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:08:29.041968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.041923 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-envoy" pod "363a5464-b736-4f6c-800c-38b13778716c" (UID: "363a5464-b736-4f6c-800c-38b13778716c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.042096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.042043 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-istio-token" pod "363a5464-b736-4f6c-800c-38b13778716c" (UID: "363a5464-b736-4f6c-800c-38b13778716c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.042211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.042189 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/363a5464-b736-4f6c-800c-38b13778716c-istio-podinfo" pod "363a5464-b736-4f6c-800c-38b13778716c" (UID: "363a5464-b736-4f6c-800c-38b13778716c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:08:29.049254 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.049225 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-kube-api-access-nlqb7" pod "363a5464-b736-4f6c-800c-38b13778716c" (UID: "363a5464-b736-4f6c-800c-38b13778716c"). InnerVolumeSpecName "kube-api-access-nlqb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.127001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.126961 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-podinfo\") pod \"407c363e-e3da-40c7-adb6-0d5e8880fd55\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.131835 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-certs\") pod \"407c363e-e3da-40c7-adb6-0d5e8880fd55\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.131906 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407c363e-e3da-40c7-adb6-0d5e8880fd55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407c363e-e3da-40c7-adb6-0d5e8880fd55-istiod-ca-cert\") pod \"407c363e-e3da-40c7-adb6-0d5e8880fd55\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.131988 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-kube-api-access-bj24b\" (UniqueName: \"kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-kube-api-access-bj24b\") pod \"407c363e-e3da-40c7-adb6-0d5e8880fd55\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.132125 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-data\") pod \"407c363e-e3da-40c7-adb6-0d5e8880fd55\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.132218 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-socket\") pod \"407c363e-e3da-40c7-adb6-0d5e8880fd55\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.132256 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-envoy\") pod \"407c363e-e3da-40c7-adb6-0d5e8880fd55\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.132315 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-credential-socket\") pod \"407c363e-e3da-40c7-adb6-0d5e8880fd55\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.132586 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-token\" (UniqueName: \"kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-token\") pod \"407c363e-e3da-40c7-adb6-0d5e8880fd55\" (UID: \"407c363e-e3da-40c7-adb6-0d5e8880fd55\") " Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.133618 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.133641 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.133656 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.133670 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.133683 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363a5464-b736-4f6c-800c-38b13778716c-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.133695 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363a5464-b736-4f6c-800c-38b13778716c-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.133709 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlqb7\" (UniqueName: \"kubernetes.io/projected/363a5464-b736-4f6c-800c-38b13778716c-kube-api-access-nlqb7\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.133722 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363a5464-b736-4f6c-800c-38b13778716c-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.134582 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-data" pod "407c363e-e3da-40c7-adb6-0d5e8880fd55" (UID: "407c363e-e3da-40c7-adb6-0d5e8880fd55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.134752 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-certs" pod "407c363e-e3da-40c7-adb6-0d5e8880fd55" (UID: "407c363e-e3da-40c7-adb6-0d5e8880fd55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.137430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.134992 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407c363e-e3da-40c7-adb6-0d5e8880fd55-istiod-ca-cert" pod "407c363e-e3da-40c7-adb6-0d5e8880fd55" (UID: "407c363e-e3da-40c7-adb6-0d5e8880fd55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:29.146250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.144867 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-credential-socket" pod "407c363e-e3da-40c7-adb6-0d5e8880fd55" (UID: "407c363e-e3da-40c7-adb6-0d5e8880fd55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.152902 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.150858 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-socket" pod "407c363e-e3da-40c7-adb6-0d5e8880fd55" (UID: "407c363e-e3da-40c7-adb6-0d5e8880fd55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.153591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.153546 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-envoy" pod "407c363e-e3da-40c7-adb6-0d5e8880fd55" (UID: "407c363e-e3da-40c7-adb6-0d5e8880fd55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.161383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.161345 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-kube-api-access-bj24b" pod "407c363e-e3da-40c7-adb6-0d5e8880fd55" (UID: "407c363e-e3da-40c7-adb6-0d5e8880fd55"). InnerVolumeSpecName "kube-api-access-bj24b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.162439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.161936 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:08:29.164694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.164654 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-podinfo" pod "407c363e-e3da-40c7-adb6-0d5e8880fd55" (UID: "407c363e-e3da-40c7-adb6-0d5e8880fd55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:08:29.166691 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.166636 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-token" pod "407c363e-e3da-40c7-adb6-0d5e8880fd55" (UID: "407c363e-e3da-40c7-adb6-0d5e8880fd55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.172408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.172385 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 05:08:29.234863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.234771 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-kube-api-access-j4rdt\" (UniqueName: \"kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-kube-api-access-j4rdt\") pod \"1dabb599-1beb-451d-b03d-87ed3942f5ff\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " Oct 30 05:08:29.235610 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.235585 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-envoy\") pod \"896a9789-3a1d-4ec0-abfb-39131100ba0d\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " Oct 30 05:08:29.235805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.235786 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-socket\") pod \"1dabb599-1beb-451d-b03d-87ed3942f5ff\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " Oct 30 05:08:29.235972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.235956 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-data\") pod \"1dabb599-1beb-451d-b03d-87ed3942f5ff\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " Oct 30 05:08:29.236114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236101 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-kube-api-access-2bt7k\" (UniqueName: \"kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-kube-api-access-2bt7k\") pod \"896a9789-3a1d-4ec0-abfb-39131100ba0d\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " Oct 30 05:08:29.236276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236219 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-socket" pod "1dabb599-1beb-451d-b03d-87ed3942f5ff" (UID: "1dabb599-1beb-451d-b03d-87ed3942f5ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.236276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236272 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-envoy\") pod \"1dabb599-1beb-451d-b03d-87ed3942f5ff\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " Oct 30 05:08:29.236601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236305 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-credential-socket\") pod \"896a9789-3a1d-4ec0-abfb-39131100ba0d\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " Oct 30 05:08:29.236601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236461 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-credential-socket\") pod \"1dabb599-1beb-451d-b03d-87ed3942f5ff\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " Oct 30 05:08:29.236601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236501 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-data\") pod \"896a9789-3a1d-4ec0-abfb-39131100ba0d\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " Oct 30 05:08:29.236601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236540 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-podinfo\") pod \"1dabb599-1beb-451d-b03d-87ed3942f5ff\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " Oct 30 05:08:29.236844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236637 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-certs\") pod \"1dabb599-1beb-451d-b03d-87ed3942f5ff\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " Oct 30 05:08:29.236844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236673 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896a9789-3a1d-4ec0-abfb-39131100ba0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896a9789-3a1d-4ec0-abfb-39131100ba0d-istiod-ca-cert\") pod \"896a9789-3a1d-4ec0-abfb-39131100ba0d\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " Oct 30 05:08:29.236844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236743 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dabb599-1beb-451d-b03d-87ed3942f5ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dabb599-1beb-451d-b03d-87ed3942f5ff-istiod-ca-cert\") pod \"1dabb599-1beb-451d-b03d-87ed3942f5ff\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " Oct 30 05:08:29.236844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236757 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-credential-socket" pod "1dabb599-1beb-451d-b03d-87ed3942f5ff" (UID: "1dabb599-1beb-451d-b03d-87ed3942f5ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.236844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236811 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-token\" (UniqueName: \"kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-token\") pod \"896a9789-3a1d-4ec0-abfb-39131100ba0d\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " Oct 30 05:08:29.237034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236891 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-token\" (UniqueName: \"kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-token\") pod \"1dabb599-1beb-451d-b03d-87ed3942f5ff\" (UID: \"1dabb599-1beb-451d-b03d-87ed3942f5ff\") " Oct 30 05:08:29.237034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236930 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-certs\") pod \"896a9789-3a1d-4ec0-abfb-39131100ba0d\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " Oct 30 05:08:29.237034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.236953 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-credential-socket" pod "896a9789-3a1d-4ec0-abfb-39131100ba0d" (UID: "896a9789-3a1d-4ec0-abfb-39131100ba0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.237263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.237237 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-certs" pod "896a9789-3a1d-4ec0-abfb-39131100ba0d" (UID: "896a9789-3a1d-4ec0-abfb-39131100ba0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.237341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.237266 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896a9789-3a1d-4ec0-abfb-39131100ba0d-istiod-ca-cert" pod "896a9789-3a1d-4ec0-abfb-39131100ba0d" (UID: "896a9789-3a1d-4ec0-abfb-39131100ba0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:29.237341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.237325 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-socket\") pod \"896a9789-3a1d-4ec0-abfb-39131100ba0d\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " Oct 30 05:08:29.237461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.237386 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-podinfo\") pod \"896a9789-3a1d-4ec0-abfb-39131100ba0d\" (UID: \"896a9789-3a1d-4ec0-abfb-39131100ba0d\") " Oct 30 05:08:29.237590 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.237569 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dabb599-1beb-451d-b03d-87ed3942f5ff-istiod-ca-cert" pod "1dabb599-1beb-451d-b03d-87ed3942f5ff" (UID: "1dabb599-1beb-451d-b03d-87ed3942f5ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:29.238581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238561 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238582 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238593 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238601 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238610 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407c363e-e3da-40c7-adb6-0d5e8880fd55-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238618 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj24b\" (UniqueName: \"kubernetes.io/projected/407c363e-e3da-40c7-adb6-0d5e8880fd55-kube-api-access-bj24b\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238628 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238636 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238644 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238652 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238660 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238669 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238677 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407c363e-e3da-40c7-adb6-0d5e8880fd55-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238685 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896a9789-3a1d-4ec0-abfb-39131100ba0d-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.238736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238694 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dabb599-1beb-451d-b03d-87ed3942f5ff-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.239446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.238934 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-data" pod "1dabb599-1beb-451d-b03d-87ed3942f5ff" (UID: "1dabb599-1beb-451d-b03d-87ed3942f5ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.239446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.239198 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-data" pod "896a9789-3a1d-4ec0-abfb-39131100ba0d" (UID: "896a9789-3a1d-4ec0-abfb-39131100ba0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.239446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.239352 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-certs" pod "1dabb599-1beb-451d-b03d-87ed3942f5ff" (UID: "1dabb599-1beb-451d-b03d-87ed3942f5ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.240467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.240438 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-socket" pod "896a9789-3a1d-4ec0-abfb-39131100ba0d" (UID: "896a9789-3a1d-4ec0-abfb-39131100ba0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.241858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.241837 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-podinfo" pod "896a9789-3a1d-4ec0-abfb-39131100ba0d" (UID: "896a9789-3a1d-4ec0-abfb-39131100ba0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:08:29.242292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.242263 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-podinfo" pod "1dabb599-1beb-451d-b03d-87ed3942f5ff" (UID: "1dabb599-1beb-451d-b03d-87ed3942f5ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:08:29.242669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.242642 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-kube-api-access-2bt7k" pod "896a9789-3a1d-4ec0-abfb-39131100ba0d" (UID: "896a9789-3a1d-4ec0-abfb-39131100ba0d"). InnerVolumeSpecName "kube-api-access-2bt7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.242948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.242782 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-envoy" pod "896a9789-3a1d-4ec0-abfb-39131100ba0d" (UID: "896a9789-3a1d-4ec0-abfb-39131100ba0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.242948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.242914 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-token" pod "1dabb599-1beb-451d-b03d-87ed3942f5ff" (UID: "1dabb599-1beb-451d-b03d-87ed3942f5ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.243670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.243648 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-envoy" pod "1dabb599-1beb-451d-b03d-87ed3942f5ff" (UID: "1dabb599-1beb-451d-b03d-87ed3942f5ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:29.244570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.244543 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-kube-api-access-j4rdt" pod "1dabb599-1beb-451d-b03d-87ed3942f5ff" (UID: "1dabb599-1beb-451d-b03d-87ed3942f5ff"). InnerVolumeSpecName "kube-api-access-j4rdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.244659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.244635 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-token" pod "896a9789-3a1d-4ec0-abfb-39131100ba0d" (UID: "896a9789-3a1d-4ec0-abfb-39131100ba0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:29.340138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.340105 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.340138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.340133 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.340332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.340147 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bt7k\" (UniqueName: \"kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-kube-api-access-2bt7k\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.340332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.340157 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.340332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.340168 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.340332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.340175 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.340332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.340184 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dabb599-1beb-451d-b03d-87ed3942f5ff-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.340332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.340192 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.340332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.340200 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.340332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.340209 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896a9789-3a1d-4ec0-abfb-39131100ba0d-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.340332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.340217 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896a9789-3a1d-4ec0-abfb-39131100ba0d-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.340332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.340226 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4rdt\" (UniqueName: \"kubernetes.io/projected/1dabb599-1beb-451d-b03d-87ed3942f5ff-kube-api-access-j4rdt\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:29.353200 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:08:29.353159 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4567cee_33c8_4e3d_a858_6b7f1082b779.slice/crio-c2ab7cd2c989b08f12b5d237769b7318990e10133019bf169f31e46f27b3becb WatchSource:0}: Error finding container c2ab7cd2c989b08f12b5d237769b7318990e10133019bf169f31e46f27b3becb: Status 404 returned error can't find the container with id c2ab7cd2c989b08f12b5d237769b7318990e10133019bf169f31e46f27b3becb Oct 30 05:08:29.355569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.355546 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft"] Oct 30 05:08:29.815233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.815193 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" event={"ID":"407c363e-e3da-40c7-adb6-0d5e8880fd55","Type":"ContainerDied","Data":"84b9c4391fd29d0fda652ce86c9902706a6f7dc66116d31b25cb90a5a386475c"} Oct 30 05:08:29.815774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.815271 6587 scope.go:122] "RemoveContainer" containerID="09e74d2b7299a5b9e37967a45f9742092e61eb988905b0a44966a8f4dabce27f" Oct 30 05:08:29.815774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.815482 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l" Oct 30 05:08:29.840776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.832108 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" Oct 30 05:08:29.840776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.832127 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk" event={"ID":"1dabb599-1beb-451d-b03d-87ed3942f5ff","Type":"ContainerDied","Data":"c640ab3ca3717f769cbfd4fb5c85c1e6403d7cd389ca198442848196c5f74dea"} Oct 30 05:08:29.840776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.838592 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" event={"ID":"363a5464-b736-4f6c-800c-38b13778716c","Type":"ContainerDied","Data":"8faec0360bfeba207cf9ec7cdd02cf607e62aae0ec822a14a9b70d9c1e3be07d"} Oct 30 05:08:29.840776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.838750 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn" Oct 30 05:08:29.846067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.846033 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" event={"ID":"e4567cee-33c8-4e3d-a858-6b7f1082b779","Type":"ContainerStarted","Data":"96c67553b8d37bc9b308bdf5cfaee847415abdafc8ddbe91753c6f9d8a4f3e76"} Oct 30 05:08:29.846223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.846089 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" event={"ID":"e4567cee-33c8-4e3d-a858-6b7f1082b779","Type":"ContainerStarted","Data":"c2ab7cd2c989b08f12b5d237769b7318990e10133019bf169f31e46f27b3becb"} Oct 30 05:08:29.846223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.846119 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" Oct 30 05:08:29.854702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.854672 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" event={"ID":"896a9789-3a1d-4ec0-abfb-39131100ba0d","Type":"ContainerDied","Data":"f263bbf38e7eacad94bb7a876adb06399f1ab5eb3ffbe0c198f4a1ca396e7c9c"} Oct 30 05:08:29.855017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.854955 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq" Oct 30 05:08:29.861767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.861731 6587 scope.go:122] "RemoveContainer" containerID="5e5453bebf90d6e50081d246ea1513b63fb55c71dddf9011ed4a3c419c26a4bc" Oct 30 05:08:29.880097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.880069 6587 scope.go:122] "RemoveContainer" containerID="07eca600f75f9e641e2a95b9105e55bc88852c65779bf795b05125c5c8898a42" Oct 30 05:08:29.893032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.893009 6587 scope.go:122] "RemoveContainer" containerID="9c2072d7b1782a9bd14cd49f90cfe60b44d032e7654e22d2a265304002344def" Oct 30 05:08:29.897302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.897266 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l"] Oct 30 05:08:29.902869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.902843 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l"] Oct 30 05:08:29.946628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.946591 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq"] Oct 30 05:08:29.956246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.956211 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq"] Oct 30 05:08:29.986963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.986925 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn"] Oct 30 05:08:29.991578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:29.991553 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn"] Oct 30 05:08:30.031056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:30.031020 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk"] Oct 30 05:08:30.037851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:30.037824 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk"] Oct 30 05:08:30.072642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:30.072544 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" podStartSLOduration=2.072532328 podStartE2EDuration="2.072532328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:30.071917775 +0000 UTC m=+1434.599684562" watchObservedRunningTime="2026-10-30 05:08:30.072532328 +0000 UTC m=+1434.600299114" Oct 30 05:08:31.594215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:31.594177 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dabb599-1beb-451d-b03d-87ed3942f5ff" path="/var/lib/kubelet/pods/1dabb599-1beb-451d-b03d-87ed3942f5ff/volumes" Oct 30 05:08:31.595037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:31.595013 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363a5464-b736-4f6c-800c-38b13778716c" path="/var/lib/kubelet/pods/363a5464-b736-4f6c-800c-38b13778716c/volumes" Oct 30 05:08:31.596782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:31.595822 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407c363e-e3da-40c7-adb6-0d5e8880fd55" path="/var/lib/kubelet/pods/407c363e-e3da-40c7-adb6-0d5e8880fd55/volumes" Oct 30 05:08:31.598183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:31.597984 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896a9789-3a1d-4ec0-abfb-39131100ba0d" path="/var/lib/kubelet/pods/896a9789-3a1d-4ec0-abfb-39131100ba0d/volumes" Oct 30 05:08:32.048148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:32.048123 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x"] Oct 30 05:08:32.063165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:32.063114 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8a28de1-jtf8x"] Oct 30 05:08:32.337188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:32.337100 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng"] Oct 30 05:08:32.348554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:32.348516 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-qtzng"] Oct 30 05:08:33.584556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:33.584521 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a048a1c-93c6-4e4d-a511-6e9378eb65ab" path="/var/lib/kubelet/pods/4a048a1c-93c6-4e4d-a511-6e9378eb65ab/volumes" Oct 30 05:08:33.585118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:33.585101 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1" path="/var/lib/kubelet/pods/d9e986bb-e7e5-49c6-ad8c-d9e6fe615bf1/volumes" Oct 30 05:08:33.698910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:33.698881 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" Oct 30 05:08:36.814377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:36.810415 6587 scope.go:122] "RemoveContainer" containerID="7b286febe342bed57e749c7f1413e2e32f854ee9879c4295f835f04b727d61f7" Oct 30 05:08:36.849732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:36.849312 6587 scope.go:122] "RemoveContainer" containerID="4c434c105254c39a4e939f3b7cc21b7cefd1294efaff0f586207190c2b5e0f2b" Oct 30 05:08:36.914298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:36.914254 6587 scope.go:122] "RemoveContainer" containerID="8f7f4444fcd0bf2c9f229ac0dba83b341140c78d5b9fdd51e8149781c6bb255f" Oct 30 05:08:36.946106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:36.946076 6587 scope.go:122] "RemoveContainer" containerID="da77a721c45b74bf89b8ab9727c4395901182634128e070e3726c66f8cddca22" Oct 30 05:08:36.960962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:36.960937 6587 scope.go:122] "RemoveContainer" containerID="751d199507a4610ae6be9128bbaf3bde38d7360ad47f8239f3c713bb92238f53" Oct 30 05:08:36.990496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:36.990460 6587 scope.go:122] "RemoveContainer" containerID="be2d0590e2d30a3757f70aed20ee703b176113ccc9e27de362f1765285f3e24d" Oct 30 05:08:37.008702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:37.008680 6587 scope.go:122] "RemoveContainer" containerID="fb993ccdcdc854383879d76b8f9221ca69956a890e9709cb7bcb5489aa2bd87d" Oct 30 05:08:37.021078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:37.021054 6587 scope.go:122] "RemoveContainer" containerID="1ed6e93e922b6024a398ecf1a307fc0a55c7fdadedf545db3c54d51acd1763fa" Oct 30 05:08:40.655998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:40.655938 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:08:40.698541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:40.698489 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:08:40.896610 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:40.896567 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" Oct 30 05:08:41.248380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:41.195002 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:08:41.248380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:41.195454 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:08:41.248380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:41.220719 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 30 05:08:41.248380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:41.222317 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:08:41.658799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:41.658759 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:08:41.983434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:41.969701 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:08:42.001217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:42.001170 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:08:42.018974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:42.018944 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:08:42.130233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:42.130144 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4"] Oct 30 05:08:42.193378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:42.187771 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcc8e386-npzm4"] Oct 30 05:08:43.585907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:43.585862 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a6b587-e673-4794-8347-8477bf0024cc" path="/var/lib/kubelet/pods/f0a6b587-e673-4794-8347-8477bf0024cc/volumes" Oct 30 05:08:54.079463 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:54.079370 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c"] Oct 30 05:08:54.079892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:54.079628 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" containerID="cri-o://ff42e523667a3265eaf4d4690177625ea88e0a5a58a4f99ef10c6b385e908dcc" gracePeriod=30 Oct 30 05:08:54.093239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:54.093210 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g"] Oct 30 05:08:54.093799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:54.093544 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" podUID="6cf37513-1094-4148-9858-2e716307c4a0" containerName="istio-proxy" containerID="cri-o://02d9688b3cb6da5ae290fac045ca19b5433f5e8f99b39fd33b29373cdea73ec2" gracePeriod=30 Oct 30 05:08:54.101550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:54.101522 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb"] Oct 30 05:08:54.101737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:54.101715 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" podUID="b3dc2a23-6d11-4863-b694-646afe196859" containerName="istio-proxy" containerID="cri-o://37049dfdc0d93f19da9f0d501c692762120b9ac1f8628f66329bcd734eefd179" gracePeriod=30 Oct 30 05:08:54.109011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:54.108983 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7"] Oct 30 05:08:54.109232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:54.109208 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerName="istio-proxy" containerID="cri-o://b98580e24f3b7bd6fde951a55af33cfc592eb321c320404347304a6d6c8d3dbc" gracePeriod=30 Oct 30 05:08:55.986632 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:08:55.986597 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:08:59.455637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.455593 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:08:59.458771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.458708 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:08:59.530906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.530868 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 05:08:59.534912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.534886 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 05:08:59.549646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.549618 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3dc2a23-6d11-4863-b694-646afe196859-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dc2a23-6d11-4863-b694-646afe196859-istio-podinfo\") pod \"b3dc2a23-6d11-4863-b694-646afe196859\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " Oct 30 05:08:59.549786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.549703 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-kube-api-access-pmc72\" (UniqueName: \"kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-kube-api-access-pmc72\") pod \"b3dc2a23-6d11-4863-b694-646afe196859\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " Oct 30 05:08:59.549786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.549724 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3dc2a23-6d11-4863-b694-646afe196859-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dc2a23-6d11-4863-b694-646afe196859-istiod-ca-cert\") pod \"b3dc2a23-6d11-4863-b694-646afe196859\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " Oct 30 05:08:59.549985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.549798 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-socket\") pod \"b3dc2a23-6d11-4863-b694-646afe196859\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " Oct 30 05:08:59.549985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.549825 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-certs\") pod \"b3dc2a23-6d11-4863-b694-646afe196859\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " Oct 30 05:08:59.550106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.550067 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-data\") pod \"b3dc2a23-6d11-4863-b694-646afe196859\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " Oct 30 05:08:59.550218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.550132 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-socket\") pod \"6cf37513-1094-4148-9858-2e716307c4a0\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " Oct 30 05:08:59.550288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.550234 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-istio-token\" (UniqueName: \"kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-istio-token\") pod \"b3dc2a23-6d11-4863-b694-646afe196859\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " Oct 30 05:08:59.550396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.550309 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-envoy\") pod \"6cf37513-1094-4148-9858-2e716307c4a0\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " Oct 30 05:08:59.550396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.550340 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-credential-socket\") pod \"6cf37513-1094-4148-9858-2e716307c4a0\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " Oct 30 05:08:59.550519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.550413 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-istio-token\" (UniqueName: \"kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-istio-token\") pod \"6cf37513-1094-4148-9858-2e716307c4a0\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " Oct 30 05:08:59.550519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.550482 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-kube-api-access-qqdfh\" (UniqueName: \"kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-kube-api-access-qqdfh\") pod \"6cf37513-1094-4148-9858-2e716307c4a0\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " Oct 30 05:08:59.550629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.550550 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-data\") pod \"6cf37513-1094-4148-9858-2e716307c4a0\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " Oct 30 05:08:59.550629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.550621 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cf37513-1094-4148-9858-2e716307c4a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf37513-1094-4148-9858-2e716307c4a0-istiod-ca-cert\") pod \"6cf37513-1094-4148-9858-2e716307c4a0\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " Oct 30 05:08:59.550736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.550664 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-certs\") pod \"6cf37513-1094-4148-9858-2e716307c4a0\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " Oct 30 05:08:59.550913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.550812 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-credential-socket\") pod \"b3dc2a23-6d11-4863-b694-646afe196859\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " Oct 30 05:08:59.550913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.550868 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-envoy\") pod \"b3dc2a23-6d11-4863-b694-646afe196859\" (UID: \"b3dc2a23-6d11-4863-b694-646afe196859\") " Oct 30 05:08:59.551196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.550977 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cf37513-1094-4148-9858-2e716307c4a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf37513-1094-4148-9858-2e716307c4a0-istio-podinfo\") pod \"6cf37513-1094-4148-9858-2e716307c4a0\" (UID: \"6cf37513-1094-4148-9858-2e716307c4a0\") " Oct 30 05:08:59.552482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.552457 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-certs" pod "6cf37513-1094-4148-9858-2e716307c4a0" (UID: "6cf37513-1094-4148-9858-2e716307c4a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.552771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.552648 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-credential-socket" pod "b3dc2a23-6d11-4863-b694-646afe196859" (UID: "b3dc2a23-6d11-4863-b694-646afe196859"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.553407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.552973 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-credential-socket" pod "6cf37513-1094-4148-9858-2e716307c4a0" (UID: "6cf37513-1094-4148-9858-2e716307c4a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.553407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.552904 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf37513-1094-4148-9858-2e716307c4a0-istiod-ca-cert" pod "6cf37513-1094-4148-9858-2e716307c4a0" (UID: "6cf37513-1094-4148-9858-2e716307c4a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:59.553407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.553157 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-data" pod "b3dc2a23-6d11-4863-b694-646afe196859" (UID: "b3dc2a23-6d11-4863-b694-646afe196859"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.553407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.553159 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-socket" pod "6cf37513-1094-4148-9858-2e716307c4a0" (UID: "6cf37513-1094-4148-9858-2e716307c4a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.553658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.553492 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dc2a23-6d11-4863-b694-646afe196859-istiod-ca-cert" pod "b3dc2a23-6d11-4863-b694-646afe196859" (UID: "b3dc2a23-6d11-4863-b694-646afe196859"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:59.553724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.553697 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-certs" pod "b3dc2a23-6d11-4863-b694-646afe196859" (UID: "b3dc2a23-6d11-4863-b694-646afe196859"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.553790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.553761 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-socket" pod "b3dc2a23-6d11-4863-b694-646afe196859" (UID: "b3dc2a23-6d11-4863-b694-646afe196859"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.554202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.554181 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-data" pod "6cf37513-1094-4148-9858-2e716307c4a0" (UID: "6cf37513-1094-4148-9858-2e716307c4a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.555617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.555589 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-envoy" pod "6cf37513-1094-4148-9858-2e716307c4a0" (UID: "6cf37513-1094-4148-9858-2e716307c4a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.555803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.555779 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-kube-api-access-qqdfh" pod "6cf37513-1094-4148-9858-2e716307c4a0" (UID: "6cf37513-1094-4148-9858-2e716307c4a0"). InnerVolumeSpecName "kube-api-access-qqdfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:59.556226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.556189 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-istio-token" pod "6cf37513-1094-4148-9858-2e716307c4a0" (UID: "6cf37513-1094-4148-9858-2e716307c4a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:59.556728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.556704 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-istio-token" pod "b3dc2a23-6d11-4863-b694-646afe196859" (UID: "b3dc2a23-6d11-4863-b694-646afe196859"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:59.557006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.556988 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-envoy" pod "b3dc2a23-6d11-4863-b694-646afe196859" (UID: "b3dc2a23-6d11-4863-b694-646afe196859"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.558153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.558132 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cf37513-1094-4148-9858-2e716307c4a0-istio-podinfo" pod "6cf37513-1094-4148-9858-2e716307c4a0" (UID: "6cf37513-1094-4148-9858-2e716307c4a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:08:59.560746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.560718 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-kube-api-access-pmc72" pod "b3dc2a23-6d11-4863-b694-646afe196859" (UID: "b3dc2a23-6d11-4863-b694-646afe196859"). InnerVolumeSpecName "kube-api-access-pmc72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:59.562089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.562062 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3dc2a23-6d11-4863-b694-646afe196859-istio-podinfo" pod "b3dc2a23-6d11-4863-b694-646afe196859" (UID: "b3dc2a23-6d11-4863-b694-646afe196859"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:08:59.601713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.601235 6587 generic.go:356] "Generic (PLEG): container finished" podID="b3dc2a23-6d11-4863-b694-646afe196859" containerID="37049dfdc0d93f19da9f0d501c692762120b9ac1f8628f66329bcd734eefd179" exitCode=0 Oct 30 05:08:59.601713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.601458 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" event={"ID":"b3dc2a23-6d11-4863-b694-646afe196859","Type":"ContainerDied","Data":"37049dfdc0d93f19da9f0d501c692762120b9ac1f8628f66329bcd734eefd179"} Oct 30 05:08:59.601713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.601508 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" event={"ID":"b3dc2a23-6d11-4863-b694-646afe196859","Type":"ContainerDied","Data":"70347407bdc0420f97d7a5fba4f7be8fe0ae5356c81d945d4294cedcf3496d2b"} Oct 30 05:08:59.601713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.601539 6587 scope.go:122] "RemoveContainer" containerID="37049dfdc0d93f19da9f0d501c692762120b9ac1f8628f66329bcd734eefd179" Oct 30 05:08:59.601713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.601518 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb" Oct 30 05:08:59.605537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.605508 6587 generic.go:356] "Generic (PLEG): container finished" podID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerID="b98580e24f3b7bd6fde951a55af33cfc592eb321c320404347304a6d6c8d3dbc" exitCode=0 Oct 30 05:08:59.607838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.607578 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" Oct 30 05:08:59.607838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.607752 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" event={"ID":"7600beab-bc24-4634-b008-6aebfb2c71c9","Type":"ContainerDied","Data":"b98580e24f3b7bd6fde951a55af33cfc592eb321c320404347304a6d6c8d3dbc"} Oct 30 05:08:59.607838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.607797 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7" event={"ID":"7600beab-bc24-4634-b008-6aebfb2c71c9","Type":"ContainerDied","Data":"53561e28b961fd5a7a9655e6e9f3a20127e97038058e3a28ff15eb5c40f9bd86"} Oct 30 05:08:59.613560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.609779 6587 generic.go:356] "Generic (PLEG): container finished" podID="6cf37513-1094-4148-9858-2e716307c4a0" containerID="02d9688b3cb6da5ae290fac045ca19b5433f5e8f99b39fd33b29373cdea73ec2" exitCode=0 Oct 30 05:08:59.613560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.609885 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" event={"ID":"6cf37513-1094-4148-9858-2e716307c4a0","Type":"ContainerDied","Data":"02d9688b3cb6da5ae290fac045ca19b5433f5e8f99b39fd33b29373cdea73ec2"} Oct 30 05:08:59.613560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.609918 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" event={"ID":"6cf37513-1094-4148-9858-2e716307c4a0","Type":"ContainerDied","Data":"90f093f59d32de154c17bc7147023abd68e7570d1b6032ef1cf8e87a108cb705"} Oct 30 05:08:59.613560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.609974 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g" Oct 30 05:08:59.618377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.615991 6587 generic.go:356] "Generic (PLEG): container finished" podID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerID="ff42e523667a3265eaf4d4690177625ea88e0a5a58a4f99ef10c6b385e908dcc" exitCode=0 Oct 30 05:08:59.618377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.616049 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" event={"ID":"a3a110c9-725e-4466-b182-1f608f0d0ba0","Type":"ContainerDied","Data":"ff42e523667a3265eaf4d4690177625ea88e0a5a58a4f99ef10c6b385e908dcc"} Oct 30 05:08:59.618377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.616101 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" event={"ID":"a3a110c9-725e-4466-b182-1f608f0d0ba0","Type":"ContainerDied","Data":"3aacee0c1a3108f94b8b0d9df9d0139718de73f548eb2fb94e1642c0680d6315"} Oct 30 05:08:59.618377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.616146 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c" Oct 30 05:08:59.624829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.624806 6587 scope.go:122] "RemoveContainer" containerID="37049dfdc0d93f19da9f0d501c692762120b9ac1f8628f66329bcd734eefd179" Oct 30 05:08:59.625084 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:08:59.625053 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37049dfdc0d93f19da9f0d501c692762120b9ac1f8628f66329bcd734eefd179\": container with ID starting with 37049dfdc0d93f19da9f0d501c692762120b9ac1f8628f66329bcd734eefd179 not found: ID does not exist" containerID="37049dfdc0d93f19da9f0d501c692762120b9ac1f8628f66329bcd734eefd179" Oct 30 05:08:59.625173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.625086 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37049dfdc0d93f19da9f0d501c692762120b9ac1f8628f66329bcd734eefd179"} err="failed to get container status \"37049dfdc0d93f19da9f0d501c692762120b9ac1f8628f66329bcd734eefd179\": rpc error: code = NotFound desc = could not find container \"37049dfdc0d93f19da9f0d501c692762120b9ac1f8628f66329bcd734eefd179\": container with ID starting with 37049dfdc0d93f19da9f0d501c692762120b9ac1f8628f66329bcd734eefd179 not found: ID does not exist" Oct 30 05:08:59.625173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.625111 6587 scope.go:122] "RemoveContainer" containerID="b98580e24f3b7bd6fde951a55af33cfc592eb321c320404347304a6d6c8d3dbc" Oct 30 05:08:59.648582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.648555 6587 scope.go:122] "RemoveContainer" containerID="b98580e24f3b7bd6fde951a55af33cfc592eb321c320404347304a6d6c8d3dbc" Oct 30 05:08:59.649781 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:08:59.649753 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98580e24f3b7bd6fde951a55af33cfc592eb321c320404347304a6d6c8d3dbc\": container with ID starting with b98580e24f3b7bd6fde951a55af33cfc592eb321c320404347304a6d6c8d3dbc not found: ID does not exist" containerID="b98580e24f3b7bd6fde951a55af33cfc592eb321c320404347304a6d6c8d3dbc" Oct 30 05:08:59.649941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.649914 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98580e24f3b7bd6fde951a55af33cfc592eb321c320404347304a6d6c8d3dbc"} err="failed to get container status \"b98580e24f3b7bd6fde951a55af33cfc592eb321c320404347304a6d6c8d3dbc\": rpc error: code = NotFound desc = could not find container \"b98580e24f3b7bd6fde951a55af33cfc592eb321c320404347304a6d6c8d3dbc\": container with ID starting with b98580e24f3b7bd6fde951a55af33cfc592eb321c320404347304a6d6c8d3dbc not found: ID does not exist" Oct 30 05:08:59.650049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.650036 6587 scope.go:122] "RemoveContainer" containerID="02d9688b3cb6da5ae290fac045ca19b5433f5e8f99b39fd33b29373cdea73ec2" Oct 30 05:08:59.653268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.653243 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-podinfo\") pod \"a3a110c9-725e-4466-b182-1f608f0d0ba0\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " Oct 30 05:08:59.653543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.653522 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-envoy\") pod \"7600beab-bc24-4634-b008-6aebfb2c71c9\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " Oct 30 05:08:59.653790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.653770 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-kube-api-access-wkmsk\" (UniqueName: \"kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-kube-api-access-wkmsk\") pod \"a3a110c9-725e-4466-b182-1f608f0d0ba0\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " Oct 30 05:08:59.654032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.654012 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-token\" (UniqueName: \"kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-token\") pod \"7600beab-bc24-4634-b008-6aebfb2c71c9\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " Oct 30 05:08:59.654163 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.654148 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-data\") pod \"7600beab-bc24-4634-b008-6aebfb2c71c9\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " Oct 30 05:08:59.654297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.654282 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7600beab-bc24-4634-b008-6aebfb2c71c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7600beab-bc24-4634-b008-6aebfb2c71c9-istiod-ca-cert\") pod \"7600beab-bc24-4634-b008-6aebfb2c71c9\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " Oct 30 05:08:59.654440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.654423 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-credential-socket\") pod \"7600beab-bc24-4634-b008-6aebfb2c71c9\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " Oct 30 05:08:59.654555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.654540 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-data\") pod \"a3a110c9-725e-4466-b182-1f608f0d0ba0\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " Oct 30 05:08:59.654797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.654779 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-socket\") pod \"a3a110c9-725e-4466-b182-1f608f0d0ba0\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " Oct 30 05:08:59.654925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.654910 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a110c9-725e-4466-b182-1f608f0d0ba0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a110c9-725e-4466-b182-1f608f0d0ba0-istiod-ca-cert\") pod \"a3a110c9-725e-4466-b182-1f608f0d0ba0\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " Oct 30 05:08:59.655048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.655034 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-socket\") pod \"7600beab-bc24-4634-b008-6aebfb2c71c9\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " Oct 30 05:08:59.655186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.655172 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-certs\") pod \"a3a110c9-725e-4466-b182-1f608f0d0ba0\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " Oct 30 05:08:59.655300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.655284 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-podinfo\") pod \"7600beab-bc24-4634-b008-6aebfb2c71c9\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " Oct 30 05:08:59.655444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.655428 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-token\" (UniqueName: \"kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-token\") pod \"a3a110c9-725e-4466-b182-1f608f0d0ba0\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " Oct 30 05:08:59.655610 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.655595 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-kube-api-access-c2vz2\" (UniqueName: \"kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-kube-api-access-c2vz2\") pod \"7600beab-bc24-4634-b008-6aebfb2c71c9\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " Oct 30 05:08:59.655846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.655829 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-envoy\") pod \"a3a110c9-725e-4466-b182-1f608f0d0ba0\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " Oct 30 05:08:59.655976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.655961 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-credential-socket\") pod \"a3a110c9-725e-4466-b182-1f608f0d0ba0\" (UID: \"a3a110c9-725e-4466-b182-1f608f0d0ba0\") " Oct 30 05:08:59.656082 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.656068 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-certs\") pod \"7600beab-bc24-4634-b008-6aebfb2c71c9\" (UID: \"7600beab-bc24-4634-b008-6aebfb2c71c9\") " Oct 30 05:08:59.657188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.657165 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.662543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.662525 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.662709 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.662694 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.662825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.662811 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.662935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.662922 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.663043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.663028 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.663156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.663142 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqdfh\" (UniqueName: \"kubernetes.io/projected/6cf37513-1094-4148-9858-2e716307c4a0-kube-api-access-qqdfh\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.663258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.663245 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.663386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.663350 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf37513-1094-4148-9858-2e716307c4a0-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.663493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.663481 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf37513-1094-4148-9858-2e716307c4a0-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.663594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.663581 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.663686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.663675 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.663783 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.663773 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf37513-1094-4148-9858-2e716307c4a0-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.663876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.663862 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dc2a23-6d11-4863-b694-646afe196859-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.663974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.663961 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmc72\" (UniqueName: \"kubernetes.io/projected/b3dc2a23-6d11-4863-b694-646afe196859-kube-api-access-pmc72\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.664104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.664090 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dc2a23-6d11-4863-b694-646afe196859-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.664250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.664238 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.664390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.664339 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dc2a23-6d11-4863-b694-646afe196859-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.664481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.657581 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-data" pod "7600beab-bc24-4634-b008-6aebfb2c71c9" (UID: "7600beab-bc24-4634-b008-6aebfb2c71c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.664576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.657679 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-socket" pod "a3a110c9-725e-4466-b182-1f608f0d0ba0" (UID: "a3a110c9-725e-4466-b182-1f608f0d0ba0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.664658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.657794 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-data" pod "a3a110c9-725e-4466-b182-1f608f0d0ba0" (UID: "a3a110c9-725e-4466-b182-1f608f0d0ba0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.664750 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.658103 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-certs" pod "a3a110c9-725e-4466-b182-1f608f0d0ba0" (UID: "a3a110c9-725e-4466-b182-1f608f0d0ba0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.664839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.658120 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7600beab-bc24-4634-b008-6aebfb2c71c9-istiod-ca-cert" pod "7600beab-bc24-4634-b008-6aebfb2c71c9" (UID: "7600beab-bc24-4634-b008-6aebfb2c71c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:59.664932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.658424 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-socket" pod "7600beab-bc24-4634-b008-6aebfb2c71c9" (UID: "7600beab-bc24-4634-b008-6aebfb2c71c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.665031 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.658971 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-credential-socket" pod "7600beab-bc24-4634-b008-6aebfb2c71c9" (UID: "7600beab-bc24-4634-b008-6aebfb2c71c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.665200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.660160 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-podinfo" pod "a3a110c9-725e-4466-b182-1f608f0d0ba0" (UID: "a3a110c9-725e-4466-b182-1f608f0d0ba0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:08:59.665282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.660837 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-certs" pod "7600beab-bc24-4634-b008-6aebfb2c71c9" (UID: "7600beab-bc24-4634-b008-6aebfb2c71c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.665282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.661418 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a110c9-725e-4466-b182-1f608f0d0ba0-istiod-ca-cert" pod "a3a110c9-725e-4466-b182-1f608f0d0ba0" (UID: "a3a110c9-725e-4466-b182-1f608f0d0ba0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:08:59.665282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.662011 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-credential-socket" pod "a3a110c9-725e-4466-b182-1f608f0d0ba0" (UID: "a3a110c9-725e-4466-b182-1f608f0d0ba0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.665282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.663253 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-envoy" pod "a3a110c9-725e-4466-b182-1f608f0d0ba0" (UID: "a3a110c9-725e-4466-b182-1f608f0d0ba0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.665282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.663904 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-envoy" pod "7600beab-bc24-4634-b008-6aebfb2c71c9" (UID: "7600beab-bc24-4634-b008-6aebfb2c71c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:08:59.665282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.664102 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-kube-api-access-wkmsk" pod "a3a110c9-725e-4466-b182-1f608f0d0ba0" (UID: "a3a110c9-725e-4466-b182-1f608f0d0ba0"). InnerVolumeSpecName "kube-api-access-wkmsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:59.665282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.664244 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-kube-api-access-c2vz2" pod "7600beab-bc24-4634-b008-6aebfb2c71c9" (UID: "7600beab-bc24-4634-b008-6aebfb2c71c9"). InnerVolumeSpecName "kube-api-access-c2vz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:59.665282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.664320 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-podinfo" pod "7600beab-bc24-4634-b008-6aebfb2c71c9" (UID: "7600beab-bc24-4634-b008-6aebfb2c71c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:08:59.665282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.664457 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-token" pod "a3a110c9-725e-4466-b182-1f608f0d0ba0" (UID: "a3a110c9-725e-4466-b182-1f608f0d0ba0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:59.666300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.666279 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-token" pod "7600beab-bc24-4634-b008-6aebfb2c71c9" (UID: "7600beab-bc24-4634-b008-6aebfb2c71c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:08:59.678214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.676526 6587 scope.go:122] "RemoveContainer" containerID="02d9688b3cb6da5ae290fac045ca19b5433f5e8f99b39fd33b29373cdea73ec2" Oct 30 05:08:59.678422 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:08:59.678398 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d9688b3cb6da5ae290fac045ca19b5433f5e8f99b39fd33b29373cdea73ec2\": container with ID starting with 02d9688b3cb6da5ae290fac045ca19b5433f5e8f99b39fd33b29373cdea73ec2 not found: ID does not exist" containerID="02d9688b3cb6da5ae290fac045ca19b5433f5e8f99b39fd33b29373cdea73ec2" Oct 30 05:08:59.678496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.678430 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d9688b3cb6da5ae290fac045ca19b5433f5e8f99b39fd33b29373cdea73ec2"} err="failed to get container status \"02d9688b3cb6da5ae290fac045ca19b5433f5e8f99b39fd33b29373cdea73ec2\": rpc error: code = NotFound desc = could not find container \"02d9688b3cb6da5ae290fac045ca19b5433f5e8f99b39fd33b29373cdea73ec2\": container with ID starting with 02d9688b3cb6da5ae290fac045ca19b5433f5e8f99b39fd33b29373cdea73ec2 not found: ID does not exist" Oct 30 05:08:59.678496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.678448 6587 scope.go:122] "RemoveContainer" containerID="ff42e523667a3265eaf4d4690177625ea88e0a5a58a4f99ef10c6b385e908dcc" Oct 30 05:08:59.700227 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.700204 6587 scope.go:122] "RemoveContainer" containerID="3d037236381e06aefd12320ec64df2c9a5de69c2703e82cf9f9d1de25efc776f" Oct 30 05:08:59.712550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.712529 6587 scope.go:122] "RemoveContainer" containerID="ff42e523667a3265eaf4d4690177625ea88e0a5a58a4f99ef10c6b385e908dcc" Oct 30 05:08:59.712886 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:08:59.712864 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff42e523667a3265eaf4d4690177625ea88e0a5a58a4f99ef10c6b385e908dcc\": container with ID starting with ff42e523667a3265eaf4d4690177625ea88e0a5a58a4f99ef10c6b385e908dcc not found: ID does not exist" containerID="ff42e523667a3265eaf4d4690177625ea88e0a5a58a4f99ef10c6b385e908dcc" Oct 30 05:08:59.712985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.712898 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff42e523667a3265eaf4d4690177625ea88e0a5a58a4f99ef10c6b385e908dcc"} err="failed to get container status \"ff42e523667a3265eaf4d4690177625ea88e0a5a58a4f99ef10c6b385e908dcc\": rpc error: code = NotFound desc = could not find container \"ff42e523667a3265eaf4d4690177625ea88e0a5a58a4f99ef10c6b385e908dcc\": container with ID starting with ff42e523667a3265eaf4d4690177625ea88e0a5a58a4f99ef10c6b385e908dcc not found: ID does not exist" Oct 30 05:08:59.712985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.712922 6587 scope.go:122] "RemoveContainer" containerID="3d037236381e06aefd12320ec64df2c9a5de69c2703e82cf9f9d1de25efc776f" Oct 30 05:08:59.713241 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:08:59.713222 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d037236381e06aefd12320ec64df2c9a5de69c2703e82cf9f9d1de25efc776f\": container with ID starting with 3d037236381e06aefd12320ec64df2c9a5de69c2703e82cf9f9d1de25efc776f not found: ID does not exist" containerID="3d037236381e06aefd12320ec64df2c9a5de69c2703e82cf9f9d1de25efc776f" Oct 30 05:08:59.713331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.713250 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d037236381e06aefd12320ec64df2c9a5de69c2703e82cf9f9d1de25efc776f"} err="failed to get container status \"3d037236381e06aefd12320ec64df2c9a5de69c2703e82cf9f9d1de25efc776f\": rpc error: code = NotFound desc = could not find container \"3d037236381e06aefd12320ec64df2c9a5de69c2703e82cf9f9d1de25efc776f\": container with ID starting with 3d037236381e06aefd12320ec64df2c9a5de69c2703e82cf9f9d1de25efc776f not found: ID does not exist" Oct 30 05:08:59.742343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.742295 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g"] Oct 30 05:08:59.749653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.749626 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g"] Oct 30 05:08:59.766090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766064 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.766090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766093 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766108 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766123 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2vz2\" (UniqueName: \"kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-kube-api-access-c2vz2\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766134 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766146 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766158 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766171 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766182 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766196 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkmsk\" (UniqueName: \"kubernetes.io/projected/a3a110c9-725e-4466-b182-1f608f0d0ba0-kube-api-access-wkmsk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766208 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766220 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766230 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7600beab-bc24-4634-b008-6aebfb2c71c9-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766241 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766251 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766264 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a110c9-725e-4466-b182-1f608f0d0ba0-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766277 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a110c9-725e-4466-b182-1f608f0d0ba0-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.767011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.766290 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7600beab-bc24-4634-b008-6aebfb2c71c9-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:08:59.785806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.785776 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb"] Oct 30 05:08:59.798434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.797072 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb"] Oct 30 05:08:59.960557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.960522 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7"] Oct 30 05:08:59.964398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:08:59.964370 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7"] Oct 30 05:09:00.002549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:00.002513 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c"] Oct 30 05:09:00.006447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:00.006425 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c"] Oct 30 05:09:01.585518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:01.585478 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf37513-1094-4148-9858-2e716307c4a0" path="/var/lib/kubelet/pods/6cf37513-1094-4148-9858-2e716307c4a0/volumes" Oct 30 05:09:01.586132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:01.586113 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" path="/var/lib/kubelet/pods/7600beab-bc24-4634-b008-6aebfb2c71c9/volumes" Oct 30 05:09:01.586627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:01.586608 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" path="/var/lib/kubelet/pods/a3a110c9-725e-4466-b182-1f608f0d0ba0/volumes" Oct 30 05:09:01.587416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:01.587396 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3dc2a23-6d11-4863-b694-646afe196859" path="/var/lib/kubelet/pods/b3dc2a23-6d11-4863-b694-646afe196859/volumes" Oct 30 05:09:18.649517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:18.649466 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:18.679851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:18.679804 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:18.698598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:18.698549 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:09:18.902572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:18.902455 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:19.092149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.090380 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:19.092149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.090781 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:09:19.092585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.092526 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:19.120383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.119445 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:09:19.255498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.253303 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:09:19.538435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.537743 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 05:09:19.667384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.666150 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:09:19.685395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.681451 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:19.685395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.681713 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:09:19.714528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.714473 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:19.785383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.749183 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:09:19.785727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.785697 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:09:19.785908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.785893 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:09:19.922385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.904467 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:09:19.922385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:19.905055 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:09:20.114383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.105154 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:20.114383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.106086 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:09:20.138568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.138532 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:09:20.138957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.138934 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:09:20.160424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.139492 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:20.160424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.139812 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:09:20.160424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.139940 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:20.160424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.140331 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:09:20.283777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.283660 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:09:20.342179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.286122 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:20.342780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.342750 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:09:20.730511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.730427 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:20.750610 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.740498 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:20.770518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.741300 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:09:20.786970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.785854 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:09:20.941852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.941804 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:09:20.942806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.942757 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" podUID="ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:20.983470 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:20.943386 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:21.152718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:21.152666 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:09:21.176391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:21.153842 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:09:21.471813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:21.471756 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:09:21.509439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:21.432004 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:09:21.711466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:21.702535 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:21.797615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:21.797500 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:21.823384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:21.807973 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:21.863661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:21.810540 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Oct 30 05:09:21.863907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:21.810604 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:09:21.985939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:21.985882 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:22.155397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:22.155207 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:22.438327 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:22.438223 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:22.478383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:22.477166 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:22.528506 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:22.528435 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:22.845382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:22.828592 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:09:23.001431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:22.955656 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:23.001431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:22.956013 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:09:23.001431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:22.956273 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:23.001431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:22.956902 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:09:23.008729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:23.005811 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:23.008729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:23.008518 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:23.235440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:23.206956 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:09:23.450017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:23.449974 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:23.479519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:23.453776 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:09:23.516044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:23.484349 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:23.538388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:23.531247 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:23.847382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:23.846943 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:23.963507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:23.963451 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:23.972382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:23.964046 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:09:23.992197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:23.988047 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:09:24.215171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:24.215063 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg"] Oct 30 05:09:24.215671 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:24.215635 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" podUID="df01352d-3e53-4bcc-a30a-a2b291f95879" containerName="istio-proxy" containerID="cri-o://4871bec13135c78c32505a0154a573254bf2af3dd8aa91ebcb1442fb03ad403c" gracePeriod=30 Oct 30 05:09:24.477440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:24.476699 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:09:24.496508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:24.489192 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:09:24.565425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:24.556485 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:24.587382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:24.577945 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:09:24.759751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:24.759717 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:09:24.868497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:24.861439 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:09:24.908144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:24.908115 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:09:24.970730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:24.970679 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:25.009637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:25.009534 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:25.096414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:25.094748 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:09:25.096414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:25.096023 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:25.122384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:25.110519 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:09:25.142385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:25.140845 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:09:25.296437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:25.295289 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:09:25.312508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:25.309893 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:09:25.312683 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:25.312623 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:09:26.414059 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:09:26.414016 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.783212 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7"] Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784480 6587 cpu_manager.go:414] "containerMap: removing container" podUID="896a9789-3a1d-4ec0-abfb-39131100ba0d" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784502 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896a9789-3a1d-4ec0-abfb-39131100ba0d" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784522 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784530 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784552 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1dabb599-1beb-451d-b03d-87ed3942f5ff" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784561 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dabb599-1beb-451d-b03d-87ed3942f5ff" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784581 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b3dc2a23-6d11-4863-b694-646afe196859" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784588 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3dc2a23-6d11-4863-b694-646afe196859" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784616 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784624 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784634 6587 cpu_manager.go:414] "containerMap: removing container" podUID="407c363e-e3da-40c7-adb6-0d5e8880fd55" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784640 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407c363e-e3da-40c7-adb6-0d5e8880fd55" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784670 6587 cpu_manager.go:414] "containerMap: removing container" podUID="363a5464-b736-4f6c-800c-38b13778716c" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784677 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363a5464-b736-4f6c-800c-38b13778716c" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784729 6587 cpu_manager.go:414] "containerMap: removing container" podUID="6cf37513-1094-4148-9858-2e716307c4a0" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784737 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf37513-1094-4148-9858-2e716307c4a0" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784912 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf37513-1094-4148-9858-2e716307c4a0" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784935 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="896a9789-3a1d-4ec0-abfb-39131100ba0d" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784947 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784957 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dabb599-1beb-451d-b03d-87ed3942f5ff" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784971 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3dc2a23-6d11-4863-b694-646afe196859" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784987 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7600beab-bc24-4634-b008-6aebfb2c71c9" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.784996 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="407c363e-e3da-40c7-adb6-0d5e8880fd55" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.785012 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="363a5464-b736-4f6c-800c-38b13778716c" containerName="istio-proxy" Oct 30 05:09:28.789378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.785022 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" Oct 30 05:09:28.831084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.815543 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" Oct 30 05:09:28.837792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.837745 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:09:28.851774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.851643 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:09:28.851774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.851646 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:09:28.851774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.851735 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:09:28.874657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.871902 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7"] Oct 30 05:09:28.885462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.884988 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-528d67-frfz7\" (UID: \"dbf76d60-8cf0-4608-bd40-7139520656c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" Oct 30 05:09:28.885462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.885045 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wvm\" (UniqueName: \"kubernetes.io/projected/dbf76d60-8cf0-4608-bd40-7139520656c9-kube-api-access-67wvm\") pod \"maas-api-revoke-keys-e2e-api-a-528d67-frfz7\" (UID: \"dbf76d60-8cf0-4608-bd40-7139520656c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" Oct 30 05:09:28.885462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.885202 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-528d67-frfz7\" (UID: \"dbf76d60-8cf0-4608-bd40-7139520656c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" Oct 30 05:09:28.986796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.986753 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-528d67-frfz7\" (UID: \"dbf76d60-8cf0-4608-bd40-7139520656c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" Oct 30 05:09:28.987105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.987079 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-528d67-frfz7\" (UID: \"dbf76d60-8cf0-4608-bd40-7139520656c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" Oct 30 05:09:28.987166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.987130 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wvm\" (UniqueName: \"kubernetes.io/projected/dbf76d60-8cf0-4608-bd40-7139520656c9-kube-api-access-67wvm\") pod \"maas-api-revoke-keys-e2e-api-a-528d67-frfz7\" (UID: \"dbf76d60-8cf0-4608-bd40-7139520656c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" Oct 30 05:09:28.988155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.988065 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-528d67-frfz7\" (UID: \"dbf76d60-8cf0-4608-bd40-7139520656c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" Oct 30 05:09:28.990697 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:28.990671 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-528d67-frfz7\" (UID: \"dbf76d60-8cf0-4608-bd40-7139520656c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" Oct 30 05:09:29.009040 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.009016 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wvm\" (UniqueName: \"kubernetes.io/projected/dbf76d60-8cf0-4608-bd40-7139520656c9-kube-api-access-67wvm\") pod \"maas-api-revoke-keys-e2e-api-a-528d67-frfz7\" (UID: \"dbf76d60-8cf0-4608-bd40-7139520656c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" Oct 30 05:09:29.194609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.194556 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" Oct 30 05:09:29.780331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.780292 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:09:29.805689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.805525 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-socket\") pod \"df01352d-3e53-4bcc-a30a-a2b291f95879\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " Oct 30 05:09:29.805689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.805627 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-data\") pod \"df01352d-3e53-4bcc-a30a-a2b291f95879\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " Oct 30 05:09:29.805689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.805667 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-kube-api-access-hrbws\" (UniqueName: \"kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-kube-api-access-hrbws\") pod \"df01352d-3e53-4bcc-a30a-a2b291f95879\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " Oct 30 05:09:29.809570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.805729 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-credential-socket\") pod \"df01352d-3e53-4bcc-a30a-a2b291f95879\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " Oct 30 05:09:29.809570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.805764 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-token\" (UniqueName: \"kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-token\") pod \"df01352d-3e53-4bcc-a30a-a2b291f95879\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " Oct 30 05:09:29.809570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.805853 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-envoy\") pod \"df01352d-3e53-4bcc-a30a-a2b291f95879\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " Oct 30 05:09:29.809570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.805907 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-certs\") pod \"df01352d-3e53-4bcc-a30a-a2b291f95879\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " Oct 30 05:09:29.809570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.805968 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df01352d-3e53-4bcc-a30a-a2b291f95879-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df01352d-3e53-4bcc-a30a-a2b291f95879-istiod-ca-cert\") pod \"df01352d-3e53-4bcc-a30a-a2b291f95879\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " Oct 30 05:09:29.809570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.806001 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-podinfo\") pod \"df01352d-3e53-4bcc-a30a-a2b291f95879\" (UID: \"df01352d-3e53-4bcc-a30a-a2b291f95879\") " Oct 30 05:09:29.809570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.807220 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-credential-socket" pod "df01352d-3e53-4bcc-a30a-a2b291f95879" (UID: "df01352d-3e53-4bcc-a30a-a2b291f95879"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:29.809570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.807433 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-socket" pod "df01352d-3e53-4bcc-a30a-a2b291f95879" (UID: "df01352d-3e53-4bcc-a30a-a2b291f95879"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:29.809570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.807614 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-data" pod "df01352d-3e53-4bcc-a30a-a2b291f95879" (UID: "df01352d-3e53-4bcc-a30a-a2b291f95879"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:29.809570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.808616 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-certs" pod "df01352d-3e53-4bcc-a30a-a2b291f95879" (UID: "df01352d-3e53-4bcc-a30a-a2b291f95879"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:29.809570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.808773 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df01352d-3e53-4bcc-a30a-a2b291f95879-istiod-ca-cert" pod "df01352d-3e53-4bcc-a30a-a2b291f95879" (UID: "df01352d-3e53-4bcc-a30a-a2b291f95879"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:29.813023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.812867 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-token" pod "df01352d-3e53-4bcc-a30a-a2b291f95879" (UID: "df01352d-3e53-4bcc-a30a-a2b291f95879"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:29.814703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.814439 6587 patch_prober.go:29] interesting pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:09:29.814703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.814529 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-envoy" pod "df01352d-3e53-4bcc-a30a-a2b291f95879" (UID: "df01352d-3e53-4bcc-a30a-a2b291f95879"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:29.814703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.814519 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:29.817789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.817759 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-podinfo" pod "df01352d-3e53-4bcc-a30a-a2b291f95879" (UID: "df01352d-3e53-4bcc-a30a-a2b291f95879"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:09:29.833656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.833615 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-kube-api-access-hrbws" pod "df01352d-3e53-4bcc-a30a-a2b291f95879" (UID: "df01352d-3e53-4bcc-a30a-a2b291f95879"). InnerVolumeSpecName "kube-api-access-hrbws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:29.893526 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:09:29.893479 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf76d60_8cf0_4608_bd40_7139520656c9.slice/crio-2b8641185f41f04046f090d3696cee8321722b7ec53dde133a13436ab2af7ca5 WatchSource:0}: Error finding container 2b8641185f41f04046f090d3696cee8321722b7ec53dde133a13436ab2af7ca5: Status 404 returned error can't find the container with id 2b8641185f41f04046f090d3696cee8321722b7ec53dde133a13436ab2af7ca5 Oct 30 05:09:29.909306 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.909261 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7"] Oct 30 05:09:29.911154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.911125 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:29.911261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.911169 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:29.911261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.911186 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df01352d-3e53-4bcc-a30a-a2b291f95879-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:29.911261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.911201 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:29.911261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.911215 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:29.911261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.911240 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:29.911261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.911255 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrbws\" (UniqueName: \"kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-kube-api-access-hrbws\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:29.911594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.911270 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df01352d-3e53-4bcc-a30a-a2b291f95879-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:29.911594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:29.911285 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df01352d-3e53-4bcc-a30a-a2b291f95879-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:30.094510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:30.094471 6587 generic.go:356] "Generic (PLEG): container finished" podID="df01352d-3e53-4bcc-a30a-a2b291f95879" containerID="4871bec13135c78c32505a0154a573254bf2af3dd8aa91ebcb1442fb03ad403c" exitCode=0 Oct 30 05:09:30.094701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:30.094550 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" Oct 30 05:09:30.094701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:30.094563 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" event={"ID":"df01352d-3e53-4bcc-a30a-a2b291f95879","Type":"ContainerDied","Data":"4871bec13135c78c32505a0154a573254bf2af3dd8aa91ebcb1442fb03ad403c"} Oct 30 05:09:30.094701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:30.094639 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg" event={"ID":"df01352d-3e53-4bcc-a30a-a2b291f95879","Type":"ContainerDied","Data":"cbad4387470471468f9c14f8e38df78dc881b201f5fb0a4ade3da00512e6b5cf"} Oct 30 05:09:30.094701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:30.094673 6587 scope.go:122] "RemoveContainer" containerID="4871bec13135c78c32505a0154a573254bf2af3dd8aa91ebcb1442fb03ad403c" Oct 30 05:09:30.098121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:30.098089 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" event={"ID":"dbf76d60-8cf0-4608-bd40-7139520656c9","Type":"ContainerStarted","Data":"2b8641185f41f04046f090d3696cee8321722b7ec53dde133a13436ab2af7ca5"} Oct 30 05:09:30.111565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:30.111540 6587 scope.go:122] "RemoveContainer" containerID="4871bec13135c78c32505a0154a573254bf2af3dd8aa91ebcb1442fb03ad403c" Oct 30 05:09:30.111886 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:09:30.111858 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4871bec13135c78c32505a0154a573254bf2af3dd8aa91ebcb1442fb03ad403c\": container with ID starting with 4871bec13135c78c32505a0154a573254bf2af3dd8aa91ebcb1442fb03ad403c not found: ID does not exist" containerID="4871bec13135c78c32505a0154a573254bf2af3dd8aa91ebcb1442fb03ad403c" Oct 30 05:09:30.111995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:30.111899 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4871bec13135c78c32505a0154a573254bf2af3dd8aa91ebcb1442fb03ad403c"} err="failed to get container status \"4871bec13135c78c32505a0154a573254bf2af3dd8aa91ebcb1442fb03ad403c\": rpc error: code = NotFound desc = could not find container \"4871bec13135c78c32505a0154a573254bf2af3dd8aa91ebcb1442fb03ad403c\": container with ID starting with 4871bec13135c78c32505a0154a573254bf2af3dd8aa91ebcb1442fb03ad403c not found: ID does not exist" Oct 30 05:09:30.158477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:30.157611 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg"] Oct 30 05:09:30.161889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:30.161501 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg"] Oct 30 05:09:31.110540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:31.110502 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" event={"ID":"dbf76d60-8cf0-4608-bd40-7139520656c9","Type":"ContainerStarted","Data":"057bc15e73b4656b1a8b000dcac298a9d17a1a83cef10f51335e43bab96ea56b"} Oct 30 05:09:31.160937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:31.160878 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" podStartSLOduration=3.160864167 podStartE2EDuration="3.160864167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:09:31.160302819 +0000 UTC m=+1495.688069606" watchObservedRunningTime="2026-10-30 05:09:31.160864167 +0000 UTC m=+1495.688630952" Oct 30 05:09:31.587859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:31.587823 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df01352d-3e53-4bcc-a30a-a2b291f95879" path="/var/lib/kubelet/pods/df01352d-3e53-4bcc-a30a-a2b291f95879/volumes" Oct 30 05:09:32.132653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.124413 6587 generic.go:356] "Generic (PLEG): container finished" podID="dbf76d60-8cf0-4608-bd40-7139520656c9" containerID="057bc15e73b4656b1a8b000dcac298a9d17a1a83cef10f51335e43bab96ea56b" exitCode=0 Oct 30 05:09:32.132653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.124752 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" event={"ID":"dbf76d60-8cf0-4608-bd40-7139520656c9","Type":"ContainerDied","Data":"057bc15e73b4656b1a8b000dcac298a9d17a1a83cef10f51335e43bab96ea56b"} Oct 30 05:09:32.332581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.332529 6587 patch_prober.go:29] interesting pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:09:32.332783 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.332605 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:32.761341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.761299 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8"] Oct 30 05:09:32.762672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.762645 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" Oct 30 05:09:32.762672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.762672 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a110c9-725e-4466-b182-1f608f0d0ba0" containerName="istio-proxy" Oct 30 05:09:32.762880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.762719 6587 cpu_manager.go:414] "containerMap: removing container" podUID="df01352d-3e53-4bcc-a30a-a2b291f95879" containerName="istio-proxy" Oct 30 05:09:32.762880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.762728 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df01352d-3e53-4bcc-a30a-a2b291f95879" containerName="istio-proxy" Oct 30 05:09:32.762987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.762907 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="df01352d-3e53-4bcc-a30a-a2b291f95879" containerName="istio-proxy" Oct 30 05:09:32.858380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.856423 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8"] Oct 30 05:09:32.858380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.856592 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" Oct 30 05:09:32.952459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.952330 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3d55bd\"" Oct 30 05:09:32.993423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.984541 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8\" (UID: \"2a7858bf-8a5c-44d6-ba1d-f37f7e567124\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" Oct 30 05:09:32.993423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:32.984747 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcc69\" (UniqueName: \"kubernetes.io/projected/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-kube-api-access-lcc69\") pod \"payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8\" (UID: \"2a7858bf-8a5c-44d6-ba1d-f37f7e567124\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" Oct 30 05:09:33.114398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.088657 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8\" (UID: \"2a7858bf-8a5c-44d6-ba1d-f37f7e567124\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" Oct 30 05:09:33.114398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.088880 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcc69\" (UniqueName: \"kubernetes.io/projected/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-kube-api-access-lcc69\") pod \"payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8\" (UID: \"2a7858bf-8a5c-44d6-ba1d-f37f7e567124\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" Oct 30 05:09:33.121013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.119341 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8\" (UID: \"2a7858bf-8a5c-44d6-ba1d-f37f7e567124\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" Oct 30 05:09:33.217888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.217808 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcc69\" (UniqueName: \"kubernetes.io/projected/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-kube-api-access-lcc69\") pod \"payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8\" (UID: \"2a7858bf-8a5c-44d6-ba1d-f37f7e567124\") " pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" Oct 30 05:09:33.322969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.322925 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" Oct 30 05:09:33.549706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.549547 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" Oct 30 05:09:33.606935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.605287 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf76d60-8cf0-4608-bd40-7139520656c9-kube-api-access-67wvm\" (UniqueName: \"kubernetes.io/projected/dbf76d60-8cf0-4608-bd40-7139520656c9-kube-api-access-67wvm\") pod \"dbf76d60-8cf0-4608-bd40-7139520656c9\" (UID: \"dbf76d60-8cf0-4608-bd40-7139520656c9\") " Oct 30 05:09:33.606935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.605573 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-xks-ca\") pod \"dbf76d60-8cf0-4608-bd40-7139520656c9\" (UID: \"dbf76d60-8cf0-4608-bd40-7139520656c9\") " Oct 30 05:09:33.606935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.605666 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-openshift-ca\") pod \"dbf76d60-8cf0-4608-bd40-7139520656c9\" (UID: \"dbf76d60-8cf0-4608-bd40-7139520656c9\") " Oct 30 05:09:33.606935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.606622 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-openshift-ca" pod "dbf76d60-8cf0-4608-bd40-7139520656c9" (UID: "dbf76d60-8cf0-4608-bd40-7139520656c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:33.610237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.610204 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf76d60-8cf0-4608-bd40-7139520656c9-kube-api-access-67wvm" pod "dbf76d60-8cf0-4608-bd40-7139520656c9" (UID: "dbf76d60-8cf0-4608-bd40-7139520656c9"). InnerVolumeSpecName "kube-api-access-67wvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:33.610237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.610210 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-xks-ca" pod "dbf76d60-8cf0-4608-bd40-7139520656c9" (UID: "dbf76d60-8cf0-4608-bd40-7139520656c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:09:33.708277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.708196 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:33.708277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.708236 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf76d60-8cf0-4608-bd40-7139520656c9-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:33.708277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.708254 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67wvm\" (UniqueName: \"kubernetes.io/projected/dbf76d60-8cf0-4608-bd40-7139520656c9-kube-api-access-67wvm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:33.723400 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:09:33.723351 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7858bf_8a5c_44d6_ba1d_f37f7e567124.slice/crio-c62285268e33bca5031a4fc3bc14a92468a7e29d3bfed40626ea124b6d2747b2 WatchSource:0}: Error finding container c62285268e33bca5031a4fc3bc14a92468a7e29d3bfed40626ea124b6d2747b2: Status 404 returned error can't find the container with id c62285268e33bca5031a4fc3bc14a92468a7e29d3bfed40626ea124b6d2747b2 Oct 30 05:09:33.724833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:33.724806 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8"] Oct 30 05:09:35.634454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:34.204476 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" event={"ID":"2a7858bf-8a5c-44d6-ba1d-f37f7e567124","Type":"ContainerStarted","Data":"3136733369e648d24a0fb2d37b0d89e8ce45361f86d10a0cae6b5708a89106dd"} Oct 30 05:09:35.634454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:34.204547 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" event={"ID":"2a7858bf-8a5c-44d6-ba1d-f37f7e567124","Type":"ContainerStarted","Data":"c62285268e33bca5031a4fc3bc14a92468a7e29d3bfed40626ea124b6d2747b2"} Oct 30 05:09:35.634454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:34.204579 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" Oct 30 05:09:35.634454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:34.206479 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" event={"ID":"dbf76d60-8cf0-4608-bd40-7139520656c9","Type":"ContainerDied","Data":"2b8641185f41f04046f090d3696cee8321722b7ec53dde133a13436ab2af7ca5"} Oct 30 05:09:35.634454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:34.206520 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b8641185f41f04046f090d3696cee8321722b7ec53dde133a13436ab2af7ca5" Oct 30 05:09:35.634454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:34.206566 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7" Oct 30 05:09:35.634454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:34.248552 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" podStartSLOduration=2.248538345 podStartE2EDuration="2.248538345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:09:34.245214712 +0000 UTC m=+1498.772981499" watchObservedRunningTime="2026-10-30 05:09:34.248538345 +0000 UTC m=+1498.776305131" Oct 30 05:09:35.899481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:35.899403 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:09:35.899481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:35.899403 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:09:36.228081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:36.227982 6587 generic.go:356] "Generic (PLEG): container finished" podID="793b0190-5368-47f1-ae26-e6f736e9e878" containerID="37e7d3b9792a464344d3b75dfd735121bd8ef138dfed4e4e7db6d480e2c5e16c" exitCode=0 Oct 30 05:09:36.228274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:36.228138 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" event={"ID":"793b0190-5368-47f1-ae26-e6f736e9e878","Type":"ContainerDied","Data":"37e7d3b9792a464344d3b75dfd735121bd8ef138dfed4e4e7db6d480e2c5e16c"} Oct 30 05:09:36.228783 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:36.228740 6587 scope.go:122] "RemoveContainer" containerID="37e7d3b9792a464344d3b75dfd735121bd8ef138dfed4e4e7db6d480e2c5e16c" Oct 30 05:09:36.231378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:36.231328 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:36.231513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:36.231429 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:36.231513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:36.231477 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:36.232091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:36.232067 6587 generic.go:356] "Generic (PLEG): container finished" podID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerID="98dd0487492a898bea84251bab5e1fb2938a4491fc2a5c450e55dc5296df4501" exitCode=0 Oct 30 05:09:36.232233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:36.232216 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" event={"ID":"8a0adf45-f6fc-41c4-9c8b-142f595fd03f","Type":"ContainerDied","Data":"98dd0487492a898bea84251bab5e1fb2938a4491fc2a5c450e55dc5296df4501"} Oct 30 05:09:36.232598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:36.232580 6587 scope.go:122] "RemoveContainer" containerID="98dd0487492a898bea84251bab5e1fb2938a4491fc2a5c450e55dc5296df4501" Oct 30 05:09:36.234615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:36.234581 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:36.234694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:36.234663 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:36.234733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:36.234709 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:37.202569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.202255 6587 scope.go:122] "RemoveContainer" containerID="04214d72b50b344b3e02f65cf9a68e43c4041a0780210b12ece763be5f93dec0" Oct 30 05:09:37.214050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.214024 6587 scope.go:122] "RemoveContainer" containerID="f5587e89959c30fdbef5ba6e515773e4dea318fb2bdf75366235a2b037e91bf2" Oct 30 05:09:37.301541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.301087 6587 scope.go:122] "RemoveContainer" containerID="72911799b67081faf4f99a0184ed7f064787b5a12f4f44be7d1a03ffd873cf57" Oct 30 05:09:37.303105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.303075 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" event={"ID":"793b0190-5368-47f1-ae26-e6f736e9e878","Type":"ContainerStarted","Data":"51ced6ec618ad6680dce6b504f410d5f891154acd39c011019adb6fd27936924"} Oct 30 05:09:37.305657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.305429 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" event={"ID":"8a0adf45-f6fc-41c4-9c8b-142f595fd03f","Type":"ContainerStarted","Data":"80be2ce5cb3062cc402f516986a1e6ecb12bf821af7d663da972ac1093bdc3ab"} Oct 30 05:09:37.312237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.312219 6587 scope.go:122] "RemoveContainer" containerID="0c234ee2b4b495280571d336ca346424239d290e85b3ccaaad9733f19b27cba7" Oct 30 05:09:37.381469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.381438 6587 scope.go:122] "RemoveContainer" containerID="9a084897c25fb371ca4f0548b783d76ae6570a6de4d180b83e5012d72bd2795c" Oct 30 05:09:37.398065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.398039 6587 scope.go:122] "RemoveContainer" containerID="302e10fb64357845c3ffc1f54b7551171cc4432f9d73932ebf5ac473425cd0c3" Oct 30 05:09:37.408968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.408883 6587 scope.go:122] "RemoveContainer" containerID="923e175705a6e8baaaa5542115457ef09ec110673db3086114ff14cb3791d40a" Oct 30 05:09:37.483596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.483558 6587 scope.go:122] "RemoveContainer" containerID="3086cb06eb8733851631394bc61088e3787eb3d2c44548d229b68caf567e99f4" Oct 30 05:09:37.499217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.499196 6587 scope.go:122] "RemoveContainer" containerID="c33f3c03473c87e1075919da48e5ed994b71adcf3f40ee3620df9271b316d7f5" Oct 30 05:09:37.504860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.504843 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:09:37.505009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.504988 6587 patch_prober.go:29] interesting pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 05:09:37.505075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.505032 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 05:09:37.565491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.563370 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:09:37.583514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.580561 6587 patch_prober.go:29] interesting pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 05:09:37.583514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:37.580624 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 05:09:38.508383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:38.505767 6587 patch_prober.go:29] interesting pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 05:09:38.508383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:38.505829 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 05:09:38.592383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:38.580373 6587 patch_prober.go:29] interesting pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 05:09:38.592383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:38.580470 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 05:09:39.215131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.215099 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg"] Oct 30 05:09:39.219389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.215668 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" podUID="25829776-064f-435f-8969-28682c538f0d" containerName="istio-proxy" containerID="cri-o://ff1fb278af756643d7776ef269a2d13c404ee694d959edbd8ab8b3e56fd5e09e" gracePeriod=30 Oct 30 05:09:39.386653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.384847 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk"] Oct 30 05:09:39.398879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.398843 6587 generic.go:356] "Generic (PLEG): container finished" podID="484fe4bc-1006-4745-ac99-408b72f728d8" containerID="fe9d7b90177dff739bb0d249895d600784cb509bda8762551a3ad665e6a01077" exitCode=0 Oct 30 05:09:39.399126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.399104 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" event={"ID":"484fe4bc-1006-4745-ac99-408b72f728d8","Type":"ContainerDied","Data":"fe9d7b90177dff739bb0d249895d600784cb509bda8762551a3ad665e6a01077"} Oct 30 05:09:39.399486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.399467 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj"] Oct 30 05:09:39.399705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.399685 6587 scope.go:122] "RemoveContainer" containerID="fe9d7b90177dff739bb0d249895d600784cb509bda8762551a3ad665e6a01077" Oct 30 05:09:39.401757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.401725 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:39.401861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.401790 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:39.401861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.401823 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:09:39.511018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.510968 6587 patch_prober.go:29] interesting pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 05:09:39.511503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.511028 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 05:09:39.588775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.588738 6587 patch_prober.go:29] interesting pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 05:09:39.588940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.588795 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 05:09:39.909155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.909121 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt"] Oct 30 05:09:39.986689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.930351 6587 cpu_manager.go:414] "containerMap: removing container" podUID="dbf76d60-8cf0-4608-bd40-7139520656c9" containerName="revoke-keys" Oct 30 05:09:39.986689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.930399 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf76d60-8cf0-4608-bd40-7139520656c9" containerName="revoke-keys" Oct 30 05:09:39.986689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.931054 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf76d60-8cf0-4608-bd40-7139520656c9" containerName="revoke-keys" Oct 30 05:09:39.994392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:39.988121 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" Oct 30 05:09:40.069390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.053468 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2vbd\" (UniqueName: \"kubernetes.io/projected/1bc93cf9-794e-4c19-b442-74e59b7e6949-kube-api-access-c2vbd\") pod \"maas-api-revoke-keys-e2e-mt-ef844616-tbkxt\" (UID: \"1bc93cf9-794e-4c19-b442-74e59b7e6949\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" Oct 30 05:09:40.069390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.053570 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ef844616-tbkxt\" (UID: \"1bc93cf9-794e-4c19-b442-74e59b7e6949\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" Oct 30 05:09:40.069390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.053853 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ef844616-tbkxt\" (UID: \"1bc93cf9-794e-4c19-b442-74e59b7e6949\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" Oct 30 05:09:40.069390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.054723 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:09:40.069390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.055443 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:09:40.069390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.055548 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:09:40.133379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.106630 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:09:40.165444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.163764 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ef844616-tbkxt\" (UID: \"1bc93cf9-794e-4c19-b442-74e59b7e6949\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" Oct 30 05:09:40.165444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.163994 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2vbd\" (UniqueName: \"kubernetes.io/projected/1bc93cf9-794e-4c19-b442-74e59b7e6949-kube-api-access-c2vbd\") pod \"maas-api-revoke-keys-e2e-mt-ef844616-tbkxt\" (UID: \"1bc93cf9-794e-4c19-b442-74e59b7e6949\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" Oct 30 05:09:40.165444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.164041 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ef844616-tbkxt\" (UID: \"1bc93cf9-794e-4c19-b442-74e59b7e6949\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" Oct 30 05:09:40.223657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.167583 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ef844616-tbkxt\" (UID: \"1bc93cf9-794e-4c19-b442-74e59b7e6949\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" Oct 30 05:09:40.223657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.167643 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt"] Oct 30 05:09:40.506497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.455121 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ef844616-tbkxt\" (UID: \"1bc93cf9-794e-4c19-b442-74e59b7e6949\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" Oct 30 05:09:40.531449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.510466 6587 patch_prober.go:29] interesting pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 05:09:40.531449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.521497 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 05:09:40.531449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.511115 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2vbd\" (UniqueName: \"kubernetes.io/projected/1bc93cf9-794e-4c19-b442-74e59b7e6949-kube-api-access-c2vbd\") pod \"maas-api-revoke-keys-e2e-mt-ef844616-tbkxt\" (UID: \"1bc93cf9-794e-4c19-b442-74e59b7e6949\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" Oct 30 05:09:40.562571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.562538 6587 patch_prober.go:29] interesting pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 05:09:40.562826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.562798 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 05:09:40.625099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.625059 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" event={"ID":"484fe4bc-1006-4745-ac99-408b72f728d8","Type":"ContainerStarted","Data":"f1910e6f4fcdf6a8707221cabf8106086038eee6e283e52afa5b73b05cea2b4f"} Oct 30 05:09:40.717035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:40.717002 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" Oct 30 05:09:41.410385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:41.410208 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:09:41.418383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:41.415341 6587 patch_prober.go:29] interesting pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 05:09:41.418383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:41.415426 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 05:09:41.507484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:41.506952 6587 patch_prober.go:29] interesting pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 05:09:41.507484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:41.507014 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 05:09:41.566421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:41.565392 6587 patch_prober.go:29] interesting pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 05:09:41.566421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:41.565447 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 05:09:41.697696 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:41.697210 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt"] Oct 30 05:09:41.699267 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:41.699236 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" event={"ID":"1bc93cf9-794e-4c19-b442-74e59b7e6949","Type":"ContainerStarted","Data":"c4c837b31814d23f8254be2b4d2114d0c8b23cc673a976cf9d4a25252cd43ac0"} Oct 30 05:09:42.398405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:42.389443 6587 patch_prober.go:29] interesting pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 05:09:42.398405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:42.389504 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 05:09:42.517442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:42.508318 6587 patch_prober.go:29] interesting pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 05:09:42.517442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:42.508397 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 05:09:42.562844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:42.562805 6587 patch_prober.go:29] interesting pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 05:09:42.563135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:42.563107 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 05:09:42.701397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:42.697525 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:09:42.766180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:42.766102 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" event={"ID":"1bc93cf9-794e-4c19-b442-74e59b7e6949","Type":"ContainerStarted","Data":"07de315e7e3fca9a7a91d6a03f21fc32a9fe41832d4864c8545bf41d09b10b76"} Oct 30 05:09:42.843912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:42.843847 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" podStartSLOduration=3.843827613 podStartE2EDuration="3.843827613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:09:42.815613272 +0000 UTC m=+1507.343380073" watchObservedRunningTime="2026-10-30 05:09:42.843827613 +0000 UTC m=+1507.371594403" Oct 30 05:09:43.001381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:42.999137 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:43.217479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:43.215906 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:09:43.381556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:43.381517 6587 patch_prober.go:29] interesting pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 05:09:43.381736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:43.381572 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 05:09:43.615278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:43.615245 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:09:43.622460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:43.622434 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:09:43.628768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:43.628745 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:09:43.797835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:43.797803 6587 generic.go:356] "Generic (PLEG): container finished" podID="1bc93cf9-794e-4c19-b442-74e59b7e6949" containerID="07de315e7e3fca9a7a91d6a03f21fc32a9fe41832d4864c8545bf41d09b10b76" exitCode=0 Oct 30 05:09:43.798661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:43.797892 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" event={"ID":"1bc93cf9-794e-4c19-b442-74e59b7e6949","Type":"ContainerDied","Data":"07de315e7e3fca9a7a91d6a03f21fc32a9fe41832d4864c8545bf41d09b10b76"} Oct 30 05:09:44.000385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:43.999844 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:09:44.001411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.001392 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:09:44.004185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.004166 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:09:44.381610 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.381580 6587 patch_prober.go:29] interesting pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 05:09:44.381743 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.381634 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 05:09:44.385820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.385762 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" podUID="f53cf9b1-71c2-418b-947c-9c2771e68784" containerName="payload-processing" containerID="cri-o://e3399684f5d420e5e1486cf1b0122c0c394beb30a8f627608f56ae5094796806" gracePeriod=25 Oct 30 05:09:44.400016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.399985 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" podUID="1a7bf576-f225-4be6-9f22-99fc919b5576" containerName="payload-pre-processing" containerID="cri-o://5bd799c68e5529b9fe88a8e30f67f8b8f45f4c5e2fa271f4151327c039b06621" gracePeriod=25 Oct 30 05:09:44.655621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.655586 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:09:44.803025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.802986 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-credential-socket\") pod \"25829776-064f-435f-8969-28682c538f0d\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " Oct 30 05:09:44.803505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.803087 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25829776-064f-435f-8969-28682c538f0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25829776-064f-435f-8969-28682c538f0d-istio-podinfo\") pod \"25829776-064f-435f-8969-28682c538f0d\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " Oct 30 05:09:44.803505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.803257 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-socket\") pod \"25829776-064f-435f-8969-28682c538f0d\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " Oct 30 05:09:44.803505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.803304 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-kube-api-access-6l9rp\" (UniqueName: \"kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-kube-api-access-6l9rp\") pod \"25829776-064f-435f-8969-28682c538f0d\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " Oct 30 05:09:44.803505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.803396 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-certs\") pod \"25829776-064f-435f-8969-28682c538f0d\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " Oct 30 05:09:44.803505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.803473 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25829776-064f-435f-8969-28682c538f0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25829776-064f-435f-8969-28682c538f0d-istiod-ca-cert\") pod \"25829776-064f-435f-8969-28682c538f0d\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " Oct 30 05:09:44.803505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.803505 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-envoy\") pod \"25829776-064f-435f-8969-28682c538f0d\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " Oct 30 05:09:44.803863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.803538 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-data\") pod \"25829776-064f-435f-8969-28682c538f0d\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " Oct 30 05:09:44.803863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.803633 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-istio-token\" (UniqueName: \"kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-istio-token\") pod \"25829776-064f-435f-8969-28682c538f0d\" (UID: \"25829776-064f-435f-8969-28682c538f0d\") " Oct 30 05:09:44.803863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.803745 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-socket" pod "25829776-064f-435f-8969-28682c538f0d" (UID: "25829776-064f-435f-8969-28682c538f0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.804533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.804021 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-certs" pod "25829776-064f-435f-8969-28682c538f0d" (UID: "25829776-064f-435f-8969-28682c538f0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.804533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.804092 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-credential-socket" pod "25829776-064f-435f-8969-28682c538f0d" (UID: "25829776-064f-435f-8969-28682c538f0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.804533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.804261 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-data" pod "25829776-064f-435f-8969-28682c538f0d" (UID: "25829776-064f-435f-8969-28682c538f0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.804533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.804437 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25829776-064f-435f-8969-28682c538f0d-istiod-ca-cert" pod "25829776-064f-435f-8969-28682c538f0d" (UID: "25829776-064f-435f-8969-28682c538f0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:44.804838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.804542 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.804838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.804562 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.804838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.804575 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.804838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.804588 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.814574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.814488 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25829776-064f-435f-8969-28682c538f0d-istio-podinfo" pod "25829776-064f-435f-8969-28682c538f0d" (UID: "25829776-064f-435f-8969-28682c538f0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:09:44.815099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.815057 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-envoy" pod "25829776-064f-435f-8969-28682c538f0d" (UID: "25829776-064f-435f-8969-28682c538f0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:44.815715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.815677 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-istio-token" pod "25829776-064f-435f-8969-28682c538f0d" (UID: "25829776-064f-435f-8969-28682c538f0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:44.855563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.832924 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-kube-api-access-6l9rp" pod "25829776-064f-435f-8969-28682c538f0d" (UID: "25829776-064f-435f-8969-28682c538f0d"). InnerVolumeSpecName "kube-api-access-6l9rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:44.870624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.866061 6587 generic.go:356] "Generic (PLEG): container finished" podID="f53cf9b1-71c2-418b-947c-9c2771e68784" containerID="e3399684f5d420e5e1486cf1b0122c0c394beb30a8f627608f56ae5094796806" exitCode=0 Oct 30 05:09:44.870624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.866161 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" event={"ID":"f53cf9b1-71c2-418b-947c-9c2771e68784","Type":"ContainerDied","Data":"e3399684f5d420e5e1486cf1b0122c0c394beb30a8f627608f56ae5094796806"} Oct 30 05:09:44.870624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.868179 6587 generic.go:356] "Generic (PLEG): container finished" podID="25829776-064f-435f-8969-28682c538f0d" containerID="ff1fb278af756643d7776ef269a2d13c404ee694d959edbd8ab8b3e56fd5e09e" exitCode=0 Oct 30 05:09:44.870624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.868478 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" event={"ID":"25829776-064f-435f-8969-28682c538f0d","Type":"ContainerDied","Data":"ff1fb278af756643d7776ef269a2d13c404ee694d959edbd8ab8b3e56fd5e09e"} Oct 30 05:09:44.870624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.868487 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" Oct 30 05:09:44.870624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.868554 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg" event={"ID":"25829776-064f-435f-8969-28682c538f0d","Type":"ContainerDied","Data":"d3418deabfbcca85a9f0c8caf32be2de7b81b93efb239dc11d24007f1f42b3d7"} Oct 30 05:09:44.870624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.868585 6587 scope.go:122] "RemoveContainer" containerID="ff1fb278af756643d7776ef269a2d13c404ee694d959edbd8ab8b3e56fd5e09e" Oct 30 05:09:44.891903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.891640 6587 generic.go:356] "Generic (PLEG): container finished" podID="1a7bf576-f225-4be6-9f22-99fc919b5576" containerID="5bd799c68e5529b9fe88a8e30f67f8b8f45f4c5e2fa271f4151327c039b06621" exitCode=0 Oct 30 05:09:44.893105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.893046 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" event={"ID":"1a7bf576-f225-4be6-9f22-99fc919b5576","Type":"ContainerDied","Data":"5bd799c68e5529b9fe88a8e30f67f8b8f45f4c5e2fa271f4151327c039b06621"} Oct 30 05:09:44.906239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.906052 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l9rp\" (UniqueName: \"kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-kube-api-access-6l9rp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.906239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.906090 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25829776-064f-435f-8969-28682c538f0d-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.906239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.906105 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25829776-064f-435f-8969-28682c538f0d-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.906239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.906119 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25829776-064f-435f-8969-28682c538f0d-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.906239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.906133 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25829776-064f-435f-8969-28682c538f0d-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:44.965336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.965209 6587 scope.go:122] "RemoveContainer" containerID="ff1fb278af756643d7776ef269a2d13c404ee694d959edbd8ab8b3e56fd5e09e" Oct 30 05:09:44.965880 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:09:44.965769 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1fb278af756643d7776ef269a2d13c404ee694d959edbd8ab8b3e56fd5e09e\": container with ID starting with ff1fb278af756643d7776ef269a2d13c404ee694d959edbd8ab8b3e56fd5e09e not found: ID does not exist" containerID="ff1fb278af756643d7776ef269a2d13c404ee694d959edbd8ab8b3e56fd5e09e" Oct 30 05:09:44.965880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.965807 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1fb278af756643d7776ef269a2d13c404ee694d959edbd8ab8b3e56fd5e09e"} err="failed to get container status \"ff1fb278af756643d7776ef269a2d13c404ee694d959edbd8ab8b3e56fd5e09e\": rpc error: code = NotFound desc = could not find container \"ff1fb278af756643d7776ef269a2d13c404ee694d959edbd8ab8b3e56fd5e09e\": container with ID starting with ff1fb278af756643d7776ef269a2d13c404ee694d959edbd8ab8b3e56fd5e09e not found: ID does not exist" Oct 30 05:09:44.966657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.966601 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg"] Oct 30 05:09:44.980332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.980301 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg"] Oct 30 05:09:44.989123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:44.989080 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" Oct 30 05:09:45.098929 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.098537 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" Oct 30 05:09:45.109681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.109642 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7bf576-f225-4be6-9f22-99fc919b5576-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7bf576-f225-4be6-9f22-99fc919b5576-plugins-config-volume\") pod \"1a7bf576-f225-4be6-9f22-99fc919b5576\" (UID: \"1a7bf576-f225-4be6-9f22-99fc919b5576\") " Oct 30 05:09:45.110374 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.110240 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7bf576-f225-4be6-9f22-99fc919b5576-plugins-config-volume" pod "1a7bf576-f225-4be6-9f22-99fc919b5576" (UID: "1a7bf576-f225-4be6-9f22-99fc919b5576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:45.110724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.110703 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7bf576-f225-4be6-9f22-99fc919b5576-kube-api-access-gxx89\" (UniqueName: \"kubernetes.io/projected/1a7bf576-f225-4be6-9f22-99fc919b5576-kube-api-access-gxx89\") pod \"1a7bf576-f225-4be6-9f22-99fc919b5576\" (UID: \"1a7bf576-f225-4be6-9f22-99fc919b5576\") " Oct 30 05:09:45.112081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.112057 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7bf576-f225-4be6-9f22-99fc919b5576-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.113791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.113759 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7bf576-f225-4be6-9f22-99fc919b5576-kube-api-access-gxx89" pod "1a7bf576-f225-4be6-9f22-99fc919b5576" (UID: "1a7bf576-f225-4be6-9f22-99fc919b5576"). InnerVolumeSpecName "kube-api-access-gxx89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:45.361441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.277288 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53cf9b1-71c2-418b-947c-9c2771e68784-kube-api-access-fspd9\" (UniqueName: \"kubernetes.io/projected/f53cf9b1-71c2-418b-947c-9c2771e68784-kube-api-access-fspd9\") pod \"f53cf9b1-71c2-418b-947c-9c2771e68784\" (UID: \"f53cf9b1-71c2-418b-947c-9c2771e68784\") " Oct 30 05:09:45.361441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.277404 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53cf9b1-71c2-418b-947c-9c2771e68784-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53cf9b1-71c2-418b-947c-9c2771e68784-plugins-config-volume\") pod \"f53cf9b1-71c2-418b-947c-9c2771e68784\" (UID: \"f53cf9b1-71c2-418b-947c-9c2771e68784\") " Oct 30 05:09:45.361441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.278166 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxx89\" (UniqueName: \"kubernetes.io/projected/1a7bf576-f225-4be6-9f22-99fc919b5576-kube-api-access-gxx89\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.361441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.313587 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" Oct 30 05:09:45.361441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.321619 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53cf9b1-71c2-418b-947c-9c2771e68784-plugins-config-volume" pod "f53cf9b1-71c2-418b-947c-9c2771e68784" (UID: "f53cf9b1-71c2-418b-947c-9c2771e68784"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:45.413538 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.413490 6587 patch_prober.go:29] interesting pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 05:09:45.413809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.413778 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 05:09:45.486385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.486342 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53cf9b1-71c2-418b-947c-9c2771e68784-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.501574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.501520 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53cf9b1-71c2-418b-947c-9c2771e68784-kube-api-access-fspd9" pod "f53cf9b1-71c2-418b-947c-9c2771e68784" (UID: "f53cf9b1-71c2-418b-947c-9c2771e68784"). InnerVolumeSpecName "kube-api-access-fspd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:45.588108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.588059 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fspd9\" (UniqueName: \"kubernetes.io/projected/f53cf9b1-71c2-418b-947c-9c2771e68784-kube-api-access-fspd9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.591922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.591902 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25829776-064f-435f-8969-28682c538f0d" path="/var/lib/kubelet/pods/25829776-064f-435f-8969-28682c538f0d/volumes" Oct 30 05:09:45.610843 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.610819 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" Oct 30 05:09:45.689005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.688919 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-openshift-ca\") pod \"1bc93cf9-794e-4c19-b442-74e59b7e6949\" (UID: \"1bc93cf9-794e-4c19-b442-74e59b7e6949\") " Oct 30 05:09:45.689296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.689264 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-xks-ca\") pod \"1bc93cf9-794e-4c19-b442-74e59b7e6949\" (UID: \"1bc93cf9-794e-4c19-b442-74e59b7e6949\") " Oct 30 05:09:45.689418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.689332 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-openshift-ca" pod "1bc93cf9-794e-4c19-b442-74e59b7e6949" (UID: "1bc93cf9-794e-4c19-b442-74e59b7e6949"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:45.689490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.689414 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc93cf9-794e-4c19-b442-74e59b7e6949-kube-api-access-c2vbd\" (UniqueName: \"kubernetes.io/projected/1bc93cf9-794e-4c19-b442-74e59b7e6949-kube-api-access-c2vbd\") pod \"1bc93cf9-794e-4c19-b442-74e59b7e6949\" (UID: \"1bc93cf9-794e-4c19-b442-74e59b7e6949\") " Oct 30 05:09:45.690034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.690015 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.692051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.692021 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-xks-ca" pod "1bc93cf9-794e-4c19-b442-74e59b7e6949" (UID: "1bc93cf9-794e-4c19-b442-74e59b7e6949"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:09:45.692156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.692033 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc93cf9-794e-4c19-b442-74e59b7e6949-kube-api-access-c2vbd" pod "1bc93cf9-794e-4c19-b442-74e59b7e6949" (UID: "1bc93cf9-794e-4c19-b442-74e59b7e6949"). InnerVolumeSpecName "kube-api-access-c2vbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:45.791318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.791285 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2vbd\" (UniqueName: \"kubernetes.io/projected/1bc93cf9-794e-4c19-b442-74e59b7e6949-kube-api-access-c2vbd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.791318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.791313 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc93cf9-794e-4c19-b442-74e59b7e6949-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:45.905866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.905505 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" event={"ID":"f53cf9b1-71c2-418b-947c-9c2771e68784","Type":"ContainerDied","Data":"b355cc2f7f598a994addb0e0601a6027669ad70e6231f9a7a91c2f70206de28f"} Oct 30 05:09:45.905866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.905554 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk" Oct 30 05:09:45.905866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.905604 6587 scope.go:122] "RemoveContainer" containerID="e3399684f5d420e5e1486cf1b0122c0c394beb30a8f627608f56ae5094796806" Oct 30 05:09:45.908942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.908915 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" event={"ID":"1bc93cf9-794e-4c19-b442-74e59b7e6949","Type":"ContainerDied","Data":"c4c837b31814d23f8254be2b4d2114d0c8b23cc673a976cf9d4a25252cd43ac0"} Oct 30 05:09:45.909067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.908966 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4c837b31814d23f8254be2b4d2114d0c8b23cc673a976cf9d4a25252cd43ac0" Oct 30 05:09:45.909067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.908953 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt" Oct 30 05:09:45.910596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.910577 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" Oct 30 05:09:45.910720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.910602 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj" event={"ID":"1a7bf576-f225-4be6-9f22-99fc919b5576","Type":"ContainerDied","Data":"95c324ed1a7c137e6407db7484bec5abfd12d541a0c7194b9d60a7498381063c"} Oct 30 05:09:45.945426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.941620 6587 scope.go:122] "RemoveContainer" containerID="5bd799c68e5529b9fe88a8e30f67f8b8f45f4c5e2fa271f4151327c039b06621" Oct 30 05:09:45.972138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.972111 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk"] Oct 30 05:09:45.975787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:45.975765 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk"] Oct 30 05:09:46.010314 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:46.010285 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj"] Oct 30 05:09:46.025529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:46.015398 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj"] Oct 30 05:09:46.381518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:46.381483 6587 patch_prober.go:29] interesting pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 05:09:46.381684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:46.381542 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 05:09:47.387158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:47.387125 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:09:47.388450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:47.388422 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:09:47.584249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:47.584217 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7bf576-f225-4be6-9f22-99fc919b5576" path="/var/lib/kubelet/pods/1a7bf576-f225-4be6-9f22-99fc919b5576/volumes" Oct 30 05:09:47.584854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:47.584836 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53cf9b1-71c2-418b-947c-9c2771e68784" path="/var/lib/kubelet/pods/f53cf9b1-71c2-418b-947c-9c2771e68784/volumes" Oct 30 05:09:50.400124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:50.400083 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx"] Oct 30 05:09:55.401344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:55.401284 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" podUID="ad008a25-4aea-4e83-99b1-1a1e7d03a877" containerName="payload-pre-processing" containerID="cri-o://7fb2b498dbb5e061c2b4a9f95374699d657d10a7fa650c56fc2d38938c593443" gracePeriod=25 Oct 30 05:09:55.700656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:55.700631 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" Oct 30 05:09:55.764670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:55.764632 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad008a25-4aea-4e83-99b1-1a1e7d03a877-kube-api-access-g8n8m\" (UniqueName: \"kubernetes.io/projected/ad008a25-4aea-4e83-99b1-1a1e7d03a877-kube-api-access-g8n8m\") pod \"ad008a25-4aea-4e83-99b1-1a1e7d03a877\" (UID: \"ad008a25-4aea-4e83-99b1-1a1e7d03a877\") " Oct 30 05:09:55.765014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:55.764993 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad008a25-4aea-4e83-99b1-1a1e7d03a877-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad008a25-4aea-4e83-99b1-1a1e7d03a877-plugins-config-volume\") pod \"ad008a25-4aea-4e83-99b1-1a1e7d03a877\" (UID: \"ad008a25-4aea-4e83-99b1-1a1e7d03a877\") " Oct 30 05:09:55.766186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:55.766157 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad008a25-4aea-4e83-99b1-1a1e7d03a877-plugins-config-volume" pod "ad008a25-4aea-4e83-99b1-1a1e7d03a877" (UID: "ad008a25-4aea-4e83-99b1-1a1e7d03a877"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:55.768298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:55.768273 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad008a25-4aea-4e83-99b1-1a1e7d03a877-kube-api-access-g8n8m" pod "ad008a25-4aea-4e83-99b1-1a1e7d03a877" (UID: "ad008a25-4aea-4e83-99b1-1a1e7d03a877"). InnerVolumeSpecName "kube-api-access-g8n8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:55.867126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:55.867088 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8n8m\" (UniqueName: \"kubernetes.io/projected/ad008a25-4aea-4e83-99b1-1a1e7d03a877-kube-api-access-g8n8m\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:55.867126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:55.867119 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad008a25-4aea-4e83-99b1-1a1e7d03a877-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:09:55.987831 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:09:55.987728 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:09:56.141394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:56.141316 6587 generic.go:356] "Generic (PLEG): container finished" podID="ad008a25-4aea-4e83-99b1-1a1e7d03a877" containerID="7fb2b498dbb5e061c2b4a9f95374699d657d10a7fa650c56fc2d38938c593443" exitCode=0 Oct 30 05:09:56.141394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:56.141389 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" Oct 30 05:09:56.141645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:56.141388 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" event={"ID":"ad008a25-4aea-4e83-99b1-1a1e7d03a877","Type":"ContainerDied","Data":"7fb2b498dbb5e061c2b4a9f95374699d657d10a7fa650c56fc2d38938c593443"} Oct 30 05:09:56.141645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:56.141536 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx" event={"ID":"ad008a25-4aea-4e83-99b1-1a1e7d03a877","Type":"ContainerDied","Data":"c9637723dc6ae5285d9da1774c6a78b837e21887cf2ac82c0bd590b82f08e345"} Oct 30 05:09:56.141645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:56.141570 6587 scope.go:122] "RemoveContainer" containerID="7fb2b498dbb5e061c2b4a9f95374699d657d10a7fa650c56fc2d38938c593443" Oct 30 05:09:56.155185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:56.155164 6587 scope.go:122] "RemoveContainer" containerID="7fb2b498dbb5e061c2b4a9f95374699d657d10a7fa650c56fc2d38938c593443" Oct 30 05:09:56.155541 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:09:56.155518 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb2b498dbb5e061c2b4a9f95374699d657d10a7fa650c56fc2d38938c593443\": container with ID starting with 7fb2b498dbb5e061c2b4a9f95374699d657d10a7fa650c56fc2d38938c593443 not found: ID does not exist" containerID="7fb2b498dbb5e061c2b4a9f95374699d657d10a7fa650c56fc2d38938c593443" Oct 30 05:09:56.155654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:56.155561 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb2b498dbb5e061c2b4a9f95374699d657d10a7fa650c56fc2d38938c593443"} err="failed to get container status \"7fb2b498dbb5e061c2b4a9f95374699d657d10a7fa650c56fc2d38938c593443\": rpc error: code = NotFound desc = could not find container \"7fb2b498dbb5e061c2b4a9f95374699d657d10a7fa650c56fc2d38938c593443\": container with ID starting with 7fb2b498dbb5e061c2b4a9f95374699d657d10a7fa650c56fc2d38938c593443 not found: ID does not exist" Oct 30 05:09:56.198557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:56.198530 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx"] Oct 30 05:09:56.211701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:56.211673 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx"] Oct 30 05:09:57.583943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:57.583908 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad008a25-4aea-4e83-99b1-1a1e7d03a877" path="/var/lib/kubelet/pods/ad008a25-4aea-4e83-99b1-1a1e7d03a877/volumes" Oct 30 05:09:58.943158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.942113 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf"] Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943381 6587 cpu_manager.go:414] "containerMap: removing container" podUID="25829776-064f-435f-8969-28682c538f0d" containerName="istio-proxy" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943403 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25829776-064f-435f-8969-28682c538f0d" containerName="istio-proxy" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943422 6587 cpu_manager.go:414] "containerMap: removing container" podUID="ad008a25-4aea-4e83-99b1-1a1e7d03a877" containerName="payload-pre-processing" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943430 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad008a25-4aea-4e83-99b1-1a1e7d03a877" containerName="payload-pre-processing" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943495 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1bc93cf9-794e-4c19-b442-74e59b7e6949" containerName="revoke-keys" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943503 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc93cf9-794e-4c19-b442-74e59b7e6949" containerName="revoke-keys" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943519 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f53cf9b1-71c2-418b-947c-9c2771e68784" containerName="payload-processing" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943526 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53cf9b1-71c2-418b-947c-9c2771e68784" containerName="payload-processing" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943544 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1a7bf576-f225-4be6-9f22-99fc919b5576" containerName="payload-pre-processing" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943551 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7bf576-f225-4be6-9f22-99fc919b5576" containerName="payload-pre-processing" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943712 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc93cf9-794e-4c19-b442-74e59b7e6949" containerName="revoke-keys" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943727 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53cf9b1-71c2-418b-947c-9c2771e68784" containerName="payload-processing" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943737 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7bf576-f225-4be6-9f22-99fc919b5576" containerName="payload-pre-processing" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943754 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="25829776-064f-435f-8969-28682c538f0d" containerName="istio-proxy" Oct 30 05:09:58.943782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.943761 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad008a25-4aea-4e83-99b1-1a1e7d03a877" containerName="payload-pre-processing" Oct 30 05:09:58.957806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.957770 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf"] Oct 30 05:09:58.958071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.958055 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" Oct 30 05:09:58.975033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.975008 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:09:58.975170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:58.975081 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:09:59.028941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:59.028905 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:09:59.029146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:59.029060 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:09:59.049391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:59.046330 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19d238c2-8p4vf\" (UID: \"1dde22e0-86b6-4456-a7eb-26e22afecd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" Oct 30 05:09:59.049391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:59.046575 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19d238c2-8p4vf\" (UID: \"1dde22e0-86b6-4456-a7eb-26e22afecd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" Oct 30 05:09:59.049391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:59.047111 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6slz\" (UniqueName: \"kubernetes.io/projected/1dde22e0-86b6-4456-a7eb-26e22afecd69-kube-api-access-r6slz\") pod \"maas-api-revoke-sub-19d238c2-8p4vf\" (UID: \"1dde22e0-86b6-4456-a7eb-26e22afecd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" Oct 30 05:09:59.152380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:59.149459 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19d238c2-8p4vf\" (UID: \"1dde22e0-86b6-4456-a7eb-26e22afecd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" Oct 30 05:09:59.152380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:59.149726 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6slz\" (UniqueName: \"kubernetes.io/projected/1dde22e0-86b6-4456-a7eb-26e22afecd69-kube-api-access-r6slz\") pod \"maas-api-revoke-sub-19d238c2-8p4vf\" (UID: \"1dde22e0-86b6-4456-a7eb-26e22afecd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" Oct 30 05:09:59.152380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:59.149859 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19d238c2-8p4vf\" (UID: \"1dde22e0-86b6-4456-a7eb-26e22afecd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" Oct 30 05:09:59.154917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:59.154884 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19d238c2-8p4vf\" (UID: \"1dde22e0-86b6-4456-a7eb-26e22afecd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" Oct 30 05:09:59.165436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:59.155907 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19d238c2-8p4vf\" (UID: \"1dde22e0-86b6-4456-a7eb-26e22afecd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" Oct 30 05:09:59.177294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:59.177257 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6slz\" (UniqueName: \"kubernetes.io/projected/1dde22e0-86b6-4456-a7eb-26e22afecd69-kube-api-access-r6slz\") pod \"maas-api-revoke-sub-19d238c2-8p4vf\" (UID: \"1dde22e0-86b6-4456-a7eb-26e22afecd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" Oct 30 05:09:59.277557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:59.277485 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" Oct 30 05:09:59.617581 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:09:59.617547 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dde22e0_86b6_4456_a7eb_26e22afecd69.slice/crio-24f78e328c94c63d98da2e2b89bb33fea4bcf5dbe8fc3b07718f6fbb5cd1d5d1 WatchSource:0}: Error finding container 24f78e328c94c63d98da2e2b89bb33fea4bcf5dbe8fc3b07718f6fbb5cd1d5d1: Status 404 returned error can't find the container with id 24f78e328c94c63d98da2e2b89bb33fea4bcf5dbe8fc3b07718f6fbb5cd1d5d1 Oct 30 05:09:59.619969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:09:59.619929 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf"] Oct 30 05:10:00.209286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:00.209191 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" event={"ID":"1dde22e0-86b6-4456-a7eb-26e22afecd69","Type":"ContainerStarted","Data":"fd94f09d18152ecd726bc1e141d6cb0d22558d73b232c1c7a55be82b7a2155ad"} Oct 30 05:10:00.209286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:00.209243 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" event={"ID":"1dde22e0-86b6-4456-a7eb-26e22afecd69","Type":"ContainerStarted","Data":"24f78e328c94c63d98da2e2b89bb33fea4bcf5dbe8fc3b07718f6fbb5cd1d5d1"} Oct 30 05:10:00.252638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:00.252575 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" podStartSLOduration=2.252555277 podStartE2EDuration="2.252555277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:00.251258486 +0000 UTC m=+1524.779025273" watchObservedRunningTime="2026-10-30 05:10:00.252555277 +0000 UTC m=+1524.780322064" Oct 30 05:10:01.219183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:01.219151 6587 generic.go:356] "Generic (PLEG): container finished" podID="1dde22e0-86b6-4456-a7eb-26e22afecd69" containerID="fd94f09d18152ecd726bc1e141d6cb0d22558d73b232c1c7a55be82b7a2155ad" exitCode=0 Oct 30 05:10:01.219675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:01.219227 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" event={"ID":"1dde22e0-86b6-4456-a7eb-26e22afecd69","Type":"ContainerDied","Data":"fd94f09d18152ecd726bc1e141d6cb0d22558d73b232c1c7a55be82b7a2155ad"} Oct 30 05:10:02.446740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:02.446708 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" Oct 30 05:10:02.609050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:02.609018 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-openshift-ca\") pod \"1dde22e0-86b6-4456-a7eb-26e22afecd69\" (UID: \"1dde22e0-86b6-4456-a7eb-26e22afecd69\") " Oct 30 05:10:02.609216 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:02.609196 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-xks-ca\") pod \"1dde22e0-86b6-4456-a7eb-26e22afecd69\" (UID: \"1dde22e0-86b6-4456-a7eb-26e22afecd69\") " Oct 30 05:10:02.609332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:02.609307 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-openshift-ca" pod "1dde22e0-86b6-4456-a7eb-26e22afecd69" (UID: "1dde22e0-86b6-4456-a7eb-26e22afecd69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:02.609391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:02.609337 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dde22e0-86b6-4456-a7eb-26e22afecd69-kube-api-access-r6slz\" (UniqueName: \"kubernetes.io/projected/1dde22e0-86b6-4456-a7eb-26e22afecd69-kube-api-access-r6slz\") pod \"1dde22e0-86b6-4456-a7eb-26e22afecd69\" (UID: \"1dde22e0-86b6-4456-a7eb-26e22afecd69\") " Oct 30 05:10:02.610029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:02.609994 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:02.612011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:02.611965 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dde22e0-86b6-4456-a7eb-26e22afecd69-kube-api-access-r6slz" pod "1dde22e0-86b6-4456-a7eb-26e22afecd69" (UID: "1dde22e0-86b6-4456-a7eb-26e22afecd69"). InnerVolumeSpecName "kube-api-access-r6slz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:02.612833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:02.612808 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-xks-ca" pod "1dde22e0-86b6-4456-a7eb-26e22afecd69" (UID: "1dde22e0-86b6-4456-a7eb-26e22afecd69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:10:02.712762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:02.711904 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dde22e0-86b6-4456-a7eb-26e22afecd69-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:02.712762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:02.711949 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6slz\" (UniqueName: \"kubernetes.io/projected/1dde22e0-86b6-4456-a7eb-26e22afecd69-kube-api-access-r6slz\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:03.238173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:03.238139 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" Oct 30 05:10:03.238350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:03.238174 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf" event={"ID":"1dde22e0-86b6-4456-a7eb-26e22afecd69","Type":"ContainerDied","Data":"24f78e328c94c63d98da2e2b89bb33fea4bcf5dbe8fc3b07718f6fbb5cd1d5d1"} Oct 30 05:10:03.238350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:03.238251 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24f78e328c94c63d98da2e2b89bb33fea4bcf5dbe8fc3b07718f6fbb5cd1d5d1" Oct 30 05:10:07.438425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.438382 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9"] Oct 30 05:10:07.439237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.439219 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1dde22e0-86b6-4456-a7eb-26e22afecd69" containerName="revoke-keys" Oct 30 05:10:07.439296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.439238 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dde22e0-86b6-4456-a7eb-26e22afecd69" containerName="revoke-keys" Oct 30 05:10:07.439433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.439399 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dde22e0-86b6-4456-a7eb-26e22afecd69" containerName="revoke-keys" Oct 30 05:10:07.442306 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.442285 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" Oct 30 05:10:07.462849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.462812 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9"] Oct 30 05:10:07.470228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.470201 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:10:07.470730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.470710 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:10:07.470924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.470750 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:10:07.470924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.470792 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:10:07.512646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.512611 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9\" (UID: \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" Oct 30 05:10:07.512825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.512750 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9\" (UID: \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" Oct 30 05:10:07.512912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.512897 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjfpw\" (UniqueName: \"kubernetes.io/projected/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-kube-api-access-jjfpw\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9\" (UID: \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" Oct 30 05:10:07.614428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.614387 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9\" (UID: \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" Oct 30 05:10:07.614628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.614534 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9\" (UID: \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" Oct 30 05:10:07.614738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.614715 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfpw\" (UniqueName: \"kubernetes.io/projected/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-kube-api-access-jjfpw\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9\" (UID: \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" Oct 30 05:10:07.615302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.615268 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9\" (UID: \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" Oct 30 05:10:07.619303 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.619278 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9\" (UID: \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" Oct 30 05:10:07.635749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.635724 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfpw\" (UniqueName: \"kubernetes.io/projected/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-kube-api-access-jjfpw\") pod \"maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9\" (UID: \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" Oct 30 05:10:07.785188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:07.785096 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" Oct 30 05:10:08.526467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:08.502704 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9"] Oct 30 05:10:08.621811 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:08.621769 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" event={"ID":"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e","Type":"ContainerStarted","Data":"a550378c15cb17e371e7517afbd65f8e0d46a505e36e7a6df48bdd735eb68111"} Oct 30 05:10:09.268862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:09.268825 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt"] Oct 30 05:10:09.282941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:09.282899 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef844616-tbkxt"] Oct 30 05:10:09.584381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:09.584337 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc93cf9-794e-4c19-b442-74e59b7e6949" path="/var/lib/kubelet/pods/1bc93cf9-794e-4c19-b442-74e59b7e6949/volumes" Oct 30 05:10:09.667673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:09.667640 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" event={"ID":"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e","Type":"ContainerStarted","Data":"744bc187addfd64ca9887801eb411a945300c7fe92968dad785cd233d0b0ee1a"} Oct 30 05:10:10.677789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:10.677752 6587 generic.go:356] "Generic (PLEG): container finished" podID="2a1ecc5b-fddb-472d-9c70-5a37164d5d9e" containerID="744bc187addfd64ca9887801eb411a945300c7fe92968dad785cd233d0b0ee1a" exitCode=0 Oct 30 05:10:10.678189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:10.677835 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" event={"ID":"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e","Type":"ContainerDied","Data":"744bc187addfd64ca9887801eb411a945300c7fe92968dad785cd233d0b0ee1a"} Oct 30 05:10:11.914770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:11.914747 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" Oct 30 05:10:12.057259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:12.057227 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-kube-api-access-jjfpw\" (UniqueName: \"kubernetes.io/projected/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-kube-api-access-jjfpw\") pod \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\" (UID: \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\") " Oct 30 05:10:12.057450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:12.057276 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-openshift-ca\") pod \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\" (UID: \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\") " Oct 30 05:10:12.057450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:12.057327 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-xks-ca\") pod \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\" (UID: \"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e\") " Oct 30 05:10:12.058337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:12.058299 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-openshift-ca" pod "2a1ecc5b-fddb-472d-9c70-5a37164d5d9e" (UID: "2a1ecc5b-fddb-472d-9c70-5a37164d5d9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:12.060149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:12.060122 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-xks-ca" pod "2a1ecc5b-fddb-472d-9c70-5a37164d5d9e" (UID: "2a1ecc5b-fddb-472d-9c70-5a37164d5d9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:10:12.060325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:12.060313 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-kube-api-access-jjfpw" pod "2a1ecc5b-fddb-472d-9c70-5a37164d5d9e" (UID: "2a1ecc5b-fddb-472d-9c70-5a37164d5d9e"). InnerVolumeSpecName "kube-api-access-jjfpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:12.164025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:12.163985 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjfpw\" (UniqueName: \"kubernetes.io/projected/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-kube-api-access-jjfpw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:12.164294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:12.164247 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:12.164294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:12.164273 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:12.712989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:12.712957 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" event={"ID":"2a1ecc5b-fddb-472d-9c70-5a37164d5d9e","Type":"ContainerDied","Data":"a550378c15cb17e371e7517afbd65f8e0d46a505e36e7a6df48bdd735eb68111"} Oct 30 05:10:12.713145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:12.713005 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a550378c15cb17e371e7517afbd65f8e0d46a505e36e7a6df48bdd735eb68111" Oct 30 05:10:12.713145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:12.713056 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9" Oct 30 05:10:15.039649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:15.039608 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx"] Oct 30 05:10:15.050872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:15.050842 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ae3bb0-tn6tx"] Oct 30 05:10:15.592862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:15.592818 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e626e6-4d6a-4568-ac7e-f100030c4f63" path="/var/lib/kubelet/pods/f8e626e6-4d6a-4568-ac7e-f100030c4f63/volumes" Oct 30 05:10:26.020448 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:10:26.020341 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:10:35.040798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.040747 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk"] Oct 30 05:10:35.041600 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.041577 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2a1ecc5b-fddb-472d-9c70-5a37164d5d9e" containerName="revoke-keys" Oct 30 05:10:35.041823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.041789 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1ecc5b-fddb-472d-9c70-5a37164d5d9e" containerName="revoke-keys" Oct 30 05:10:35.042108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.042094 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1ecc5b-fddb-472d-9c70-5a37164d5d9e" containerName="revoke-keys" Oct 30 05:10:35.045210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.045186 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.056862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.056840 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92"] Oct 30 05:10:35.060668 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.060642 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.080154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.080128 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk"] Oct 30 05:10:35.086924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.086868 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92"] Oct 30 05:10:35.219985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.219953 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.220164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.219999 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.220164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220092 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.220164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220153 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.220277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220248 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.220329 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220317 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.220393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220380 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.220438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220419 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.220517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220449 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.220517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220466 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.220583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220536 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.220621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220582 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mfff\" (UniqueName: \"kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-kube-api-access-9mfff\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.220657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220619 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.220657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220640 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.220732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220709 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.220808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220796 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.220846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220828 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.220886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.220857 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kdrz\" (UniqueName: \"kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-kube-api-access-8kdrz\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.322230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322188 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.322539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322294 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.322539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322367 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.322539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322400 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.322539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322441 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.322539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322468 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.322539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322517 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.322891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322599 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mfff\" (UniqueName: \"kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-kube-api-access-9mfff\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.322891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322666 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.322891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322691 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.322891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322733 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.322891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322808 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.322891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322860 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.323201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322913 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kdrz\" (UniqueName: \"kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-kube-api-access-8kdrz\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.323201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322914 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.323201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322959 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.323201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.322975 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.323201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.323030 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.323201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.323092 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.323201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.323144 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.323618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.323307 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.323618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.323541 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.323618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.323555 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.323618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.323600 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.323885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.323829 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.324046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.324024 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.324288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.324256 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.324520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.324493 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.325503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.325475 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.325591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.325549 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.325680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.325663 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.325840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.325820 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.344187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.344159 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.344322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.344228 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.345023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.344997 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kdrz\" (UniqueName: \"kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-kube-api-access-8kdrz\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.345023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.345014 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mfff\" (UniqueName: \"kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-kube-api-access-9mfff\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.361453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.361427 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:35.376327 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.376297 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:35.696156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.696127 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk"] Oct 30 05:10:35.697990 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:10:35.697962 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae29f15_fdba_4ab5_8901_7b95e3d936e1.slice/crio-81cb0d331c418851a760925f5f5b28a9f500632d64a4a7a594b9b92828966df6 WatchSource:0}: Error finding container 81cb0d331c418851a760925f5f5b28a9f500632d64a4a7a594b9b92828966df6: Status 404 returned error can't find the container with id 81cb0d331c418851a760925f5f5b28a9f500632d64a4a7a594b9b92828966df6 Oct 30 05:10:35.700752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.700722 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92"] Oct 30 05:10:35.701281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.701245 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:35.701460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.701325 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:35.701460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.701372 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:35.709432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.709393 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:35.709560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.709535 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:35.709681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:35.709628 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:10:36.006126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:36.006038 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" event={"ID":"5ae29f15-fdba-4ab5-8901-7b95e3d936e1","Type":"ContainerStarted","Data":"5730bf6c99b1a2f77d9b1aea9337073f06a875ee362e71e46d7dcbe02a4ec70c"} Oct 30 05:10:36.006347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:36.006138 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" event={"ID":"5ae29f15-fdba-4ab5-8901-7b95e3d936e1","Type":"ContainerStarted","Data":"81cb0d331c418851a760925f5f5b28a9f500632d64a4a7a594b9b92828966df6"} Oct 30 05:10:36.008988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:36.008952 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" event={"ID":"e1124d49-ccfe-456e-a0ba-871f0d8d0992","Type":"ContainerStarted","Data":"65e95848d50902821cafc856de6391e530e1bda36b3a58a6a3f5c4ba3979094b"} Oct 30 05:10:36.009114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:36.009017 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" event={"ID":"e1124d49-ccfe-456e-a0ba-871f0d8d0992","Type":"ContainerStarted","Data":"fc2743f3b23c75fc0254b8af9c2489ba8a54feace73fab21441fb76a8db2a16e"} Oct 30 05:10:36.051783 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:36.051726 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" podStartSLOduration=1.051712783 podStartE2EDuration="1.051712783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:36.050516977 +0000 UTC m=+1560.578283772" watchObservedRunningTime="2026-10-30 05:10:36.051712783 +0000 UTC m=+1560.579479568" Oct 30 05:10:36.100089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:36.100022 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" podStartSLOduration=1.100000178 podStartE2EDuration="1.100000178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:36.094258337 +0000 UTC m=+1560.622025137" watchObservedRunningTime="2026-10-30 05:10:36.100000178 +0000 UTC m=+1560.627766965" Oct 30 05:10:36.362435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:36.362404 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:36.363907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:36.363878 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 05:10:36.364020 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:36.363923 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" podUID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 05:10:36.377054 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:36.377032 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:36.378470 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:36.378448 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 05:10:36.378577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:36.378493 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" podUID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 05:10:37.363816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:37.363767 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 05:10:37.364459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:37.364427 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" podUID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 05:10:37.377384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:37.376784 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 05:10:37.377384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:37.376840 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" podUID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 05:10:37.907689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:37.907651 6587 scope.go:122] "RemoveContainer" containerID="ee9fd681185ce3ff3072cc97772a74ebb35684841da6ead65b5b9a8f9f2ce8ee" Oct 30 05:10:37.969624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:37.964746 6587 scope.go:122] "RemoveContainer" containerID="fab138b415cd5d6c9f171412b4d0cecddfc85b03561cf0e5a2f2c65425ecf698" Oct 30 05:10:38.374412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:38.373058 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 05:10:38.374412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:38.373115 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" podUID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 05:10:38.415629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:38.410727 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 05:10:38.415629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:38.410876 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" podUID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 05:10:39.171157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:39.171121 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7"] Oct 30 05:10:39.208383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:39.187846 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:10:39.290420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:39.290327 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-528d67-frfz7"] Oct 30 05:10:39.371917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:39.371876 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 05:10:39.373503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:39.373460 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" podUID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 05:10:39.380698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:39.380520 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 05:10:39.380698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:39.380573 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" podUID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 05:10:39.593986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:39.593945 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf76d60-8cf0-4608-bd40-7139520656c9" path="/var/lib/kubelet/pods/dbf76d60-8cf0-4608-bd40-7139520656c9/volumes" Oct 30 05:10:40.378383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:40.376694 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 05:10:40.378383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:40.376756 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" podUID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 05:10:40.378383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:40.377382 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 05:10:40.378383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:40.377423 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" podUID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 05:10:41.372427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:41.371802 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 05:10:41.372427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:41.371871 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" podUID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 05:10:41.387397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:41.380764 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 05:10:41.387397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:41.380822 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" podUID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 05:10:41.814055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:41.814016 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9"] Oct 30 05:10:41.896231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:41.895783 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c613f5ea-s67z9"] Oct 30 05:10:42.367384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:42.365746 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 05:10:42.367384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:42.365808 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" podUID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 05:10:42.378817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:42.378775 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 05:10:42.379291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:42.378833 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" podUID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 05:10:43.361824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:43.361779 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 05:10:43.362036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:43.361845 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" podUID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 05:10:43.377276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:43.377239 6587 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 05:10:43.377468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:43.377316 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" podUID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 05:10:43.596963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:43.596923 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1ecc5b-fddb-472d-9c70-5a37164d5d9e" path="/var/lib/kubelet/pods/2a1ecc5b-fddb-472d-9c70-5a37164d5d9e/volumes" Oct 30 05:10:44.664756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:44.664678 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:44.666476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:44.666450 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:44.668944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:44.668924 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:10:44.755431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:44.755390 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:44.755981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:44.755914 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:44.761781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:44.761496 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:10:46.261247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:46.261210 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm"] Oct 30 05:10:46.262085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:46.262051 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" containerID="cri-o://ecdc88c635db71add9b411c6f8c4e091aa7fa08fd538f400e9eb489a1aaf5763" gracePeriod=30 Oct 30 05:10:46.856161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:46.856119 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt"] Oct 30 05:10:46.861861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:46.861839 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" Oct 30 05:10:46.878054 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:46.878026 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt"] Oct 30 05:10:46.892384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:46.885536 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:10:46.892384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:46.885568 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:10:46.892384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:46.885592 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:10:46.892384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:46.885659 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:10:46.940794 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:46.940760 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-645aef-6xfmt\" (UID: \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" Oct 30 05:10:46.941186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:46.941155 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsm8p\" (UniqueName: \"kubernetes.io/projected/68fa70ec-593f-49ff-be9d-e5159c2e1f61-kube-api-access-fsm8p\") pod \"maas-api-revoke-keys-e2e-api-b-645aef-6xfmt\" (UID: \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" Oct 30 05:10:46.941264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:46.941245 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-645aef-6xfmt\" (UID: \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" Oct 30 05:10:47.043394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:47.043338 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsm8p\" (UniqueName: \"kubernetes.io/projected/68fa70ec-593f-49ff-be9d-e5159c2e1f61-kube-api-access-fsm8p\") pod \"maas-api-revoke-keys-e2e-api-b-645aef-6xfmt\" (UID: \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" Oct 30 05:10:47.043599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:47.043429 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-645aef-6xfmt\" (UID: \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" Oct 30 05:10:47.043599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:47.043564 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-645aef-6xfmt\" (UID: \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" Oct 30 05:10:47.044143 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:47.044117 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-645aef-6xfmt\" (UID: \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" Oct 30 05:10:47.046114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:47.046091 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-645aef-6xfmt\" (UID: \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" Oct 30 05:10:47.064249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:47.064223 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsm8p\" (UniqueName: \"kubernetes.io/projected/68fa70ec-593f-49ff-be9d-e5159c2e1f61-kube-api-access-fsm8p\") pod \"maas-api-revoke-keys-e2e-api-b-645aef-6xfmt\" (UID: \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" Oct 30 05:10:47.178595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:47.178492 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" Oct 30 05:10:47.490801 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:10:47.490768 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fa70ec_593f_49ff_be9d_e5159c2e1f61.slice/crio-dd34c380eb19129c6a42a75c23b1ea4e0163820968037271551acd1061b8e844 WatchSource:0}: Error finding container dd34c380eb19129c6a42a75c23b1ea4e0163820968037271551acd1061b8e844: Status 404 returned error can't find the container with id dd34c380eb19129c6a42a75c23b1ea4e0163820968037271551acd1061b8e844 Oct 30 05:10:47.493875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:47.493812 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt"] Oct 30 05:10:47.519662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:47.519635 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" event={"ID":"68fa70ec-593f-49ff-be9d-e5159c2e1f61","Type":"ContainerStarted","Data":"dd34c380eb19129c6a42a75c23b1ea4e0163820968037271551acd1061b8e844"} Oct 30 05:10:48.530613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:48.530580 6587 generic.go:356] "Generic (PLEG): container finished" podID="68fa70ec-593f-49ff-be9d-e5159c2e1f61" containerID="6f332238272d7e975d5e455f7c495b3a11f40427f2db6f770813c49079dcf80b" exitCode=0 Oct 30 05:10:48.531066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:48.530667 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" event={"ID":"68fa70ec-593f-49ff-be9d-e5159c2e1f61","Type":"ContainerDied","Data":"6f332238272d7e975d5e455f7c495b3a11f40427f2db6f770813c49079dcf80b"} Oct 30 05:10:49.778506 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:49.777034 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" Oct 30 05:10:49.898835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:49.898798 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-xks-ca\") pod \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\" (UID: \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\") " Oct 30 05:10:49.899042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:49.898961 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68fa70ec-593f-49ff-be9d-e5159c2e1f61-kube-api-access-fsm8p\" (UniqueName: \"kubernetes.io/projected/68fa70ec-593f-49ff-be9d-e5159c2e1f61-kube-api-access-fsm8p\") pod \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\" (UID: \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\") " Oct 30 05:10:49.899198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:49.899183 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-openshift-ca\") pod \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\" (UID: \"68fa70ec-593f-49ff-be9d-e5159c2e1f61\") " Oct 30 05:10:49.899631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:49.899600 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-openshift-ca" pod "68fa70ec-593f-49ff-be9d-e5159c2e1f61" (UID: "68fa70ec-593f-49ff-be9d-e5159c2e1f61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:49.900120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:49.900097 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:49.901110 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:49.901080 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-xks-ca" pod "68fa70ec-593f-49ff-be9d-e5159c2e1f61" (UID: "68fa70ec-593f-49ff-be9d-e5159c2e1f61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:10:49.901208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:49.901152 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fa70ec-593f-49ff-be9d-e5159c2e1f61-kube-api-access-fsm8p" pod "68fa70ec-593f-49ff-be9d-e5159c2e1f61" (UID: "68fa70ec-593f-49ff-be9d-e5159c2e1f61"). InnerVolumeSpecName "kube-api-access-fsm8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:50.001327 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:50.001287 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsm8p\" (UniqueName: \"kubernetes.io/projected/68fa70ec-593f-49ff-be9d-e5159c2e1f61-kube-api-access-fsm8p\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:50.001327 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:50.001324 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68fa70ec-593f-49ff-be9d-e5159c2e1f61-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:50.553850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:50.553817 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" Oct 30 05:10:50.553850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:50.553834 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt" event={"ID":"68fa70ec-593f-49ff-be9d-e5159c2e1f61","Type":"ContainerDied","Data":"dd34c380eb19129c6a42a75c23b1ea4e0163820968037271551acd1061b8e844"} Oct 30 05:10:50.554075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:50.553886 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd34c380eb19129c6a42a75c23b1ea4e0163820968037271551acd1061b8e844" Oct 30 05:10:51.571676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.571633 6587 generic.go:356] "Generic (PLEG): container finished" podID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerID="ecdc88c635db71add9b411c6f8c4e091aa7fa08fd538f400e9eb489a1aaf5763" exitCode=0 Oct 30 05:10:51.572141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.571686 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" event={"ID":"a07d2e44-c567-42b0-a939-3d2f33b2a078","Type":"ContainerDied","Data":"ecdc88c635db71add9b411c6f8c4e091aa7fa08fd538f400e9eb489a1aaf5763"} Oct 30 05:10:51.572141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.571781 6587 scope.go:122] "RemoveContainer" containerID="16d1d7930e931dca3ed9ec2af156e8c6c7389331ea66571f92579cfc639912b5" Oct 30 05:10:51.605510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.605490 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:10:51.742741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.742697 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-credential-socket\") pod \"a07d2e44-c567-42b0-a939-3d2f33b2a078\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " Oct 30 05:10:51.743120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.742774 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-socket\") pod \"a07d2e44-c567-42b0-a939-3d2f33b2a078\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " Oct 30 05:10:51.743120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.742819 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-token\" (UniqueName: \"kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-token\") pod \"a07d2e44-c567-42b0-a939-3d2f33b2a078\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " Oct 30 05:10:51.743120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.742868 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-data\") pod \"a07d2e44-c567-42b0-a939-3d2f33b2a078\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " Oct 30 05:10:51.743120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.742968 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07d2e44-c567-42b0-a939-3d2f33b2a078-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07d2e44-c567-42b0-a939-3d2f33b2a078-istiod-ca-cert\") pod \"a07d2e44-c567-42b0-a939-3d2f33b2a078\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " Oct 30 05:10:51.743120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.743031 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-envoy\") pod \"a07d2e44-c567-42b0-a939-3d2f33b2a078\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " Oct 30 05:10:51.743120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.743065 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-kube-api-access-jkmb9\" (UniqueName: \"kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-kube-api-access-jkmb9\") pod \"a07d2e44-c567-42b0-a939-3d2f33b2a078\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " Oct 30 05:10:51.743594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.743152 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-podinfo\") pod \"a07d2e44-c567-42b0-a939-3d2f33b2a078\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " Oct 30 05:10:51.743594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.743299 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-certs\") pod \"a07d2e44-c567-42b0-a939-3d2f33b2a078\" (UID: \"a07d2e44-c567-42b0-a939-3d2f33b2a078\") " Oct 30 05:10:51.744584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.744545 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-data" pod "a07d2e44-c567-42b0-a939-3d2f33b2a078" (UID: "a07d2e44-c567-42b0-a939-3d2f33b2a078"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:51.744793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.744761 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-credential-socket" pod "a07d2e44-c567-42b0-a939-3d2f33b2a078" (UID: "a07d2e44-c567-42b0-a939-3d2f33b2a078"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:51.744793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.744777 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07d2e44-c567-42b0-a939-3d2f33b2a078-istiod-ca-cert" pod "a07d2e44-c567-42b0-a939-3d2f33b2a078" (UID: "a07d2e44-c567-42b0-a939-3d2f33b2a078"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:51.745211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.744944 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-socket" pod "a07d2e44-c567-42b0-a939-3d2f33b2a078" (UID: "a07d2e44-c567-42b0-a939-3d2f33b2a078"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:51.745426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.745385 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-certs" pod "a07d2e44-c567-42b0-a939-3d2f33b2a078" (UID: "a07d2e44-c567-42b0-a939-3d2f33b2a078"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:51.748153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.748116 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-podinfo" pod "a07d2e44-c567-42b0-a939-3d2f33b2a078" (UID: "a07d2e44-c567-42b0-a939-3d2f33b2a078"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:10:51.765503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.765444 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-envoy" pod "a07d2e44-c567-42b0-a939-3d2f33b2a078" (UID: "a07d2e44-c567-42b0-a939-3d2f33b2a078"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:51.765503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.765478 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-token" pod "a07d2e44-c567-42b0-a939-3d2f33b2a078" (UID: "a07d2e44-c567-42b0-a939-3d2f33b2a078"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:51.770858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.770831 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-kube-api-access-jkmb9" pod "a07d2e44-c567-42b0-a939-3d2f33b2a078" (UID: "a07d2e44-c567-42b0-a939-3d2f33b2a078"). InnerVolumeSpecName "kube-api-access-jkmb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:51.846109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.846024 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07d2e44-c567-42b0-a939-3d2f33b2a078-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:51.846109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.846062 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:51.846109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.846078 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkmb9\" (UniqueName: \"kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-kube-api-access-jkmb9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:51.846109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.846096 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:51.846109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.846111 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:51.846440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.846124 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:51.846440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.846139 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:51.846440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.846153 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:51.846440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:51.846166 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07d2e44-c567-42b0-a939-3d2f33b2a078-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:10:52.582060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:52.581988 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" Oct 30 05:10:52.582060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:52.582014 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm" event={"ID":"a07d2e44-c567-42b0-a939-3d2f33b2a078","Type":"ContainerDied","Data":"7201d0e8eb2fdbcab57e980cd4be9f09bf5af497ffb7578e03d5e88ff44a1356"} Oct 30 05:10:52.582060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:52.582073 6587 scope.go:122] "RemoveContainer" containerID="ecdc88c635db71add9b411c6f8c4e091aa7fa08fd538f400e9eb489a1aaf5763" Oct 30 05:10:52.692013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:52.689545 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm"] Oct 30 05:10:52.695849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:52.695816 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm"] Oct 30 05:10:53.586027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:53.585982 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" path="/var/lib/kubelet/pods/a07d2e44-c567-42b0-a939-3d2f33b2a078/volumes" Oct 30 05:10:56.022649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:56.014841 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:10:56.043394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:55.991628 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:56.170021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:56.169965 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:56.197701 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:10:56.197640 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:10:56.681385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:56.674033 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:56.805382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:56.804521 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:56.837382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:56.836432 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:56.837382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:56.836811 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:10:57.257864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.257807 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:57.264400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.264372 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:10:57.281073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.279482 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:57.296491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.295599 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:10:57.297670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.297316 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:10:57.302104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.298506 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:57.302104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.298887 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:10:57.458382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.457339 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:57.485635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.485476 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:57.772531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.772221 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:10:57.813474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.804980 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:57.814870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.814846 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:10:57.857382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.840311 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:57.857382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.847851 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:10:57.857382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.852035 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:10:57.857382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.852099 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:10:57.958927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.958893 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs"] Oct 30 05:10:57.980762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:57.980718 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:58.258781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:58.258747 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:10:58.269661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:58.269633 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk"] Oct 30 05:10:58.325677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:58.325329 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:10:58.507382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:58.485788 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:58.507382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:58.486301 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:10:58.755639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:58.755604 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:10:58.875131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:58.875003 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:10:58.933415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:58.907942 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:10:58.933628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:58.933511 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:10:59.491385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:59.471051 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:10:59.493062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:59.493018 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:10:59.686064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:59.686030 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:10:59.718416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:59.718336 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:10:59.718597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:10:59.718457 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:11:00.264285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.264249 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x"] Oct 30 05:11:00.265373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.265339 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" Oct 30 05:11:00.265373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.265374 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" Oct 30 05:11:00.265585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.265402 6587 cpu_manager.go:414] "containerMap: removing container" podUID="68fa70ec-593f-49ff-be9d-e5159c2e1f61" containerName="revoke-keys" Oct 30 05:11:00.265585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.265410 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68fa70ec-593f-49ff-be9d-e5159c2e1f61" containerName="revoke-keys" Oct 30 05:11:00.265585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.265450 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" Oct 30 05:11:00.265585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.265458 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" Oct 30 05:11:00.265797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.265669 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" Oct 30 05:11:00.265797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.265686 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="68fa70ec-593f-49ff-be9d-e5159c2e1f61" containerName="revoke-keys" Oct 30 05:11:00.265797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.265702 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07d2e44-c567-42b0-a939-3d2f33b2a078" containerName="istio-proxy" Oct 30 05:11:00.269731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.269711 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" Oct 30 05:11:00.282596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.282567 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x"] Oct 30 05:11:00.285926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.285881 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:11:00.290312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.290288 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:11:00.301621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.301592 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:11:00.301765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.301701 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:11:00.396146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.396101 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-543d08-qhr2x\" (UID: \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" Oct 30 05:11:00.396421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.396392 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-543d08-qhr2x\" (UID: \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" Oct 30 05:11:00.396679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.396648 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwzmw\" (UniqueName: \"kubernetes.io/projected/ec75741a-2af8-4f2f-8ca3-f891f46fd507-kube-api-access-xwzmw\") pod \"maas-api-revoke-keys-e2e-api-a-543d08-qhr2x\" (UID: \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" Oct 30 05:11:00.498210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.498166 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwzmw\" (UniqueName: \"kubernetes.io/projected/ec75741a-2af8-4f2f-8ca3-f891f46fd507-kube-api-access-xwzmw\") pod \"maas-api-revoke-keys-e2e-api-a-543d08-qhr2x\" (UID: \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" Oct 30 05:11:00.498672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.498422 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-543d08-qhr2x\" (UID: \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" Oct 30 05:11:00.498672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.498472 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-543d08-qhr2x\" (UID: \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" Oct 30 05:11:00.499203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.499180 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-543d08-qhr2x\" (UID: \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" Oct 30 05:11:00.500870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.500849 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-543d08-qhr2x\" (UID: \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" Oct 30 05:11:00.519751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.519683 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwzmw\" (UniqueName: \"kubernetes.io/projected/ec75741a-2af8-4f2f-8ca3-f891f46fd507-kube-api-access-xwzmw\") pod \"maas-api-revoke-keys-e2e-api-a-543d08-qhr2x\" (UID: \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" Oct 30 05:11:00.587870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.587817 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" Oct 30 05:11:00.917107 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:11:00.917076 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec75741a_2af8_4f2f_8ca3_f891f46fd507.slice/crio-e814c691b1a0601195ca7bff9fe07afe58d63d331c536871300454c4f39ea67a WatchSource:0}: Error finding container e814c691b1a0601195ca7bff9fe07afe58d63d331c536871300454c4f39ea67a: Status 404 returned error can't find the container with id e814c691b1a0601195ca7bff9fe07afe58d63d331c536871300454c4f39ea67a Oct 30 05:11:00.920009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:00.919981 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x"] Oct 30 05:11:01.736528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:01.736423 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" event={"ID":"ec75741a-2af8-4f2f-8ca3-f891f46fd507","Type":"ContainerStarted","Data":"256aae25030a239a61d45db8f58ca3482940b001f751b5eae02c0d08f4dc89fe"} Oct 30 05:11:01.736528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:01.736494 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" event={"ID":"ec75741a-2af8-4f2f-8ca3-f891f46fd507","Type":"ContainerStarted","Data":"e814c691b1a0601195ca7bff9fe07afe58d63d331c536871300454c4f39ea67a"} Oct 30 05:11:01.786326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:01.786272 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" podStartSLOduration=1.7862589789999999 podStartE2EDuration="1.786258979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:01.785054111 +0000 UTC m=+1586.312820897" watchObservedRunningTime="2026-10-30 05:11:01.786258979 +0000 UTC m=+1586.314025764" Oct 30 05:11:02.748602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:02.748569 6587 generic.go:356] "Generic (PLEG): container finished" podID="ec75741a-2af8-4f2f-8ca3-f891f46fd507" containerID="256aae25030a239a61d45db8f58ca3482940b001f751b5eae02c0d08f4dc89fe" exitCode=0 Oct 30 05:11:02.748994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:02.748653 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" event={"ID":"ec75741a-2af8-4f2f-8ca3-f891f46fd507","Type":"ContainerDied","Data":"256aae25030a239a61d45db8f58ca3482940b001f751b5eae02c0d08f4dc89fe"} Oct 30 05:11:02.960253 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:02.960192 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" podUID="67e16cca-13be-49d8-95a0-22b479aea9bd" containerName="payload-processing" containerID="cri-o://c981afade5c0c5896712bc7330f7e706df3000fb38301ec970da31aace415a1d" gracePeriod=25 Oct 30 05:11:03.262064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.262033 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" Oct 30 05:11:03.270651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.270619 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" podUID="cd420f62-adff-401e-8c12-6bf7eab929e6" containerName="payload-pre-processing" containerID="cri-o://9d58b27b686009c3fa686cdd2909b4819f4f5f82f8fe1ec6aee345b5fe7ec2ee" gracePeriod=25 Oct 30 05:11:03.362488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.358151 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e16cca-13be-49d8-95a0-22b479aea9bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e16cca-13be-49d8-95a0-22b479aea9bd-plugins-config-volume\") pod \"67e16cca-13be-49d8-95a0-22b479aea9bd\" (UID: \"67e16cca-13be-49d8-95a0-22b479aea9bd\") " Oct 30 05:11:03.362488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.358227 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e16cca-13be-49d8-95a0-22b479aea9bd-kube-api-access-fm49c\" (UniqueName: \"kubernetes.io/projected/67e16cca-13be-49d8-95a0-22b479aea9bd-kube-api-access-fm49c\") pod \"67e16cca-13be-49d8-95a0-22b479aea9bd\" (UID: \"67e16cca-13be-49d8-95a0-22b479aea9bd\") " Oct 30 05:11:03.362488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.358618 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e16cca-13be-49d8-95a0-22b479aea9bd-plugins-config-volume" pod "67e16cca-13be-49d8-95a0-22b479aea9bd" (UID: "67e16cca-13be-49d8-95a0-22b479aea9bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:03.362488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.359127 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e16cca-13be-49d8-95a0-22b479aea9bd-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.399389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.372845 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e16cca-13be-49d8-95a0-22b479aea9bd-kube-api-access-fm49c" pod "67e16cca-13be-49d8-95a0-22b479aea9bd" (UID: "67e16cca-13be-49d8-95a0-22b479aea9bd"). InnerVolumeSpecName "kube-api-access-fm49c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:03.461404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.460881 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm49c\" (UniqueName: \"kubernetes.io/projected/67e16cca-13be-49d8-95a0-22b479aea9bd-kube-api-access-fm49c\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.550744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.550418 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" Oct 30 05:11:03.663572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.663481 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd420f62-adff-401e-8c12-6bf7eab929e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd420f62-adff-401e-8c12-6bf7eab929e6-plugins-config-volume\") pod \"cd420f62-adff-401e-8c12-6bf7eab929e6\" (UID: \"cd420f62-adff-401e-8c12-6bf7eab929e6\") " Oct 30 05:11:03.663732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.663584 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd420f62-adff-401e-8c12-6bf7eab929e6-kube-api-access-jl49h\" (UniqueName: \"kubernetes.io/projected/cd420f62-adff-401e-8c12-6bf7eab929e6-kube-api-access-jl49h\") pod \"cd420f62-adff-401e-8c12-6bf7eab929e6\" (UID: \"cd420f62-adff-401e-8c12-6bf7eab929e6\") " Oct 30 05:11:03.663915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.663889 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd420f62-adff-401e-8c12-6bf7eab929e6-plugins-config-volume" pod "cd420f62-adff-401e-8c12-6bf7eab929e6" (UID: "cd420f62-adff-401e-8c12-6bf7eab929e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:03.664630 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.664603 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd420f62-adff-401e-8c12-6bf7eab929e6-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.665731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.665705 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd420f62-adff-401e-8c12-6bf7eab929e6-kube-api-access-jl49h" pod "cd420f62-adff-401e-8c12-6bf7eab929e6" (UID: "cd420f62-adff-401e-8c12-6bf7eab929e6"). InnerVolumeSpecName "kube-api-access-jl49h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:03.759595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.759559 6587 generic.go:356] "Generic (PLEG): container finished" podID="67e16cca-13be-49d8-95a0-22b479aea9bd" containerID="c981afade5c0c5896712bc7330f7e706df3000fb38301ec970da31aace415a1d" exitCode=0 Oct 30 05:11:03.760055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.759600 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" Oct 30 05:11:03.760055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.759634 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" event={"ID":"67e16cca-13be-49d8-95a0-22b479aea9bd","Type":"ContainerDied","Data":"c981afade5c0c5896712bc7330f7e706df3000fb38301ec970da31aace415a1d"} Oct 30 05:11:03.760055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.759689 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs" event={"ID":"67e16cca-13be-49d8-95a0-22b479aea9bd","Type":"ContainerDied","Data":"6a0e1481254cb65d612b5a18f60e15a9260225e7a7559c2ffd5a618401e95f21"} Oct 30 05:11:03.760055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.759719 6587 scope.go:122] "RemoveContainer" containerID="c981afade5c0c5896712bc7330f7e706df3000fb38301ec970da31aace415a1d" Oct 30 05:11:03.761978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.761948 6587 generic.go:356] "Generic (PLEG): container finished" podID="cd420f62-adff-401e-8c12-6bf7eab929e6" containerID="9d58b27b686009c3fa686cdd2909b4819f4f5f82f8fe1ec6aee345b5fe7ec2ee" exitCode=0 Oct 30 05:11:03.762688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.762506 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" Oct 30 05:11:03.763795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.763336 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" event={"ID":"cd420f62-adff-401e-8c12-6bf7eab929e6","Type":"ContainerDied","Data":"9d58b27b686009c3fa686cdd2909b4819f4f5f82f8fe1ec6aee345b5fe7ec2ee"} Oct 30 05:11:03.763795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.763431 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk" event={"ID":"cd420f62-adff-401e-8c12-6bf7eab929e6","Type":"ContainerDied","Data":"c28c873742ef3dac42051335667f23195167dd17569fe92f1bb104c32d684084"} Oct 30 05:11:03.765960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.765937 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl49h\" (UniqueName: \"kubernetes.io/projected/cd420f62-adff-401e-8c12-6bf7eab929e6-kube-api-access-jl49h\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.774405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.774340 6587 scope.go:122] "RemoveContainer" containerID="c981afade5c0c5896712bc7330f7e706df3000fb38301ec970da31aace415a1d" Oct 30 05:11:03.774669 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:11:03.774650 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c981afade5c0c5896712bc7330f7e706df3000fb38301ec970da31aace415a1d\": container with ID starting with c981afade5c0c5896712bc7330f7e706df3000fb38301ec970da31aace415a1d not found: ID does not exist" containerID="c981afade5c0c5896712bc7330f7e706df3000fb38301ec970da31aace415a1d" Oct 30 05:11:03.774722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.774676 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c981afade5c0c5896712bc7330f7e706df3000fb38301ec970da31aace415a1d"} err="failed to get container status \"c981afade5c0c5896712bc7330f7e706df3000fb38301ec970da31aace415a1d\": rpc error: code = NotFound desc = could not find container \"c981afade5c0c5896712bc7330f7e706df3000fb38301ec970da31aace415a1d\": container with ID starting with c981afade5c0c5896712bc7330f7e706df3000fb38301ec970da31aace415a1d not found: ID does not exist" Oct 30 05:11:03.774722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.774692 6587 scope.go:122] "RemoveContainer" containerID="9d58b27b686009c3fa686cdd2909b4819f4f5f82f8fe1ec6aee345b5fe7ec2ee" Oct 30 05:11:03.785049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.785028 6587 scope.go:122] "RemoveContainer" containerID="9d58b27b686009c3fa686cdd2909b4819f4f5f82f8fe1ec6aee345b5fe7ec2ee" Oct 30 05:11:03.785329 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:11:03.785307 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d58b27b686009c3fa686cdd2909b4819f4f5f82f8fe1ec6aee345b5fe7ec2ee\": container with ID starting with 9d58b27b686009c3fa686cdd2909b4819f4f5f82f8fe1ec6aee345b5fe7ec2ee not found: ID does not exist" containerID="9d58b27b686009c3fa686cdd2909b4819f4f5f82f8fe1ec6aee345b5fe7ec2ee" Oct 30 05:11:03.785452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.785339 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d58b27b686009c3fa686cdd2909b4819f4f5f82f8fe1ec6aee345b5fe7ec2ee"} err="failed to get container status \"9d58b27b686009c3fa686cdd2909b4819f4f5f82f8fe1ec6aee345b5fe7ec2ee\": rpc error: code = NotFound desc = could not find container \"9d58b27b686009c3fa686cdd2909b4819f4f5f82f8fe1ec6aee345b5fe7ec2ee\": container with ID starting with 9d58b27b686009c3fa686cdd2909b4819f4f5f82f8fe1ec6aee345b5fe7ec2ee not found: ID does not exist" Oct 30 05:11:03.833552 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.827977 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk"] Oct 30 05:11:03.852373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.852324 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk"] Oct 30 05:11:03.871711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.871679 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs"] Oct 30 05:11:03.876027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.875995 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs"] Oct 30 05:11:03.918946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.918878 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" Oct 30 05:11:03.969057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.969027 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-openshift-ca\") pod \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\" (UID: \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\") " Oct 30 05:11:03.969429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.969245 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec75741a-2af8-4f2f-8ca3-f891f46fd507-kube-api-access-xwzmw\" (UniqueName: \"kubernetes.io/projected/ec75741a-2af8-4f2f-8ca3-f891f46fd507-kube-api-access-xwzmw\") pod \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\" (UID: \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\") " Oct 30 05:11:03.969945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.969889 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-xks-ca\") pod \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\" (UID: \"ec75741a-2af8-4f2f-8ca3-f891f46fd507\") " Oct 30 05:11:03.971047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.971020 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-openshift-ca" pod "ec75741a-2af8-4f2f-8ca3-f891f46fd507" (UID: "ec75741a-2af8-4f2f-8ca3-f891f46fd507"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:03.972159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.972132 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec75741a-2af8-4f2f-8ca3-f891f46fd507-kube-api-access-xwzmw" pod "ec75741a-2af8-4f2f-8ca3-f891f46fd507" (UID: "ec75741a-2af8-4f2f-8ca3-f891f46fd507"). InnerVolumeSpecName "kube-api-access-xwzmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:03.973130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:03.973106 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-xks-ca" pod "ec75741a-2af8-4f2f-8ca3-f891f46fd507" (UID: "ec75741a-2af8-4f2f-8ca3-f891f46fd507"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:11:04.072248 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:04.072213 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:04.072248 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:04.072243 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec75741a-2af8-4f2f-8ca3-f891f46fd507-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:04.072248 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:04.072253 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwzmw\" (UniqueName: \"kubernetes.io/projected/ec75741a-2af8-4f2f-8ca3-f891f46fd507-kube-api-access-xwzmw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:04.773282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:04.773254 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" Oct 30 05:11:04.773677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:04.773281 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x" event={"ID":"ec75741a-2af8-4f2f-8ca3-f891f46fd507","Type":"ContainerDied","Data":"e814c691b1a0601195ca7bff9fe07afe58d63d331c536871300454c4f39ea67a"} Oct 30 05:11:04.773677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:04.773336 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e814c691b1a0601195ca7bff9fe07afe58d63d331c536871300454c4f39ea67a" Oct 30 05:11:05.583550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:05.583513 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e16cca-13be-49d8-95a0-22b479aea9bd" path="/var/lib/kubelet/pods/67e16cca-13be-49d8-95a0-22b479aea9bd/volumes" Oct 30 05:11:05.584208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:05.584182 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd420f62-adff-401e-8c12-6bf7eab929e6" path="/var/lib/kubelet/pods/cd420f62-adff-401e-8c12-6bf7eab929e6/volumes" Oct 30 05:11:22.417019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.416983 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx"] Oct 30 05:11:22.417936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.417899 6587 cpu_manager.go:414] "containerMap: removing container" podUID="67e16cca-13be-49d8-95a0-22b479aea9bd" containerName="payload-processing" Oct 30 05:11:22.418002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.417938 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e16cca-13be-49d8-95a0-22b479aea9bd" containerName="payload-processing" Oct 30 05:11:22.418043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.418020 6587 cpu_manager.go:414] "containerMap: removing container" podUID="ec75741a-2af8-4f2f-8ca3-f891f46fd507" containerName="revoke-keys" Oct 30 05:11:22.418043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.418026 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec75741a-2af8-4f2f-8ca3-f891f46fd507" containerName="revoke-keys" Oct 30 05:11:22.418107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.418054 6587 cpu_manager.go:414] "containerMap: removing container" podUID="cd420f62-adff-401e-8c12-6bf7eab929e6" containerName="payload-pre-processing" Oct 30 05:11:22.418107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.418060 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd420f62-adff-401e-8c12-6bf7eab929e6" containerName="payload-pre-processing" Oct 30 05:11:22.418246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.418235 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec75741a-2af8-4f2f-8ca3-f891f46fd507" containerName="revoke-keys" Oct 30 05:11:22.418280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.418253 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd420f62-adff-401e-8c12-6bf7eab929e6" containerName="payload-pre-processing" Oct 30 05:11:22.418280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.418265 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e16cca-13be-49d8-95a0-22b479aea9bd" containerName="payload-processing" Oct 30 05:11:22.423530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.423509 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" Oct 30 05:11:22.443119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.443091 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx"] Oct 30 05:11:22.451936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.451905 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:11:22.452097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.451975 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:11:22.473066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.473033 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:11:22.473247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.473229 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:11:22.522318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.522281 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-288e01-mxmqx\" (UID: \"f99d8290-5005-4241-8d45-8791f5640d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" Oct 30 05:11:22.522596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.522572 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnpx6\" (UniqueName: \"kubernetes.io/projected/f99d8290-5005-4241-8d45-8791f5640d58-kube-api-access-vnpx6\") pod \"maas-api-revoke-keys-e2e-api-a-288e01-mxmqx\" (UID: \"f99d8290-5005-4241-8d45-8791f5640d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" Oct 30 05:11:22.522682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.522649 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-288e01-mxmqx\" (UID: \"f99d8290-5005-4241-8d45-8791f5640d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" Oct 30 05:11:22.618863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.618827 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478"] Oct 30 05:11:22.624545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.624517 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnpx6\" (UniqueName: \"kubernetes.io/projected/f99d8290-5005-4241-8d45-8791f5640d58-kube-api-access-vnpx6\") pod \"maas-api-revoke-keys-e2e-api-a-288e01-mxmqx\" (UID: \"f99d8290-5005-4241-8d45-8791f5640d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" Oct 30 05:11:22.624696 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.624586 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-288e01-mxmqx\" (UID: \"f99d8290-5005-4241-8d45-8791f5640d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" Oct 30 05:11:22.624820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.624799 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-288e01-mxmqx\" (UID: \"f99d8290-5005-4241-8d45-8791f5640d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" Oct 30 05:11:22.624992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.624972 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.625618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.625593 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-288e01-mxmqx\" (UID: \"f99d8290-5005-4241-8d45-8791f5640d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" Oct 30 05:11:22.627202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.627177 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-288e01-mxmqx\" (UID: \"f99d8290-5005-4241-8d45-8791f5640d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" Oct 30 05:11:22.654735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.654707 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478"] Oct 30 05:11:22.655492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.655470 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnpx6\" (UniqueName: \"kubernetes.io/projected/f99d8290-5005-4241-8d45-8791f5640d58-kube-api-access-vnpx6\") pod \"maas-api-revoke-keys-e2e-api-a-288e01-mxmqx\" (UID: \"f99d8290-5005-4241-8d45-8791f5640d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" Oct 30 05:11:22.718137 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.718034 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4225df-openshift-default-dockercfg-n8vvh\"" Oct 30 05:11:22.729833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.729797 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-envoy\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.730060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.729850 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flfjj\" (UniqueName: \"kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-kube-api-access-flfjj\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.730060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.729981 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-credential-socket\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.730159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.730115 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-socket\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.730213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.730162 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-data\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.730262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.730252 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-certs\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.730339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.730307 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-token\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.730714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.730435 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istiod-ca-cert\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.730714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.730589 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-podinfo\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.739208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.739178 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" Oct 30 05:11:22.833336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.832666 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-socket\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.833336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.832733 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-data\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.833336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.832815 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-certs\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.833336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.832873 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-token\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.833336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.832935 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istiod-ca-cert\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.833336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.832990 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-podinfo\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.833336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.833117 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-envoy\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.833336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.833149 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flfjj\" (UniqueName: \"kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-kube-api-access-flfjj\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.833336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.833227 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-credential-socket\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.834192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.834127 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-socket\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.834695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.834674 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-credential-socket\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.835530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.835504 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istiod-ca-cert\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.835860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.835805 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-certs\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.836191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.836118 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-data\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.838506 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.838483 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-envoy\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.840084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.840062 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-podinfo\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.864012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.863978 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flfjj\" (UniqueName: \"kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-kube-api-access-flfjj\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.864162 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.864017 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-token\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:22.961462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:22.961428 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:23.100908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:23.100875 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx"] Oct 30 05:11:23.280457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:23.280430 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478"] Oct 30 05:11:23.319276 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:11:23.319239 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d74c3e2_27d2_4650_97ea_7c99efe8f924.slice/crio-677aee15e62845d3b4135c9727083ef0e199c36a6fab09f8ceee95c2146a7778 WatchSource:0}: Error finding container 677aee15e62845d3b4135c9727083ef0e199c36a6fab09f8ceee95c2146a7778: Status 404 returned error can't find the container with id 677aee15e62845d3b4135c9727083ef0e199c36a6fab09f8ceee95c2146a7778 Oct 30 05:11:23.323565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:23.323525 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:23.323681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:23.323619 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:23.323681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:23.323662 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:23.969148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:23.969116 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" event={"ID":"f99d8290-5005-4241-8d45-8791f5640d58","Type":"ContainerStarted","Data":"cac5beb02c6e492329ffb789441d353859a293faf20002c1e2d96868c3dd0457"} Oct 30 05:11:23.969627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:23.969167 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" event={"ID":"f99d8290-5005-4241-8d45-8791f5640d58","Type":"ContainerStarted","Data":"84fa48a95bdb19ef4e6f64cad528b3bedce4f4f5b0ebcbca46768af4aa822ecf"} Oct 30 05:11:23.970956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:23.970930 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" event={"ID":"7d74c3e2-27d2-4650-97ea-7c99efe8f924","Type":"ContainerStarted","Data":"e4692988d03f1fec873183e3610f3db84c28bb99c45bb826458d22306d3c87fb"} Oct 30 05:11:23.971118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:23.970985 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" event={"ID":"7d74c3e2-27d2-4650-97ea-7c99efe8f924","Type":"ContainerStarted","Data":"677aee15e62845d3b4135c9727083ef0e199c36a6fab09f8ceee95c2146a7778"} Oct 30 05:11:24.013918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:24.013869 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" podStartSLOduration=2.013856018 podStartE2EDuration="2.013856018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:24.011706025 +0000 UTC m=+1608.539472810" watchObservedRunningTime="2026-10-30 05:11:24.013856018 +0000 UTC m=+1608.541622803" Oct 30 05:11:24.072160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:24.072109 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" podStartSLOduration=2.072094811 podStartE2EDuration="2.072094811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:24.069053358 +0000 UTC m=+1608.596820144" watchObservedRunningTime="2026-10-30 05:11:24.072094811 +0000 UTC m=+1608.599861596" Oct 30 05:11:24.965380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:24.964711 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:24.987412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:24.969474 6587 patch_prober.go:29] interesting pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:11:24.987412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:24.969537 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" podUID="7d74c3e2-27d2-4650-97ea-7c99efe8f924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 05:11:25.035381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:25.034755 6587 generic.go:356] "Generic (PLEG): container finished" podID="f99d8290-5005-4241-8d45-8791f5640d58" containerID="cac5beb02c6e492329ffb789441d353859a293faf20002c1e2d96868c3dd0457" exitCode=0 Oct 30 05:11:25.040378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:25.036393 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" event={"ID":"f99d8290-5005-4241-8d45-8791f5640d58","Type":"ContainerDied","Data":"cac5beb02c6e492329ffb789441d353859a293faf20002c1e2d96868c3dd0457"} Oct 30 05:11:25.961922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:25.961881 6587 patch_prober.go:29] interesting pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:11:25.962083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:25.961948 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" podUID="7d74c3e2-27d2-4650-97ea-7c99efe8f924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 05:11:25.987594 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:11:25.987560 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:11:26.304563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:26.304534 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" Oct 30 05:11:26.452930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:26.452889 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-xks-ca\") pod \"f99d8290-5005-4241-8d45-8791f5640d58\" (UID: \"f99d8290-5005-4241-8d45-8791f5640d58\") " Oct 30 05:11:26.453140 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:26.453073 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-openshift-ca\") pod \"f99d8290-5005-4241-8d45-8791f5640d58\" (UID: \"f99d8290-5005-4241-8d45-8791f5640d58\") " Oct 30 05:11:26.453205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:26.453139 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99d8290-5005-4241-8d45-8791f5640d58-kube-api-access-vnpx6\" (UniqueName: \"kubernetes.io/projected/f99d8290-5005-4241-8d45-8791f5640d58-kube-api-access-vnpx6\") pod \"f99d8290-5005-4241-8d45-8791f5640d58\" (UID: \"f99d8290-5005-4241-8d45-8791f5640d58\") " Oct 30 05:11:26.454173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:26.453669 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-openshift-ca" pod "f99d8290-5005-4241-8d45-8791f5640d58" (UID: "f99d8290-5005-4241-8d45-8791f5640d58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:26.454173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:26.454139 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:26.456082 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:26.456039 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-xks-ca" pod "f99d8290-5005-4241-8d45-8791f5640d58" (UID: "f99d8290-5005-4241-8d45-8791f5640d58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:11:26.456411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:26.456339 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99d8290-5005-4241-8d45-8791f5640d58-kube-api-access-vnpx6" pod "f99d8290-5005-4241-8d45-8791f5640d58" (UID: "f99d8290-5005-4241-8d45-8791f5640d58"). InnerVolumeSpecName "kube-api-access-vnpx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:26.556146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:26.556083 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99d8290-5005-4241-8d45-8791f5640d58-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:26.556146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:26.556121 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnpx6\" (UniqueName: \"kubernetes.io/projected/f99d8290-5005-4241-8d45-8791f5640d58-kube-api-access-vnpx6\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:26.962493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:26.962411 6587 patch_prober.go:29] interesting pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:11:26.962493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:26.962466 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" podUID="7d74c3e2-27d2-4650-97ea-7c99efe8f924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 05:11:27.066622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:27.066591 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" Oct 30 05:11:27.067019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:27.066631 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx" event={"ID":"f99d8290-5005-4241-8d45-8791f5640d58","Type":"ContainerDied","Data":"84fa48a95bdb19ef4e6f64cad528b3bedce4f4f5b0ebcbca46768af4aa822ecf"} Oct 30 05:11:27.067019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:27.066691 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84fa48a95bdb19ef4e6f64cad528b3bedce4f4f5b0ebcbca46768af4aa822ecf" Oct 30 05:11:27.962590 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:27.962556 6587 patch_prober.go:29] interesting pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:11:27.962861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:27.962835 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" podUID="7d74c3e2-27d2-4650-97ea-7c99efe8f924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 05:11:28.961967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:28.961934 6587 patch_prober.go:29] interesting pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:11:28.962414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:28.961990 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" podUID="7d74c3e2-27d2-4650-97ea-7c99efe8f924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 05:11:29.964538 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:29.964470 6587 patch_prober.go:29] interesting pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:11:29.965103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:29.965064 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" podUID="7d74c3e2-27d2-4650-97ea-7c99efe8f924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 05:11:30.948086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:30.948049 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc"] Oct 30 05:11:30.949289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:30.949265 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f99d8290-5005-4241-8d45-8791f5640d58" containerName="revoke-keys" Oct 30 05:11:30.949390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:30.949293 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99d8290-5005-4241-8d45-8791f5640d58" containerName="revoke-keys" Oct 30 05:11:30.949548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:30.949533 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99d8290-5005-4241-8d45-8791f5640d58" containerName="revoke-keys" Oct 30 05:11:30.954317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:30.954294 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:30.967409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:30.967343 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:30.968017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:30.967996 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:30.968939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:30.968920 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:11:30.975282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:30.975261 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc"] Oct 30 05:11:30.984212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:30.984180 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ac96f3-openshift-default-dockercfg-ppqp4\"" Oct 30 05:11:31.054037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.053998 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-podinfo\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.054242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.054128 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-data\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.054242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.054226 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istiod-ca-cert\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.054409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.054337 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-certs\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.054530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.054494 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-token\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.054595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.054529 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-socket\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.054595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.054559 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-credential-socket\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.054814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.054793 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5r6s\" (UniqueName: \"kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-kube-api-access-p5r6s\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.054888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.054833 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-envoy\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.156698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.156661 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-podinfo\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.156890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.156728 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-data\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.156890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.156815 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istiod-ca-cert\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.156890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.156868 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-certs\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.157052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.156934 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-token\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.157052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.156959 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-socket\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.157052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.156988 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-credential-socket\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.157239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.157214 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5r6s\" (UniqueName: \"kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-kube-api-access-p5r6s\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.157293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.157235 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-data\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.157293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.157252 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-envoy\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.157425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.157282 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-certs\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.157582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.157560 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-socket\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.157780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.157738 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-credential-socket\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.157925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.157901 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istiod-ca-cert\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.159611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.159587 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-envoy\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.160300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.160283 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-podinfo\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.187188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.187153 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-token\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.188423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.188389 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5r6s\" (UniqueName: \"kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-kube-api-access-p5r6s\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.271821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.271742 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:31.606053 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:11:31.606027 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ede7fd7_21cf_4250_bb20_708d1f2d6240.slice/crio-79bcaca4f61e9706ec297b42d1083c2fddd7149b3f8dc7e5cbc1c50df9c33712 WatchSource:0}: Error finding container 79bcaca4f61e9706ec297b42d1083c2fddd7149b3f8dc7e5cbc1c50df9c33712: Status 404 returned error can't find the container with id 79bcaca4f61e9706ec297b42d1083c2fddd7149b3f8dc7e5cbc1c50df9c33712 Oct 30 05:11:31.607929 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.607905 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc"] Oct 30 05:11:31.608772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.608745 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:31.608859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.608846 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:31.608897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:31.608881 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:32.112298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:32.112256 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" event={"ID":"7ede7fd7-21cf-4250-bb20-708d1f2d6240","Type":"ContainerStarted","Data":"a390aa386c46073ea644e42affaa85db462900a2bbe60f69e52e2c578798d0b4"} Oct 30 05:11:32.112769 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:32.112313 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" event={"ID":"7ede7fd7-21cf-4250-bb20-708d1f2d6240","Type":"ContainerStarted","Data":"79bcaca4f61e9706ec297b42d1083c2fddd7149b3f8dc7e5cbc1c50df9c33712"} Oct 30 05:11:32.190273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:32.190221 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" podStartSLOduration=2.19020813 podStartE2EDuration="2.19020813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:32.176207409 +0000 UTC m=+1616.703974194" watchObservedRunningTime="2026-10-30 05:11:32.19020813 +0000 UTC m=+1616.717974916" Oct 30 05:11:32.272164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:32.272131 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:32.273973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:32.273940 6587 patch_prober.go:29] interesting pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 05:11:32.274135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:32.273996 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" podUID="7ede7fd7-21cf-4250-bb20-708d1f2d6240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 05:11:33.094575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:33.094543 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p"] Oct 30 05:11:33.273226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:33.269419 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk"] Oct 30 05:11:33.273226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:33.269750 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" podUID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" containerName="istio-proxy" containerID="cri-o://5730bf6c99b1a2f77d9b1aea9337073f06a875ee362e71e46d7dcbe02a4ec70c" gracePeriod=30 Oct 30 05:11:33.273226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:33.272458 6587 patch_prober.go:29] interesting pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 05:11:33.273226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:33.272499 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" podUID="7ede7fd7-21cf-4250-bb20-708d1f2d6240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 05:11:33.281043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:33.279032 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92"] Oct 30 05:11:33.281043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:33.279246 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" podUID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" containerName="istio-proxy" containerID="cri-o://65e95848d50902821cafc856de6391e530e1bda36b3a58a6a3f5c4ba3979094b" gracePeriod=30 Oct 30 05:11:33.291136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:33.284578 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s"] Oct 30 05:11:33.291136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:33.284798 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" containerID="cri-o://72f37956350a0373d3bfdad901e799af45c50c818a25e5464ff212d8ba96679d" gracePeriod=30 Oct 30 05:11:34.282389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:34.282030 6587 patch_prober.go:29] interesting pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 05:11:34.282389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:34.282100 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" podUID="7ede7fd7-21cf-4250-bb20-708d1f2d6240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 05:11:35.273173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:35.273136 6587 patch_prober.go:29] interesting pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 05:11:35.273395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:35.273187 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" podUID="7ede7fd7-21cf-4250-bb20-708d1f2d6240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 05:11:36.272963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:36.272921 6587 patch_prober.go:29] interesting pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 05:11:36.273401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:36.272979 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" podUID="7ede7fd7-21cf-4250-bb20-708d1f2d6240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 05:11:37.272918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:37.272880 6587 patch_prober.go:29] interesting pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 05:11:37.273101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:37.272935 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" podUID="7ede7fd7-21cf-4250-bb20-708d1f2d6240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 05:11:38.094840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.094799 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" podUID="0e8198e2-ee22-44d5-9ea8-a9d40e635880" containerName="payload-pre-processing" containerID="cri-o://d755a0688d848cf7683d7f331c4280de6ba9bb67c08eda47db2cf5087451b695" gracePeriod=25 Oct 30 05:11:38.272854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.272812 6587 patch_prober.go:29] interesting pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 05:11:38.273116 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.272880 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" podUID="7ede7fd7-21cf-4250-bb20-708d1f2d6240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 05:11:38.386803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.370726 6587 generic.go:356] "Generic (PLEG): container finished" podID="0e8198e2-ee22-44d5-9ea8-a9d40e635880" containerID="d755a0688d848cf7683d7f331c4280de6ba9bb67c08eda47db2cf5087451b695" exitCode=0 Oct 30 05:11:38.386803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.370834 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" event={"ID":"0e8198e2-ee22-44d5-9ea8-a9d40e635880","Type":"ContainerDied","Data":"d755a0688d848cf7683d7f331c4280de6ba9bb67c08eda47db2cf5087451b695"} Oct 30 05:11:38.577663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.574222 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" Oct 30 05:11:38.627434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.617297 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8198e2-ee22-44d5-9ea8-a9d40e635880-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8198e2-ee22-44d5-9ea8-a9d40e635880-plugins-config-volume\") pod \"0e8198e2-ee22-44d5-9ea8-a9d40e635880\" (UID: \"0e8198e2-ee22-44d5-9ea8-a9d40e635880\") " Oct 30 05:11:38.627434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.617625 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8198e2-ee22-44d5-9ea8-a9d40e635880-kube-api-access-nf8h2\" (UniqueName: \"kubernetes.io/projected/0e8198e2-ee22-44d5-9ea8-a9d40e635880-kube-api-access-nf8h2\") pod \"0e8198e2-ee22-44d5-9ea8-a9d40e635880\" (UID: \"0e8198e2-ee22-44d5-9ea8-a9d40e635880\") " Oct 30 05:11:38.627434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.621066 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8198e2-ee22-44d5-9ea8-a9d40e635880-plugins-config-volume" pod "0e8198e2-ee22-44d5-9ea8-a9d40e635880" (UID: "0e8198e2-ee22-44d5-9ea8-a9d40e635880"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:38.669807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.655957 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8198e2-ee22-44d5-9ea8-a9d40e635880-kube-api-access-nf8h2" pod "0e8198e2-ee22-44d5-9ea8-a9d40e635880" (UID: "0e8198e2-ee22-44d5-9ea8-a9d40e635880"). InnerVolumeSpecName "kube-api-access-nf8h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:38.674909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.674012 6587 scope.go:122] "RemoveContainer" containerID="113268d3e88e0d3d19abb67dcecca8e4bb342c3f6c8ee7e0001ac1a6d03fc81e" Oct 30 05:11:38.709512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.708533 6587 scope.go:122] "RemoveContainer" containerID="8ad9e2837e5049ec536d82d8aa278b53264cc6b8c3d64cd5455b28634cda8d31" Oct 30 05:11:38.731776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.730535 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf8h2\" (UniqueName: \"kubernetes.io/projected/0e8198e2-ee22-44d5-9ea8-a9d40e635880-kube-api-access-nf8h2\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.731776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.730569 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8198e2-ee22-44d5-9ea8-a9d40e635880-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.741857 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.740920 6587 scope.go:122] "RemoveContainer" containerID="8ebbbb5bc08faf2ebde7477e8ba30e77e8837faef0481add17893de0272641ea" Oct 30 05:11:38.759324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.759256 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:11:38.832443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.832419 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-kube-api-access-9mfff\" (UniqueName: \"kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-kube-api-access-9mfff\") pod \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " Oct 30 05:11:38.832688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.832668 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-podinfo\") pod \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " Oct 30 05:11:38.833206 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.833189 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-token\" (UniqueName: \"kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-token\") pod \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " Oct 30 05:11:38.833777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.833760 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-data\") pod \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " Oct 30 05:11:38.834298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.834171 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-data" pod "5ae29f15-fdba-4ab5-8901-7b95e3d936e1" (UID: "5ae29f15-fdba-4ab5-8901-7b95e3d936e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.834743 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.834703 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-credential-socket" pod "5ae29f15-fdba-4ab5-8901-7b95e3d936e1" (UID: "5ae29f15-fdba-4ab5-8901-7b95e3d936e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.834880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.834860 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-credential-socket\") pod \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " Oct 30 05:11:38.835127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.835112 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-envoy\") pod \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " Oct 30 05:11:38.836061 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.835979 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-socket" pod "5ae29f15-fdba-4ab5-8901-7b95e3d936e1" (UID: "5ae29f15-fdba-4ab5-8901-7b95e3d936e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.836169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.836154 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-socket\") pod \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " Oct 30 05:11:38.836316 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.836301 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-certs\") pod \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " Oct 30 05:11:38.836743 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.836555 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-certs" pod "5ae29f15-fdba-4ab5-8901-7b95e3d936e1" (UID: "5ae29f15-fdba-4ab5-8901-7b95e3d936e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.836903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.836887 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istiod-ca-cert\") pod \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\" (UID: \"5ae29f15-fdba-4ab5-8901-7b95e3d936e1\") " Oct 30 05:11:38.837400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.837216 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istiod-ca-cert" pod "5ae29f15-fdba-4ab5-8901-7b95e3d936e1" (UID: "5ae29f15-fdba-4ab5-8901-7b95e3d936e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:38.837950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.837922 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-envoy" pod "5ae29f15-fdba-4ab5-8901-7b95e3d936e1" (UID: "5ae29f15-fdba-4ab5-8901-7b95e3d936e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.838621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.838601 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.838745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.838733 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.838840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.838829 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.838921 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.838910 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.839005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.838993 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.839093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.839082 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.870036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.869976 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-podinfo" pod "5ae29f15-fdba-4ab5-8901-7b95e3d936e1" (UID: "5ae29f15-fdba-4ab5-8901-7b95e3d936e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:11:38.876921 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.876666 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-kube-api-access-9mfff" pod "5ae29f15-fdba-4ab5-8901-7b95e3d936e1" (UID: "5ae29f15-fdba-4ab5-8901-7b95e3d936e1"). InnerVolumeSpecName "kube-api-access-9mfff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:38.876921 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.876870 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-token" pod "5ae29f15-fdba-4ab5-8901-7b95e3d936e1" (UID: "5ae29f15-fdba-4ab5-8901-7b95e3d936e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:38.898402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.898120 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:11:38.913049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.913021 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:11:38.940190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.940157 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-data\") pod \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " Oct 30 05:11:38.940387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.940261 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-envoy\") pod \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " Oct 30 05:11:38.940387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.940338 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-credential-socket\") pod \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " Oct 30 05:11:38.940474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.940440 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-certs\") pod \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " Oct 30 05:11:38.940517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.940491 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-data" pod "e1124d49-ccfe-456e-a0ba-871f0d8d0992" (UID: "e1124d49-ccfe-456e-a0ba-871f0d8d0992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.940517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.940505 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-token\" (UniqueName: \"kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-token\") pod \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " Oct 30 05:11:38.940801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.940657 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-socket\") pod \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " Oct 30 05:11:38.940801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.940714 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istiod-ca-cert\") pod \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " Oct 30 05:11:38.940801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.940728 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-credential-socket" pod "e1124d49-ccfe-456e-a0ba-871f0d8d0992" (UID: "e1124d49-ccfe-456e-a0ba-871f0d8d0992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.940951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.940915 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-kube-api-access-8kdrz\" (UniqueName: \"kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-kube-api-access-8kdrz\") pod \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " Oct 30 05:11:38.941019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.940954 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-podinfo\") pod \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\" (UID: \"e1124d49-ccfe-456e-a0ba-871f0d8d0992\") " Oct 30 05:11:38.941127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.940918 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-socket" pod "e1124d49-ccfe-456e-a0ba-871f0d8d0992" (UID: "e1124d49-ccfe-456e-a0ba-871f0d8d0992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.941233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.941094 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-certs" pod "e1124d49-ccfe-456e-a0ba-871f0d8d0992" (UID: "e1124d49-ccfe-456e-a0ba-871f0d8d0992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:38.941233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.941194 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istiod-ca-cert" pod "e1124d49-ccfe-456e-a0ba-871f0d8d0992" (UID: "e1124d49-ccfe-456e-a0ba-871f0d8d0992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:38.941877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.941853 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.942134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.942016 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.942134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.942041 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.942134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.942057 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.942134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.942071 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mfff\" (UniqueName: \"kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-kube-api-access-9mfff\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.942134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.942084 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.942134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.942100 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.942134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.942113 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae29f15-fdba-4ab5-8901-7b95e3d936e1-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:38.943597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.943559 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-kube-api-access-8kdrz" pod "e1124d49-ccfe-456e-a0ba-871f0d8d0992" (UID: "e1124d49-ccfe-456e-a0ba-871f0d8d0992"). InnerVolumeSpecName "kube-api-access-8kdrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:38.943597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.943582 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-podinfo" pod "e1124d49-ccfe-456e-a0ba-871f0d8d0992" (UID: "e1124d49-ccfe-456e-a0ba-871f0d8d0992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:11:38.943715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.943687 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-token" pod "e1124d49-ccfe-456e-a0ba-871f0d8d0992" (UID: "e1124d49-ccfe-456e-a0ba-871f0d8d0992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:38.944269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:38.944248 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-envoy" pod "e1124d49-ccfe-456e-a0ba-871f0d8d0992" (UID: "e1124d49-ccfe-456e-a0ba-871f0d8d0992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:39.042979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.042904 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-istio-token\" (UniqueName: \"kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-istio-token\") pod \"c59851d6-7c9f-483a-b778-9f78aba34404\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " Oct 30 05:11:39.043154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.043087 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c59851d6-7c9f-483a-b778-9f78aba34404-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59851d6-7c9f-483a-b778-9f78aba34404-istiod-ca-cert\") pod \"c59851d6-7c9f-483a-b778-9f78aba34404\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " Oct 30 05:11:39.043154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.043135 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-envoy\") pod \"c59851d6-7c9f-483a-b778-9f78aba34404\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " Oct 30 05:11:39.043291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.043203 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-credential-socket\") pod \"c59851d6-7c9f-483a-b778-9f78aba34404\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " Oct 30 05:11:39.043291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.043284 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-certs\") pod \"c59851d6-7c9f-483a-b778-9f78aba34404\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " Oct 30 05:11:39.043654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.043335 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c59851d6-7c9f-483a-b778-9f78aba34404-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59851d6-7c9f-483a-b778-9f78aba34404-istio-podinfo\") pod \"c59851d6-7c9f-483a-b778-9f78aba34404\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " Oct 30 05:11:39.043654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.043395 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-kube-api-access-sxrrg\" (UniqueName: \"kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-kube-api-access-sxrrg\") pod \"c59851d6-7c9f-483a-b778-9f78aba34404\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " Oct 30 05:11:39.043654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.043467 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-data\") pod \"c59851d6-7c9f-483a-b778-9f78aba34404\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " Oct 30 05:11:39.046099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.046067 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59851d6-7c9f-483a-b778-9f78aba34404-istiod-ca-cert" pod "c59851d6-7c9f-483a-b778-9f78aba34404" (UID: "c59851d6-7c9f-483a-b778-9f78aba34404"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:39.046212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.046157 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-credential-socket" pod "c59851d6-7c9f-483a-b778-9f78aba34404" (UID: "c59851d6-7c9f-483a-b778-9f78aba34404"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:39.047097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.047007 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-data" pod "c59851d6-7c9f-483a-b778-9f78aba34404" (UID: "c59851d6-7c9f-483a-b778-9f78aba34404"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:39.047244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.047150 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-socket\") pod \"c59851d6-7c9f-483a-b778-9f78aba34404\" (UID: \"c59851d6-7c9f-483a-b778-9f78aba34404\") " Oct 30 05:11:39.048015 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.047991 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:39.048131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.048020 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:39.048131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.048035 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kdrz\" (UniqueName: \"kubernetes.io/projected/e1124d49-ccfe-456e-a0ba-871f0d8d0992-kube-api-access-8kdrz\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:39.048131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.048051 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:39.048131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.048064 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59851d6-7c9f-483a-b778-9f78aba34404-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:39.048131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.048076 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1124d49-ccfe-456e-a0ba-871f0d8d0992-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:39.048131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.048089 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:39.052832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.052786 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-certs" pod "c59851d6-7c9f-483a-b778-9f78aba34404" (UID: "c59851d6-7c9f-483a-b778-9f78aba34404"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:39.053007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.052975 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-socket" pod "c59851d6-7c9f-483a-b778-9f78aba34404" (UID: "c59851d6-7c9f-483a-b778-9f78aba34404"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:39.055814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.055776 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-istio-token" pod "c59851d6-7c9f-483a-b778-9f78aba34404" (UID: "c59851d6-7c9f-483a-b778-9f78aba34404"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:39.057416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.057395 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c59851d6-7c9f-483a-b778-9f78aba34404-istio-podinfo" pod "c59851d6-7c9f-483a-b778-9f78aba34404" (UID: "c59851d6-7c9f-483a-b778-9f78aba34404"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:11:39.057534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.057439 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-kube-api-access-sxrrg" pod "c59851d6-7c9f-483a-b778-9f78aba34404" (UID: "c59851d6-7c9f-483a-b778-9f78aba34404"). InnerVolumeSpecName "kube-api-access-sxrrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:39.058914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.058889 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-envoy" pod "c59851d6-7c9f-483a-b778-9f78aba34404" (UID: "c59851d6-7c9f-483a-b778-9f78aba34404"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:39.149473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.149384 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:39.149473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.149413 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:39.149473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.149426 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59851d6-7c9f-483a-b778-9f78aba34404-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:39.149473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.149438 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxrrg\" (UniqueName: \"kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-kube-api-access-sxrrg\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:39.149473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.149451 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59851d6-7c9f-483a-b778-9f78aba34404-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:39.149473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.149467 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59851d6-7c9f-483a-b778-9f78aba34404-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:39.276975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.276947 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:39.277395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.277383 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:39.278036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.278017 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:11:39.387921 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.387286 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" Oct 30 05:11:39.387921 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.387792 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p" event={"ID":"0e8198e2-ee22-44d5-9ea8-a9d40e635880","Type":"ContainerDied","Data":"48ffc8f95337ef56542a06fee8f5268c03d408940c9a9070f938b29959789ba2"} Oct 30 05:11:39.387921 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.387853 6587 scope.go:122] "RemoveContainer" containerID="d755a0688d848cf7683d7f331c4280de6ba9bb67c08eda47db2cf5087451b695" Oct 30 05:11:39.389828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.389797 6587 generic.go:356] "Generic (PLEG): container finished" podID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" containerID="65e95848d50902821cafc856de6391e530e1bda36b3a58a6a3f5c4ba3979094b" exitCode=0 Oct 30 05:11:39.389952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.389862 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" Oct 30 05:11:39.389952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.389915 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" event={"ID":"e1124d49-ccfe-456e-a0ba-871f0d8d0992","Type":"ContainerDied","Data":"65e95848d50902821cafc856de6391e530e1bda36b3a58a6a3f5c4ba3979094b"} Oct 30 05:11:39.390094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.389957 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92" event={"ID":"e1124d49-ccfe-456e-a0ba-871f0d8d0992","Type":"ContainerDied","Data":"fc2743f3b23c75fc0254b8af9c2489ba8a54feace73fab21441fb76a8db2a16e"} Oct 30 05:11:39.392069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.392045 6587 generic.go:356] "Generic (PLEG): container finished" podID="c59851d6-7c9f-483a-b778-9f78aba34404" containerID="72f37956350a0373d3bfdad901e799af45c50c818a25e5464ff212d8ba96679d" exitCode=0 Oct 30 05:11:39.392202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.392189 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" event={"ID":"c59851d6-7c9f-483a-b778-9f78aba34404","Type":"ContainerDied","Data":"72f37956350a0373d3bfdad901e799af45c50c818a25e5464ff212d8ba96679d"} Oct 30 05:11:39.392264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.392222 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" event={"ID":"c59851d6-7c9f-483a-b778-9f78aba34404","Type":"ContainerDied","Data":"9961c9d606cbf6ca753176a58fb7bbf7de6365607c18ccbd3fce892d2e242f6c"} Oct 30 05:11:39.392319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.392277 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s" Oct 30 05:11:39.402750 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.402566 6587 generic.go:356] "Generic (PLEG): container finished" podID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" containerID="5730bf6c99b1a2f77d9b1aea9337073f06a875ee362e71e46d7dcbe02a4ec70c" exitCode=0 Oct 30 05:11:39.404195 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.404173 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" Oct 30 05:11:39.408268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.406782 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" event={"ID":"5ae29f15-fdba-4ab5-8901-7b95e3d936e1","Type":"ContainerDied","Data":"5730bf6c99b1a2f77d9b1aea9337073f06a875ee362e71e46d7dcbe02a4ec70c"} Oct 30 05:11:39.408268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.406885 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk" event={"ID":"5ae29f15-fdba-4ab5-8901-7b95e3d936e1","Type":"ContainerDied","Data":"81cb0d331c418851a760925f5f5b28a9f500632d64a4a7a594b9b92828966df6"} Oct 30 05:11:39.408268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.408138 6587 scope.go:122] "RemoveContainer" containerID="65e95848d50902821cafc856de6391e530e1bda36b3a58a6a3f5c4ba3979094b" Oct 30 05:11:39.470245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.470221 6587 scope.go:122] "RemoveContainer" containerID="65e95848d50902821cafc856de6391e530e1bda36b3a58a6a3f5c4ba3979094b" Oct 30 05:11:39.471425 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:11:39.471271 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e95848d50902821cafc856de6391e530e1bda36b3a58a6a3f5c4ba3979094b\": container with ID starting with 65e95848d50902821cafc856de6391e530e1bda36b3a58a6a3f5c4ba3979094b not found: ID does not exist" containerID="65e95848d50902821cafc856de6391e530e1bda36b3a58a6a3f5c4ba3979094b" Oct 30 05:11:39.471425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.471390 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e95848d50902821cafc856de6391e530e1bda36b3a58a6a3f5c4ba3979094b"} err="failed to get container status \"65e95848d50902821cafc856de6391e530e1bda36b3a58a6a3f5c4ba3979094b\": rpc error: code = NotFound desc = could not find container \"65e95848d50902821cafc856de6391e530e1bda36b3a58a6a3f5c4ba3979094b\": container with ID starting with 65e95848d50902821cafc856de6391e530e1bda36b3a58a6a3f5c4ba3979094b not found: ID does not exist" Oct 30 05:11:39.471425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.471411 6587 scope.go:122] "RemoveContainer" containerID="72f37956350a0373d3bfdad901e799af45c50c818a25e5464ff212d8ba96679d" Oct 30 05:11:39.495127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.495101 6587 scope.go:122] "RemoveContainer" containerID="b28fa3a0544653a02386e6c3f05a5bbe807c13d1c69a7641da2bbb5fe9162c25" Oct 30 05:11:39.507400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.507338 6587 scope.go:122] "RemoveContainer" containerID="72f37956350a0373d3bfdad901e799af45c50c818a25e5464ff212d8ba96679d" Oct 30 05:11:39.507718 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:11:39.507695 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f37956350a0373d3bfdad901e799af45c50c818a25e5464ff212d8ba96679d\": container with ID starting with 72f37956350a0373d3bfdad901e799af45c50c818a25e5464ff212d8ba96679d not found: ID does not exist" containerID="72f37956350a0373d3bfdad901e799af45c50c818a25e5464ff212d8ba96679d" Oct 30 05:11:39.507773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.507729 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f37956350a0373d3bfdad901e799af45c50c818a25e5464ff212d8ba96679d"} err="failed to get container status \"72f37956350a0373d3bfdad901e799af45c50c818a25e5464ff212d8ba96679d\": rpc error: code = NotFound desc = could not find container \"72f37956350a0373d3bfdad901e799af45c50c818a25e5464ff212d8ba96679d\": container with ID starting with 72f37956350a0373d3bfdad901e799af45c50c818a25e5464ff212d8ba96679d not found: ID does not exist" Oct 30 05:11:39.507773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.507746 6587 scope.go:122] "RemoveContainer" containerID="b28fa3a0544653a02386e6c3f05a5bbe807c13d1c69a7641da2bbb5fe9162c25" Oct 30 05:11:39.508051 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:11:39.508031 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28fa3a0544653a02386e6c3f05a5bbe807c13d1c69a7641da2bbb5fe9162c25\": container with ID starting with b28fa3a0544653a02386e6c3f05a5bbe807c13d1c69a7641da2bbb5fe9162c25 not found: ID does not exist" containerID="b28fa3a0544653a02386e6c3f05a5bbe807c13d1c69a7641da2bbb5fe9162c25" Oct 30 05:11:39.508124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.508061 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28fa3a0544653a02386e6c3f05a5bbe807c13d1c69a7641da2bbb5fe9162c25"} err="failed to get container status \"b28fa3a0544653a02386e6c3f05a5bbe807c13d1c69a7641da2bbb5fe9162c25\": rpc error: code = NotFound desc = could not find container \"b28fa3a0544653a02386e6c3f05a5bbe807c13d1c69a7641da2bbb5fe9162c25\": container with ID starting with b28fa3a0544653a02386e6c3f05a5bbe807c13d1c69a7641da2bbb5fe9162c25 not found: ID does not exist" Oct 30 05:11:39.508124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.508084 6587 scope.go:122] "RemoveContainer" containerID="5730bf6c99b1a2f77d9b1aea9337073f06a875ee362e71e46d7dcbe02a4ec70c" Oct 30 05:11:39.512397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.512340 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92"] Oct 30 05:11:39.522798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.522774 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92"] Oct 30 05:11:39.523910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.523892 6587 scope.go:122] "RemoveContainer" containerID="5730bf6c99b1a2f77d9b1aea9337073f06a875ee362e71e46d7dcbe02a4ec70c" Oct 30 05:11:39.524342 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:11:39.524315 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5730bf6c99b1a2f77d9b1aea9337073f06a875ee362e71e46d7dcbe02a4ec70c\": container with ID starting with 5730bf6c99b1a2f77d9b1aea9337073f06a875ee362e71e46d7dcbe02a4ec70c not found: ID does not exist" containerID="5730bf6c99b1a2f77d9b1aea9337073f06a875ee362e71e46d7dcbe02a4ec70c" Oct 30 05:11:39.524475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.524350 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5730bf6c99b1a2f77d9b1aea9337073f06a875ee362e71e46d7dcbe02a4ec70c"} err="failed to get container status \"5730bf6c99b1a2f77d9b1aea9337073f06a875ee362e71e46d7dcbe02a4ec70c\": rpc error: code = NotFound desc = could not find container \"5730bf6c99b1a2f77d9b1aea9337073f06a875ee362e71e46d7dcbe02a4ec70c\": container with ID starting with 5730bf6c99b1a2f77d9b1aea9337073f06a875ee362e71e46d7dcbe02a4ec70c not found: ID does not exist" Oct 30 05:11:39.556455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.556432 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk"] Oct 30 05:11:39.562588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.562559 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk"] Oct 30 05:11:39.583578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.583555 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" path="/var/lib/kubelet/pods/5ae29f15-fdba-4ab5-8901-7b95e3d936e1/volumes" Oct 30 05:11:39.584084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.584069 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" path="/var/lib/kubelet/pods/e1124d49-ccfe-456e-a0ba-871f0d8d0992/volumes" Oct 30 05:11:39.594917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.594894 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s"] Oct 30 05:11:39.598736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.598716 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s"] Oct 30 05:11:39.634880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.634849 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p"] Oct 30 05:11:39.641248 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:39.641219 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p"] Oct 30 05:11:40.001051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.001018 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld"] Oct 30 05:11:40.001792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.001774 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" Oct 30 05:11:40.001847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.001793 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" Oct 30 05:11:40.001847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.001842 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0e8198e2-ee22-44d5-9ea8-a9d40e635880" containerName="payload-pre-processing" Oct 30 05:11:40.001927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.001847 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8198e2-ee22-44d5-9ea8-a9d40e635880" containerName="payload-pre-processing" Oct 30 05:11:40.001927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.001863 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" containerName="istio-proxy" Oct 30 05:11:40.001927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.001868 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" containerName="istio-proxy" Oct 30 05:11:40.001927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.001881 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" containerName="istio-proxy" Oct 30 05:11:40.001927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.001886 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" containerName="istio-proxy" Oct 30 05:11:40.002166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.002017 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" Oct 30 05:11:40.002166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.002033 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8198e2-ee22-44d5-9ea8-a9d40e635880" containerName="payload-pre-processing" Oct 30 05:11:40.002166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.002042 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae29f15-fdba-4ab5-8901-7b95e3d936e1" containerName="istio-proxy" Oct 30 05:11:40.002166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.002049 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1124d49-ccfe-456e-a0ba-871f0d8d0992" containerName="istio-proxy" Oct 30 05:11:40.008014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.007993 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" Oct 30 05:11:40.025777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.025755 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:11:40.026452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.026277 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:11:40.026452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.026334 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:11:40.026452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.026381 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:11:40.027616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.027592 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld"] Oct 30 05:11:40.072912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.069927 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0a27f4-4rqld\" (UID: \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" Oct 30 05:11:40.072912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.069996 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0a27f4-4rqld\" (UID: \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" Oct 30 05:11:40.072912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.070317 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cmkl\" (UniqueName: \"kubernetes.io/projected/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-kube-api-access-6cmkl\") pod \"maas-api-revoke-sub-cd0a27f4-4rqld\" (UID: \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" Oct 30 05:11:40.294753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.286501 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cmkl\" (UniqueName: \"kubernetes.io/projected/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-kube-api-access-6cmkl\") pod \"maas-api-revoke-sub-cd0a27f4-4rqld\" (UID: \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" Oct 30 05:11:40.325380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.305422 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0a27f4-4rqld\" (UID: \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" Oct 30 05:11:40.325380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.305476 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0a27f4-4rqld\" (UID: \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" Oct 30 05:11:40.325380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.316911 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0a27f4-4rqld\" (UID: \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" Oct 30 05:11:40.358694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.358607 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cmkl\" (UniqueName: \"kubernetes.io/projected/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-kube-api-access-6cmkl\") pod \"maas-api-revoke-sub-cd0a27f4-4rqld\" (UID: \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" Oct 30 05:11:40.366629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.365699 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0a27f4-4rqld\" (UID: \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" Oct 30 05:11:40.660385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:40.652638 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" Oct 30 05:11:41.097726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:41.097697 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld"] Oct 30 05:11:41.584320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:41.584284 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8198e2-ee22-44d5-9ea8-a9d40e635880" path="/var/lib/kubelet/pods/0e8198e2-ee22-44d5-9ea8-a9d40e635880/volumes" Oct 30 05:11:41.585042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:41.585020 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" path="/var/lib/kubelet/pods/c59851d6-7c9f-483a-b778-9f78aba34404/volumes" Oct 30 05:11:41.667714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:41.667665 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" event={"ID":"faa45334-b5d2-4cd7-ad85-91ff4f8953c5","Type":"ContainerStarted","Data":"159ea297208fb3da5dc19f45a4ed487881a05560cc5a360d026a6c687e57bbf5"} Oct 30 05:11:41.667863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:41.667736 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" event={"ID":"faa45334-b5d2-4cd7-ad85-91ff4f8953c5","Type":"ContainerStarted","Data":"1abddf338c2e9ca490c35f63286e4ff86d63a34883efc1b4cbe51400a3ae8c0e"} Oct 30 05:11:41.714203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:41.714109 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" podStartSLOduration=2.714094256 podStartE2EDuration="2.714094256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:41.712094493 +0000 UTC m=+1626.239861280" watchObservedRunningTime="2026-10-30 05:11:41.714094256 +0000 UTC m=+1626.241861042" Oct 30 05:11:42.684110 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:42.684010 6587 generic.go:356] "Generic (PLEG): container finished" podID="faa45334-b5d2-4cd7-ad85-91ff4f8953c5" containerID="159ea297208fb3da5dc19f45a4ed487881a05560cc5a360d026a6c687e57bbf5" exitCode=0 Oct 30 05:11:42.684584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:42.684095 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" event={"ID":"faa45334-b5d2-4cd7-ad85-91ff4f8953c5","Type":"ContainerDied","Data":"159ea297208fb3da5dc19f45a4ed487881a05560cc5a360d026a6c687e57bbf5"} Oct 30 05:11:43.859717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:43.859688 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" Oct 30 05:11:43.936486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:43.936453 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-openshift-ca\") pod \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\" (UID: \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\") " Oct 30 05:11:43.936695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:43.936583 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-kube-api-access-6cmkl\" (UniqueName: \"kubernetes.io/projected/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-kube-api-access-6cmkl\") pod \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\" (UID: \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\") " Oct 30 05:11:43.936695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:43.936621 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-xks-ca\") pod \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\" (UID: \"faa45334-b5d2-4cd7-ad85-91ff4f8953c5\") " Oct 30 05:11:43.936902 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:43.936879 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-openshift-ca" pod "faa45334-b5d2-4cd7-ad85-91ff4f8953c5" (UID: "faa45334-b5d2-4cd7-ad85-91ff4f8953c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:43.937567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:43.937540 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:43.939414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:43.939375 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-kube-api-access-6cmkl" pod "faa45334-b5d2-4cd7-ad85-91ff4f8953c5" (UID: "faa45334-b5d2-4cd7-ad85-91ff4f8953c5"). InnerVolumeSpecName "kube-api-access-6cmkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:43.939771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:43.939745 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-xks-ca" pod "faa45334-b5d2-4cd7-ad85-91ff4f8953c5" (UID: "faa45334-b5d2-4cd7-ad85-91ff4f8953c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:11:44.038964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:44.038926 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cmkl\" (UniqueName: \"kubernetes.io/projected/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-kube-api-access-6cmkl\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:44.038964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:44.038964 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa45334-b5d2-4cd7-ad85-91ff4f8953c5-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:11:44.701493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:44.701462 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" event={"ID":"faa45334-b5d2-4cd7-ad85-91ff4f8953c5","Type":"ContainerDied","Data":"1abddf338c2e9ca490c35f63286e4ff86d63a34883efc1b4cbe51400a3ae8c0e"} Oct 30 05:11:44.701657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:44.701497 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld" Oct 30 05:11:44.701657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:44.701511 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1abddf338c2e9ca490c35f63286e4ff86d63a34883efc1b4cbe51400a3ae8c0e" Oct 30 05:11:46.689255 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:46.689162 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs"] Oct 30 05:11:46.690154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:46.690129 6587 cpu_manager.go:414] "containerMap: removing container" podUID="faa45334-b5d2-4cd7-ad85-91ff4f8953c5" containerName="revoke-keys" Oct 30 05:11:46.690154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:46.690155 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa45334-b5d2-4cd7-ad85-91ff4f8953c5" containerName="revoke-keys" Oct 30 05:11:46.690320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:46.690234 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" Oct 30 05:11:46.690320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:46.690242 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" Oct 30 05:11:46.690513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:46.690496 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59851d6-7c9f-483a-b778-9f78aba34404" containerName="istio-proxy" Oct 30 05:11:46.690574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:46.690520 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa45334-b5d2-4cd7-ad85-91ff4f8953c5" containerName="revoke-keys" Oct 30 05:11:46.694653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:46.694631 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" Oct 30 05:11:46.722740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:46.722705 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs"] Oct 30 05:11:46.728856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:46.728803 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c7f962\"" Oct 30 05:11:46.729107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:46.728812 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c7f962-dockercfg-cdl5p\"" Oct 30 05:11:46.896411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:46.894901 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4012f48b-8a86-4077-9dd5-c6886b47a38d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs\" (UID: \"4012f48b-8a86-4077-9dd5-c6886b47a38d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" Oct 30 05:11:46.896411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:46.895281 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9667w\" (UniqueName: \"kubernetes.io/projected/4012f48b-8a86-4077-9dd5-c6886b47a38d-kube-api-access-9667w\") pod \"payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs\" (UID: \"4012f48b-8a86-4077-9dd5-c6886b47a38d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" Oct 30 05:11:47.011473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:47.010421 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9667w\" (UniqueName: \"kubernetes.io/projected/4012f48b-8a86-4077-9dd5-c6886b47a38d-kube-api-access-9667w\") pod \"payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs\" (UID: \"4012f48b-8a86-4077-9dd5-c6886b47a38d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" Oct 30 05:11:47.011473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:47.010647 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4012f48b-8a86-4077-9dd5-c6886b47a38d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs\" (UID: \"4012f48b-8a86-4077-9dd5-c6886b47a38d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" Oct 30 05:11:47.016483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:47.011906 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4012f48b-8a86-4077-9dd5-c6886b47a38d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs\" (UID: \"4012f48b-8a86-4077-9dd5-c6886b47a38d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" Oct 30 05:11:47.063110 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:47.063075 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9667w\" (UniqueName: \"kubernetes.io/projected/4012f48b-8a86-4077-9dd5-c6886b47a38d-kube-api-access-9667w\") pod \"payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs\" (UID: \"4012f48b-8a86-4077-9dd5-c6886b47a38d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" Oct 30 05:11:47.310110 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:47.310079 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" Oct 30 05:11:47.621769 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:11:47.621738 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4012f48b_8a86_4077_9dd5_c6886b47a38d.slice/crio-e594cb2930ae5aa88ac2156ac09d8c8b7f6768e71e053056a5cced5b8d83ad0e WatchSource:0}: Error finding container e594cb2930ae5aa88ac2156ac09d8c8b7f6768e71e053056a5cced5b8d83ad0e: Status 404 returned error can't find the container with id e594cb2930ae5aa88ac2156ac09d8c8b7f6768e71e053056a5cced5b8d83ad0e Oct 30 05:11:47.623123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:47.622838 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs"] Oct 30 05:11:47.731867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:47.731836 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" event={"ID":"4012f48b-8a86-4077-9dd5-c6886b47a38d","Type":"ContainerStarted","Data":"c90512f68e5aa1ddf9db108c4e8307d3394487287bf859f6fcd679823bcfc21c"} Oct 30 05:11:47.732307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:47.731889 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" event={"ID":"4012f48b-8a86-4077-9dd5-c6886b47a38d","Type":"ContainerStarted","Data":"e594cb2930ae5aa88ac2156ac09d8c8b7f6768e71e053056a5cced5b8d83ad0e"} Oct 30 05:11:47.732307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:47.731975 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" Oct 30 05:11:47.780968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:47.780919 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" podStartSLOduration=1.780905704 podStartE2EDuration="1.780905704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:47.776828888 +0000 UTC m=+1632.304595675" watchObservedRunningTime="2026-10-30 05:11:47.780905704 +0000 UTC m=+1632.308672489" Oct 30 05:11:51.176185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.176147 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq"] Oct 30 05:11:51.182196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.182166 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.217992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.217960 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq"] Oct 30 05:11:51.223410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.221691 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f8eb4-openshift-default-dockercfg-kktfc\"" Oct 30 05:11:51.292195 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.292159 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-envoy\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.292392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.292275 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-certs\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.292392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.292340 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.292473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.292433 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-credential-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.292473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.292458 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-token\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.292573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.292556 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwszq\" (UniqueName: \"kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-kube-api-access-nwszq\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.292641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.292630 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-podinfo\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.292719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.292707 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istiod-ca-cert\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.292755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.292739 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-data\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.399382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.399095 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-podinfo\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.399382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.399192 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istiod-ca-cert\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.399382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.399230 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-data\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.399686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.399413 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-envoy\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.399686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.399514 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-certs\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.399686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.399549 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.399686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.399601 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-credential-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.399686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.399633 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-token\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.399946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.399697 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwszq\" (UniqueName: \"kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-kube-api-access-nwszq\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.434469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.420632 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-certs\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.434469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.421320 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istiod-ca-cert\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.434469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.421594 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-data\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.434469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.426899 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-credential-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.434469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.427151 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.438608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.437754 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-envoy\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.452991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.452957 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-token\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.466747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.454310 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-podinfo\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.478735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.478609 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwszq\" (UniqueName: \"kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-kube-api-access-nwszq\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.512951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.508613 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:51.948499 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:11:51.948465 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fb6f3a_1091_4baa_be60_aab470fa8e84.slice/crio-52c2bf3f2c7ddbb6c2b2df10823894cfcb38abaafd06527c81a57ee5ecd164c2 WatchSource:0}: Error finding container 52c2bf3f2c7ddbb6c2b2df10823894cfcb38abaafd06527c81a57ee5ecd164c2: Status 404 returned error can't find the container with id 52c2bf3f2c7ddbb6c2b2df10823894cfcb38abaafd06527c81a57ee5ecd164c2 Oct 30 05:11:51.951423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.951396 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq"] Oct 30 05:11:51.951704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.951676 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:51.951798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.951739 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:51.951798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:51.951768 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:52.786537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:52.786499 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" event={"ID":"a2fb6f3a-1091-4baa-be60-aab470fa8e84","Type":"ContainerStarted","Data":"b34cc9f6e511e072b08ead79fd4c498b10b19a27db5226970364e2e6e7d8af55"} Oct 30 05:11:52.787068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:52.786568 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" event={"ID":"a2fb6f3a-1091-4baa-be60-aab470fa8e84","Type":"ContainerStarted","Data":"52c2bf3f2c7ddbb6c2b2df10823894cfcb38abaafd06527c81a57ee5ecd164c2"} Oct 30 05:11:52.860494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:52.860440 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" podStartSLOduration=1.860423993 podStartE2EDuration="1.860423993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:52.845002508 +0000 UTC m=+1637.372769295" watchObservedRunningTime="2026-10-30 05:11:52.860423993 +0000 UTC m=+1637.388190779" Oct 30 05:11:53.509340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:53.509308 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:53.515678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:53.515651 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:53.795964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:53.795886 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:53.796778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:53.796755 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:11:55.199626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.199588 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk"] Oct 30 05:11:55.205892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.205867 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.246380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.243513 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c3486b-openshift-default-dockercfg-4vhlb\"" Oct 30 05:11:55.255424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.251290 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk"] Oct 30 05:11:55.289197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.288321 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-certs\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.289197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.288600 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-data\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.289197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.288700 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-token\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.289197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.288778 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-credential-socket\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.289197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.288870 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-socket\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.289197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.289050 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bqhk\" (UniqueName: \"kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-kube-api-access-7bqhk\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.289705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.289280 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-envoy\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.289705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.289351 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istiod-ca-cert\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.289705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.289580 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-podinfo\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.452495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.406750 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bqhk\" (UniqueName: \"kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-kube-api-access-7bqhk\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.452495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.406882 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-envoy\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.452495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.406915 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istiod-ca-cert\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.452495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.406998 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-podinfo\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.452495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.407162 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-certs\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.452495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.407313 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-data\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.452495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.407379 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-token\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.452495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.407423 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-credential-socket\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.452495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.407470 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-socket\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.452495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.408018 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-socket\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.522425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.488280 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-certs\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.522425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.488946 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istiod-ca-cert\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.522425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.490194 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-data\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.522425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.506967 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-credential-socket\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.576450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.545461 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-envoy\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.595767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.584253 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bqhk\" (UniqueName: \"kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-kube-api-access-7bqhk\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.622108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.621202 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-podinfo\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.626160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.625978 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-token\") pod \"e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.832816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:55.832784 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:55.998210 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:11:55.998176 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:11:56.178179 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:11:56.178145 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e4e00a_ee98_4cf5_a016_cbf70647fea9.slice/crio-c543fb4fa9f74c5880b543949a646b54a2b766f82b77877a53b46a33c00ba81c WatchSource:0}: Error finding container c543fb4fa9f74c5880b543949a646b54a2b766f82b77877a53b46a33c00ba81c: Status 404 returned error can't find the container with id c543fb4fa9f74c5880b543949a646b54a2b766f82b77877a53b46a33c00ba81c Oct 30 05:11:56.179458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:56.179428 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk"] Oct 30 05:11:56.181177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:56.181148 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:56.181264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:56.181211 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:56.181264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:56.181240 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:11:56.824108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:56.824070 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" event={"ID":"e5e4e00a-ee98-4cf5-a016-cbf70647fea9","Type":"ContainerStarted","Data":"8c851debf1dd9da389a6ab50742afa32f1556ac8fabe2b89a39b5d92266a3d52"} Oct 30 05:11:56.824648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:56.824135 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" event={"ID":"e5e4e00a-ee98-4cf5-a016-cbf70647fea9","Type":"ContainerStarted","Data":"c543fb4fa9f74c5880b543949a646b54a2b766f82b77877a53b46a33c00ba81c"} Oct 30 05:11:56.833464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:56.833437 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:56.892199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:56.892148 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" podStartSLOduration=1.8921339449999999 podStartE2EDuration="1.892133945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:56.888401138 +0000 UTC m=+1641.416167921" watchObservedRunningTime="2026-10-30 05:11:56.892133945 +0000 UTC m=+1641.419900730" Oct 30 05:11:57.753436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:57.747992 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq"] Oct 30 05:11:57.760024 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:57.759796 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq"] Oct 30 05:11:57.760024 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:57.759931 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" Oct 30 05:11:57.782852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:57.781649 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1f8eb4-dockercfg-68lmh\"" Oct 30 05:11:57.791166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:57.791133 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1f8eb4\"" Oct 30 05:11:57.854827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:57.854737 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:57.855731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:57.855380 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gtnr\" (UniqueName: \"kubernetes.io/projected/901c74b9-8384-451e-ab88-ad7f7069c48e-kube-api-access-6gtnr\") pod \"payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq\" (UID: \"901c74b9-8384-451e-ab88-ad7f7069c48e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" Oct 30 05:11:57.855864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:57.855823 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901c74b9-8384-451e-ab88-ad7f7069c48e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq\" (UID: \"901c74b9-8384-451e-ab88-ad7f7069c48e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" Oct 30 05:11:57.957063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:57.957034 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901c74b9-8384-451e-ab88-ad7f7069c48e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq\" (UID: \"901c74b9-8384-451e-ab88-ad7f7069c48e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" Oct 30 05:11:57.957674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:57.957643 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gtnr\" (UniqueName: \"kubernetes.io/projected/901c74b9-8384-451e-ab88-ad7f7069c48e-kube-api-access-6gtnr\") pod \"payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq\" (UID: \"901c74b9-8384-451e-ab88-ad7f7069c48e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" Oct 30 05:11:57.958040 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:57.958013 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901c74b9-8384-451e-ab88-ad7f7069c48e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq\" (UID: \"901c74b9-8384-451e-ab88-ad7f7069c48e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" Oct 30 05:11:57.982631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:57.982600 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gtnr\" (UniqueName: \"kubernetes.io/projected/901c74b9-8384-451e-ab88-ad7f7069c48e-kube-api-access-6gtnr\") pod \"payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq\" (UID: \"901c74b9-8384-451e-ab88-ad7f7069c48e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" Oct 30 05:11:58.095895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:58.095863 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" Oct 30 05:11:58.403077 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:11:58.403043 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901c74b9_8384_451e_ab88_ad7f7069c48e.slice/crio-8d5db8d34726ca2b19a27927e8240fdc01d2ccad3bafd2e09328d9d283881b6b WatchSource:0}: Error finding container 8d5db8d34726ca2b19a27927e8240fdc01d2ccad3bafd2e09328d9d283881b6b: Status 404 returned error can't find the container with id 8d5db8d34726ca2b19a27927e8240fdc01d2ccad3bafd2e09328d9d283881b6b Oct 30 05:11:58.405893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:58.405867 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq"] Oct 30 05:11:58.743002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:58.742916 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" Oct 30 05:11:58.857915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:58.857874 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" event={"ID":"901c74b9-8384-451e-ab88-ad7f7069c48e","Type":"ContainerStarted","Data":"a77d0357ce64ef75cf0234c2152e24604b86709dad459cb565a36ee5ff410bdc"} Oct 30 05:11:58.859226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:58.857936 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" event={"ID":"901c74b9-8384-451e-ab88-ad7f7069c48e","Type":"ContainerStarted","Data":"8d5db8d34726ca2b19a27927e8240fdc01d2ccad3bafd2e09328d9d283881b6b"} Oct 30 05:11:58.859226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:58.857965 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:58.859226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:58.858004 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" Oct 30 05:11:58.859479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:58.859230 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:11:58.908634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:11:58.908576 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" podStartSLOduration=1.908561714 podStartE2EDuration="1.908561714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:58.905495005 +0000 UTC m=+1643.433261793" watchObservedRunningTime="2026-10-30 05:11:58.908561714 +0000 UTC m=+1643.436328513" Oct 30 05:12:10.878321 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:10.878288 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" Oct 30 05:12:19.281335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:19.281301 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x"] Oct 30 05:12:19.294084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:19.294056 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-543d08-qhr2x"] Oct 30 05:12:19.583764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:19.583731 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec75741a-2af8-4f2f-8ca3-f891f46fd507" path="/var/lib/kubelet/pods/ec75741a-2af8-4f2f-8ca3-f891f46fd507/volumes" Oct 30 05:12:22.119190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:22.119123 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt"] Oct 30 05:12:22.140222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:22.140184 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-645aef-6xfmt"] Oct 30 05:12:23.583189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:23.583158 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68fa70ec-593f-49ff-be9d-e5159c2e1f61" path="/var/lib/kubelet/pods/68fa70ec-593f-49ff-be9d-e5159c2e1f61/volumes" Oct 30 05:12:24.817834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:24.817801 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl"] Oct 30 05:12:24.818406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:24.818024 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" podUID="7d9f1cd1-5271-4630-b240-27b4cf22b9a8" containerName="istio-proxy" containerID="cri-o://ded0e003c4ae947b4f371f58fb875a6980f402d6178c554c114531c735a5d8fb" gracePeriod=30 Oct 30 05:12:25.997642 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:12:25.997584 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:12:29.620223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:29.620183 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx"] Oct 30 05:12:29.620809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:29.620467 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" containerID="cri-o://f1910e6f4fcdf6a8707221cabf8106086038eee6e283e52afa5b73b05cea2b4f" gracePeriod=30 Oct 30 05:12:30.295113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.295075 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 05:12:30.357289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.356885 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-envoy\") pod \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " Oct 30 05:12:30.357289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.356958 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-credential-socket\") pod \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " Oct 30 05:12:30.357289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.357027 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-podinfo\") pod \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " Oct 30 05:12:30.357289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.357119 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-token\" (UniqueName: \"kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-token\") pod \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " Oct 30 05:12:30.357289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.357160 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-socket\") pod \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " Oct 30 05:12:30.357909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.357353 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istiod-ca-cert\") pod \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " Oct 30 05:12:30.380916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.379779 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istiod-ca-cert" pod "7d9f1cd1-5271-4630-b240-27b4cf22b9a8" (UID: "7d9f1cd1-5271-4630-b240-27b4cf22b9a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:30.392004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.381573 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-kube-api-access-hxrm4\" (UniqueName: \"kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-kube-api-access-hxrm4\") pod \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " Oct 30 05:12:30.392004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.381650 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-certs\") pod \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " Oct 30 05:12:30.392004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.381805 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-data\") pod \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\" (UID: \"7d9f1cd1-5271-4630-b240-27b4cf22b9a8\") " Oct 30 05:12:30.392004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.382729 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:30.392004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.382951 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-data" pod "7d9f1cd1-5271-4630-b240-27b4cf22b9a8" (UID: "7d9f1cd1-5271-4630-b240-27b4cf22b9a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:30.397917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.397622 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-credential-socket" pod "7d9f1cd1-5271-4630-b240-27b4cf22b9a8" (UID: "7d9f1cd1-5271-4630-b240-27b4cf22b9a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:30.404372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.404143 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-podinfo" pod "7d9f1cd1-5271-4630-b240-27b4cf22b9a8" (UID: "7d9f1cd1-5271-4630-b240-27b4cf22b9a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:12:30.413490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.404680 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-socket" pod "7d9f1cd1-5271-4630-b240-27b4cf22b9a8" (UID: "7d9f1cd1-5271-4630-b240-27b4cf22b9a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:30.413490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.408551 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-certs" pod "7d9f1cd1-5271-4630-b240-27b4cf22b9a8" (UID: "7d9f1cd1-5271-4630-b240-27b4cf22b9a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:30.413490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.413241 6587 generic.go:356] "Generic (PLEG): container finished" podID="7d9f1cd1-5271-4630-b240-27b4cf22b9a8" containerID="ded0e003c4ae947b4f371f58fb875a6980f402d6178c554c114531c735a5d8fb" exitCode=0 Oct 30 05:12:30.413490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.413307 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" event={"ID":"7d9f1cd1-5271-4630-b240-27b4cf22b9a8","Type":"ContainerDied","Data":"ded0e003c4ae947b4f371f58fb875a6980f402d6178c554c114531c735a5d8fb"} Oct 30 05:12:30.413490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.413404 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" event={"ID":"7d9f1cd1-5271-4630-b240-27b4cf22b9a8","Type":"ContainerDied","Data":"1d37eb7c051aad34ea5c9435d3fe7fb87986f33bd46a944f75f5c89618593890"} Oct 30 05:12:30.413490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.413435 6587 scope.go:122] "RemoveContainer" containerID="ded0e003c4ae947b4f371f58fb875a6980f402d6178c554c114531c735a5d8fb" Oct 30 05:12:30.413917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.413625 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl" Oct 30 05:12:30.445381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.440582 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-token" pod "7d9f1cd1-5271-4630-b240-27b4cf22b9a8" (UID: "7d9f1cd1-5271-4630-b240-27b4cf22b9a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:30.447856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.447794 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-envoy" pod "7d9f1cd1-5271-4630-b240-27b4cf22b9a8" (UID: "7d9f1cd1-5271-4630-b240-27b4cf22b9a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:30.461011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.460975 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-kube-api-access-hxrm4" pod "7d9f1cd1-5271-4630-b240-27b4cf22b9a8" (UID: "7d9f1cd1-5271-4630-b240-27b4cf22b9a8"). InnerVolumeSpecName "kube-api-access-hxrm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:30.474327 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.474196 6587 scope.go:122] "RemoveContainer" containerID="ded0e003c4ae947b4f371f58fb875a6980f402d6178c554c114531c735a5d8fb" Oct 30 05:12:30.474862 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:12:30.474787 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded0e003c4ae947b4f371f58fb875a6980f402d6178c554c114531c735a5d8fb\": container with ID starting with ded0e003c4ae947b4f371f58fb875a6980f402d6178c554c114531c735a5d8fb not found: ID does not exist" containerID="ded0e003c4ae947b4f371f58fb875a6980f402d6178c554c114531c735a5d8fb" Oct 30 05:12:30.474862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.474826 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded0e003c4ae947b4f371f58fb875a6980f402d6178c554c114531c735a5d8fb"} err="failed to get container status \"ded0e003c4ae947b4f371f58fb875a6980f402d6178c554c114531c735a5d8fb\": rpc error: code = NotFound desc = could not find container \"ded0e003c4ae947b4f371f58fb875a6980f402d6178c554c114531c735a5d8fb\": container with ID starting with ded0e003c4ae947b4f371f58fb875a6980f402d6178c554c114531c735a5d8fb not found: ID does not exist" Oct 30 05:12:30.484304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.484166 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:30.484304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.484196 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:30.484304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.484211 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:30.484304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.484225 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:30.484304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.484239 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:30.484304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.484253 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:30.484304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.484266 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxrm4\" (UniqueName: \"kubernetes.io/projected/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-kube-api-access-hxrm4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:30.484304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.484280 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9f1cd1-5271-4630-b240-27b4cf22b9a8-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:30.497707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.497669 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g"] Oct 30 05:12:30.498520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.498131 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" containerID="cri-o://80be2ce5cb3062cc402f516986a1e6ecb12bf821af7d663da972ac1093bdc3ab" gracePeriod=30 Oct 30 05:12:30.524227 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.524196 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n"] Oct 30 05:12:30.526132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.526022 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7d9f1cd1-5271-4630-b240-27b4cf22b9a8" containerName="istio-proxy" Oct 30 05:12:30.526558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.526538 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9f1cd1-5271-4630-b240-27b4cf22b9a8" containerName="istio-proxy" Oct 30 05:12:30.526960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.526945 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9f1cd1-5271-4630-b240-27b4cf22b9a8" containerName="istio-proxy" Oct 30 05:12:30.558651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.553055 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" Oct 30 05:12:30.558651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.556876 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n"] Oct 30 05:12:30.571109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.570184 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:12:30.571109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.570413 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:12:30.571109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.570522 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:12:30.571109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.570560 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:12:30.599279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.598505 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n\" (UID: \"555589a4-8974-4af2-a29a-c9b77a664020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" Oct 30 05:12:30.599279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.598815 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jmjt\" (UniqueName: \"kubernetes.io/projected/555589a4-8974-4af2-a29a-c9b77a664020-kube-api-access-5jmjt\") pod \"maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n\" (UID: \"555589a4-8974-4af2-a29a-c9b77a664020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" Oct 30 05:12:30.599279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.598889 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n\" (UID: \"555589a4-8974-4af2-a29a-c9b77a664020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" Oct 30 05:12:30.732763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.732619 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n\" (UID: \"555589a4-8974-4af2-a29a-c9b77a664020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" Oct 30 05:12:30.733285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.731765 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n\" (UID: \"555589a4-8974-4af2-a29a-c9b77a664020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" Oct 30 05:12:30.733680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.733654 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jmjt\" (UniqueName: \"kubernetes.io/projected/555589a4-8974-4af2-a29a-c9b77a664020-kube-api-access-5jmjt\") pod \"maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n\" (UID: \"555589a4-8974-4af2-a29a-c9b77a664020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" Oct 30 05:12:30.734237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.734216 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n\" (UID: \"555589a4-8974-4af2-a29a-c9b77a664020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" Oct 30 05:12:30.845713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.845405 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n\" (UID: \"555589a4-8974-4af2-a29a-c9b77a664020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" Oct 30 05:12:30.851572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.849699 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jmjt\" (UniqueName: \"kubernetes.io/projected/555589a4-8974-4af2-a29a-c9b77a664020-kube-api-access-5jmjt\") pod \"maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n\" (UID: \"555589a4-8974-4af2-a29a-c9b77a664020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" Oct 30 05:12:30.910446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:30.900857 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" Oct 30 05:12:31.028762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.028728 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl"] Oct 30 05:12:31.042703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.042665 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl"] Oct 30 05:12:31.253871 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:12:31.253837 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555589a4_8974_4af2_a29a_c9b77a664020.slice/crio-aa4c9d1b5f66547356a64b093cb4e2cc0b551b0fb28af5f1db2a4270080a48b2 WatchSource:0}: Error finding container aa4c9d1b5f66547356a64b093cb4e2cc0b551b0fb28af5f1db2a4270080a48b2: Status 404 returned error can't find the container with id aa4c9d1b5f66547356a64b093cb4e2cc0b551b0fb28af5f1db2a4270080a48b2 Oct 30 05:12:31.255718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.255697 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n"] Oct 30 05:12:31.426009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.425978 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" event={"ID":"555589a4-8974-4af2-a29a-c9b77a664020","Type":"ContainerStarted","Data":"aa4c9d1b5f66547356a64b093cb4e2cc0b551b0fb28af5f1db2a4270080a48b2"} Oct 30 05:12:31.583655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.583621 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d9f1cd1-5271-4630-b240-27b4cf22b9a8" path="/var/lib/kubelet/pods/7d9f1cd1-5271-4630-b240-27b4cf22b9a8/volumes" Oct 30 05:12:31.828932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.828897 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv"] Oct 30 05:12:31.834286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.834204 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:31.860913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.860879 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-010013-openshift-default-dockercfg-z64dg\"" Oct 30 05:12:31.873115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.873084 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv"] Oct 30 05:12:31.969144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.969087 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-credential-socket\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:31.969339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.969229 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7d7m\" (UniqueName: \"kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-kube-api-access-t7d7m\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:31.969339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.969298 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-certs\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:31.969511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.969352 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-token\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:31.969511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.969420 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istiod-ca-cert\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:31.969626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.969575 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-socket\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:31.969689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.969674 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-podinfo\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:31.969785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.969769 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-envoy\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:31.969848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:31.969837 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-data\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.082402 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7d7m\" (UniqueName: \"kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-kube-api-access-t7d7m\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.082455 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-certs\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.082485 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-token\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.082512 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istiod-ca-cert\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.082565 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-socket\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.082606 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-podinfo\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.082676 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-envoy\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.082704 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-data\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.082808 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-credential-socket\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.083183 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-credential-socket\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.083760 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-certs\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.084633 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istiod-ca-cert\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.084866 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-socket\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.088465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.086450 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-data\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.133509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.121955 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-podinfo\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.133509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.126663 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-envoy\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.147036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.147002 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-token\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.183746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.168227 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7d7m\" (UniqueName: \"kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-kube-api-access-t7d7m\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-scchv\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.215811 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.193421 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:32.397378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.391234 6587 patch_prober.go:29] interesting pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:12:32.397378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.391298 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:12:32.441946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.441913 6587 generic.go:356] "Generic (PLEG): container finished" podID="555589a4-8974-4af2-a29a-c9b77a664020" containerID="ae777fc4250dba1a9cd9288c432ede9548a1d54d8c12bc939d5999b3f4fe1100" exitCode=0 Oct 30 05:12:32.442511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.442418 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" event={"ID":"555589a4-8974-4af2-a29a-c9b77a664020","Type":"ContainerDied","Data":"ae777fc4250dba1a9cd9288c432ede9548a1d54d8c12bc939d5999b3f4fe1100"} Oct 30 05:12:32.631144 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:12:32.631118 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff795b5_eea0_47c6_acba_5aea88e9fea4.slice/crio-7feaff8fa5eae31d119d879e4cf7242dafb8b3db631f059818e8ffbbee45b76f WatchSource:0}: Error finding container 7feaff8fa5eae31d119d879e4cf7242dafb8b3db631f059818e8ffbbee45b76f: Status 404 returned error can't find the container with id 7feaff8fa5eae31d119d879e4cf7242dafb8b3db631f059818e8ffbbee45b76f Oct 30 05:12:32.634178 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.634144 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:12:32.634288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.634213 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:12:32.634288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.634243 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:12:32.635788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:32.635762 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv"] Oct 30 05:12:33.452081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.451986 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" event={"ID":"2ff795b5-eea0-47c6-acba-5aea88e9fea4","Type":"ContainerStarted","Data":"d33226465478fffc519b132e3e3baa41f9ae6b147d6c9a5af221d3ab12b9824f"} Oct 30 05:12:33.452081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.452048 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" event={"ID":"2ff795b5-eea0-47c6-acba-5aea88e9fea4","Type":"ContainerStarted","Data":"7feaff8fa5eae31d119d879e4cf7242dafb8b3db631f059818e8ffbbee45b76f"} Oct 30 05:12:33.542544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.542478 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" podStartSLOduration=2.542458597 podStartE2EDuration="2.542458597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:33.539876865 +0000 UTC m=+1678.067643658" watchObservedRunningTime="2026-10-30 05:12:33.542458597 +0000 UTC m=+1678.070225384" Oct 30 05:12:33.702261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.702188 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" Oct 30 05:12:33.715724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.715681 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t"] Oct 30 05:12:33.717456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.717422 6587 cpu_manager.go:414] "containerMap: removing container" podUID="555589a4-8974-4af2-a29a-c9b77a664020" containerName="revoke-keys" Oct 30 05:12:33.717641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.717621 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555589a4-8974-4af2-a29a-c9b77a664020" containerName="revoke-keys" Oct 30 05:12:33.717878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.717854 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="555589a4-8974-4af2-a29a-c9b77a664020" containerName="revoke-keys" Oct 30 05:12:33.722185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.722162 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" Oct 30 05:12:33.742834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.742792 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t"] Oct 30 05:12:33.772244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.765112 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-openshift-ca\") pod \"555589a4-8974-4af2-a29a-c9b77a664020\" (UID: \"555589a4-8974-4af2-a29a-c9b77a664020\") " Oct 30 05:12:33.772244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.765272 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-xks-ca\") pod \"555589a4-8974-4af2-a29a-c9b77a664020\" (UID: \"555589a4-8974-4af2-a29a-c9b77a664020\") " Oct 30 05:12:33.772244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.765616 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555589a4-8974-4af2-a29a-c9b77a664020-kube-api-access-5jmjt\" (UniqueName: \"kubernetes.io/projected/555589a4-8974-4af2-a29a-c9b77a664020-kube-api-access-5jmjt\") pod \"555589a4-8974-4af2-a29a-c9b77a664020\" (UID: \"555589a4-8974-4af2-a29a-c9b77a664020\") " Oct 30 05:12:33.777548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.772524 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-openshift-ca" pod "555589a4-8974-4af2-a29a-c9b77a664020" (UID: "555589a4-8974-4af2-a29a-c9b77a664020"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:33.784843 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.784811 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555589a4-8974-4af2-a29a-c9b77a664020-kube-api-access-5jmjt" pod "555589a4-8974-4af2-a29a-c9b77a664020" (UID: "555589a4-8974-4af2-a29a-c9b77a664020"). InnerVolumeSpecName "kube-api-access-5jmjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:33.786443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.786408 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-xks-ca" pod "555589a4-8974-4af2-a29a-c9b77a664020" (UID: "555589a4-8974-4af2-a29a-c9b77a664020"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:12:33.868335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.868297 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnqjz\" (UniqueName: \"kubernetes.io/projected/97f8e41c-430f-49a5-ac42-ac192db72560-kube-api-access-bnqjz\") pod \"maas-api-revoke-sub-c0cb4e82-6667t\" (UID: \"97f8e41c-430f-49a5-ac42-ac192db72560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" Oct 30 05:12:33.868682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.868619 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0cb4e82-6667t\" (UID: \"97f8e41c-430f-49a5-ac42-ac192db72560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" Oct 30 05:12:33.868761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.868680 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0cb4e82-6667t\" (UID: \"97f8e41c-430f-49a5-ac42-ac192db72560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" Oct 30 05:12:33.868819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.868800 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:33.868819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.868813 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555589a4-8974-4af2-a29a-c9b77a664020-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:33.868941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.868822 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jmjt\" (UniqueName: \"kubernetes.io/projected/555589a4-8974-4af2-a29a-c9b77a664020-kube-api-access-5jmjt\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:33.970044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.969955 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0cb4e82-6667t\" (UID: \"97f8e41c-430f-49a5-ac42-ac192db72560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" Oct 30 05:12:33.970044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.970009 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0cb4e82-6667t\" (UID: \"97f8e41c-430f-49a5-ac42-ac192db72560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" Oct 30 05:12:33.970278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.970126 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqjz\" (UniqueName: \"kubernetes.io/projected/97f8e41c-430f-49a5-ac42-ac192db72560-kube-api-access-bnqjz\") pod \"maas-api-revoke-sub-c0cb4e82-6667t\" (UID: \"97f8e41c-430f-49a5-ac42-ac192db72560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" Oct 30 05:12:33.970765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.970737 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0cb4e82-6667t\" (UID: \"97f8e41c-430f-49a5-ac42-ac192db72560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" Oct 30 05:12:33.972351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.972331 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0cb4e82-6667t\" (UID: \"97f8e41c-430f-49a5-ac42-ac192db72560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" Oct 30 05:12:33.991796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:33.991775 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqjz\" (UniqueName: \"kubernetes.io/projected/97f8e41c-430f-49a5-ac42-ac192db72560-kube-api-access-bnqjz\") pod \"maas-api-revoke-sub-c0cb4e82-6667t\" (UID: \"97f8e41c-430f-49a5-ac42-ac192db72560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" Oct 30 05:12:34.041836 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:34.041804 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" Oct 30 05:12:34.196247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:34.196213 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:34.203616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:34.203592 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:34.371348 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:12:34.371309 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f8e41c_430f_49a5_ac42_ac192db72560.slice/crio-2847c484311a2db1dab23fd24be527a07a2120f9fd68246ee5a603cfd3932ab4 WatchSource:0}: Error finding container 2847c484311a2db1dab23fd24be527a07a2120f9fd68246ee5a603cfd3932ab4: Status 404 returned error can't find the container with id 2847c484311a2db1dab23fd24be527a07a2120f9fd68246ee5a603cfd3932ab4 Oct 30 05:12:34.372286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:34.372259 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t"] Oct 30 05:12:34.462154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:34.462128 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" Oct 30 05:12:34.462566 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:34.462127 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n" event={"ID":"555589a4-8974-4af2-a29a-c9b77a664020","Type":"ContainerDied","Data":"aa4c9d1b5f66547356a64b093cb4e2cc0b551b0fb28af5f1db2a4270080a48b2"} Oct 30 05:12:34.462566 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:34.462263 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa4c9d1b5f66547356a64b093cb4e2cc0b551b0fb28af5f1db2a4270080a48b2" Oct 30 05:12:34.463556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:34.463531 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" event={"ID":"97f8e41c-430f-49a5-ac42-ac192db72560","Type":"ContainerStarted","Data":"2847c484311a2db1dab23fd24be527a07a2120f9fd68246ee5a603cfd3932ab4"} Oct 30 05:12:34.463751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:34.463726 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:34.464723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:34.464707 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:12:35.002548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.002519 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:12:35.099947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.099906 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-istio-token\" (UniqueName: \"kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-istio-token\") pod \"484fe4bc-1006-4745-ac99-408b72f728d8\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " Oct 30 05:12:35.100249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.100226 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/484fe4bc-1006-4745-ac99-408b72f728d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484fe4bc-1006-4745-ac99-408b72f728d8-istio-podinfo\") pod \"484fe4bc-1006-4745-ac99-408b72f728d8\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " Oct 30 05:12:35.100478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.100457 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-envoy\") pod \"484fe4bc-1006-4745-ac99-408b72f728d8\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " Oct 30 05:12:35.103536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.100599 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/484fe4bc-1006-4745-ac99-408b72f728d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484fe4bc-1006-4745-ac99-408b72f728d8-istiod-ca-cert\") pod \"484fe4bc-1006-4745-ac99-408b72f728d8\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " Oct 30 05:12:35.103773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.103744 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484fe4bc-1006-4745-ac99-408b72f728d8-istiod-ca-cert" pod "484fe4bc-1006-4745-ac99-408b72f728d8" (UID: "484fe4bc-1006-4745-ac99-408b72f728d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:35.103927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.103908 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-kube-api-access-mrp7s\" (UniqueName: \"kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-kube-api-access-mrp7s\") pod \"484fe4bc-1006-4745-ac99-408b72f728d8\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " Oct 30 05:12:35.104003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.103991 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-socket\") pod \"484fe4bc-1006-4745-ac99-408b72f728d8\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " Oct 30 05:12:35.104081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.104067 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-certs\") pod \"484fe4bc-1006-4745-ac99-408b72f728d8\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " Oct 30 05:12:35.104168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.104152 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-credential-socket\") pod \"484fe4bc-1006-4745-ac99-408b72f728d8\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " Oct 30 05:12:35.104222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.104181 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-data\") pod \"484fe4bc-1006-4745-ac99-408b72f728d8\" (UID: \"484fe4bc-1006-4745-ac99-408b72f728d8\") " Oct 30 05:12:35.105137 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.105017 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484fe4bc-1006-4745-ac99-408b72f728d8-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.105313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.105208 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-data" pod "484fe4bc-1006-4745-ac99-408b72f728d8" (UID: "484fe4bc-1006-4745-ac99-408b72f728d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:35.105410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.105349 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-certs" pod "484fe4bc-1006-4745-ac99-408b72f728d8" (UID: "484fe4bc-1006-4745-ac99-408b72f728d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:35.105529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.105507 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-credential-socket" pod "484fe4bc-1006-4745-ac99-408b72f728d8" (UID: "484fe4bc-1006-4745-ac99-408b72f728d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:35.105771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.105750 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-socket" pod "484fe4bc-1006-4745-ac99-408b72f728d8" (UID: "484fe4bc-1006-4745-ac99-408b72f728d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:35.108553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.108524 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-kube-api-access-mrp7s" pod "484fe4bc-1006-4745-ac99-408b72f728d8" (UID: "484fe4bc-1006-4745-ac99-408b72f728d8"). InnerVolumeSpecName "kube-api-access-mrp7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:35.109351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.109313 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-istio-token" pod "484fe4bc-1006-4745-ac99-408b72f728d8" (UID: "484fe4bc-1006-4745-ac99-408b72f728d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:35.109556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.109518 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-envoy" pod "484fe4bc-1006-4745-ac99-408b72f728d8" (UID: "484fe4bc-1006-4745-ac99-408b72f728d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:35.110901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.110875 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/484fe4bc-1006-4745-ac99-408b72f728d8-istio-podinfo" pod "484fe4bc-1006-4745-ac99-408b72f728d8" (UID: "484fe4bc-1006-4745-ac99-408b72f728d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:12:35.207906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.207803 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrp7s\" (UniqueName: \"kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-kube-api-access-mrp7s\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.207906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.207850 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.207906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.207864 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.207906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.207876 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.207906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.207888 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.207906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.207903 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484fe4bc-1006-4745-ac99-408b72f728d8-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.208255 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.207917 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484fe4bc-1006-4745-ac99-408b72f728d8-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.208255 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.207927 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484fe4bc-1006-4745-ac99-408b72f728d8-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.560312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.559166 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" event={"ID":"97f8e41c-430f-49a5-ac42-ac192db72560","Type":"ContainerStarted","Data":"b8a3c16ed187731d5aa692196e6bc0e9a7cb99bd43e0b754ebad91d65417f976"} Oct 30 05:12:35.562308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.562279 6587 generic.go:356] "Generic (PLEG): container finished" podID="484fe4bc-1006-4745-ac99-408b72f728d8" containerID="f1910e6f4fcdf6a8707221cabf8106086038eee6e283e52afa5b73b05cea2b4f" exitCode=0 Oct 30 05:12:35.563419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.563395 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" Oct 30 05:12:35.568497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.568439 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" event={"ID":"484fe4bc-1006-4745-ac99-408b72f728d8","Type":"ContainerDied","Data":"f1910e6f4fcdf6a8707221cabf8106086038eee6e283e52afa5b73b05cea2b4f"} Oct 30 05:12:35.568644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.568502 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx" event={"ID":"484fe4bc-1006-4745-ac99-408b72f728d8","Type":"ContainerDied","Data":"1bd57d11874092dead9aa355cc047203050c072e779723f17cd15b6284192353"} Oct 30 05:12:35.568644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.568540 6587 scope.go:122] "RemoveContainer" containerID="f1910e6f4fcdf6a8707221cabf8106086038eee6e283e52afa5b73b05cea2b4f" Oct 30 05:12:35.583970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.583942 6587 scope.go:122] "RemoveContainer" containerID="fe9d7b90177dff739bb0d249895d600784cb509bda8762551a3ad665e6a01077" Oct 30 05:12:35.608662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.608608 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" podStartSLOduration=2.6085611 podStartE2EDuration="2.6085611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:35.604564966 +0000 UTC m=+1680.132331755" watchObservedRunningTime="2026-10-30 05:12:35.6085611 +0000 UTC m=+1680.136327903" Oct 30 05:12:35.712411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.712389 6587 scope.go:122] "RemoveContainer" containerID="f1910e6f4fcdf6a8707221cabf8106086038eee6e283e52afa5b73b05cea2b4f" Oct 30 05:12:35.712742 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:12:35.712718 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1910e6f4fcdf6a8707221cabf8106086038eee6e283e52afa5b73b05cea2b4f\": container with ID starting with f1910e6f4fcdf6a8707221cabf8106086038eee6e283e52afa5b73b05cea2b4f not found: ID does not exist" containerID="f1910e6f4fcdf6a8707221cabf8106086038eee6e283e52afa5b73b05cea2b4f" Oct 30 05:12:35.712788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.712753 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1910e6f4fcdf6a8707221cabf8106086038eee6e283e52afa5b73b05cea2b4f"} err="failed to get container status \"f1910e6f4fcdf6a8707221cabf8106086038eee6e283e52afa5b73b05cea2b4f\": rpc error: code = NotFound desc = could not find container \"f1910e6f4fcdf6a8707221cabf8106086038eee6e283e52afa5b73b05cea2b4f\": container with ID starting with f1910e6f4fcdf6a8707221cabf8106086038eee6e283e52afa5b73b05cea2b4f not found: ID does not exist" Oct 30 05:12:35.712788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.712774 6587 scope.go:122] "RemoveContainer" containerID="fe9d7b90177dff739bb0d249895d600784cb509bda8762551a3ad665e6a01077" Oct 30 05:12:35.713069 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:12:35.713048 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9d7b90177dff739bb0d249895d600784cb509bda8762551a3ad665e6a01077\": container with ID starting with fe9d7b90177dff739bb0d249895d600784cb509bda8762551a3ad665e6a01077 not found: ID does not exist" containerID="fe9d7b90177dff739bb0d249895d600784cb509bda8762551a3ad665e6a01077" Oct 30 05:12:35.713125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.713081 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9d7b90177dff739bb0d249895d600784cb509bda8762551a3ad665e6a01077"} err="failed to get container status \"fe9d7b90177dff739bb0d249895d600784cb509bda8762551a3ad665e6a01077\": rpc error: code = NotFound desc = could not find container \"fe9d7b90177dff739bb0d249895d600784cb509bda8762551a3ad665e6a01077\": container with ID starting with fe9d7b90177dff739bb0d249895d600784cb509bda8762551a3ad665e6a01077 not found: ID does not exist" Oct 30 05:12:35.750931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.750898 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx"] Oct 30 05:12:35.767425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.766975 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx"] Oct 30 05:12:35.854790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.854762 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:12:35.861616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.861586 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx"] Oct 30 05:12:35.866638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.866615 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288e01-mxmqx"] Oct 30 05:12:35.931912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.931854 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-podinfo\") pod \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " Oct 30 05:12:35.940803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.937909 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-kube-api-access-szslv\" (UniqueName: \"kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-kube-api-access-szslv\") pod \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " Oct 30 05:12:35.943458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.941701 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-certs\") pod \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " Oct 30 05:12:35.943458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.941813 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-socket\") pod \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " Oct 30 05:12:35.943458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.941873 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-token\" (UniqueName: \"kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-token\") pod \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " Oct 30 05:12:35.943458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.941964 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-credential-socket\") pod \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " Oct 30 05:12:35.943458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.942021 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-envoy\") pod \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " Oct 30 05:12:35.943458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.942060 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istiod-ca-cert\") pod \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " Oct 30 05:12:35.943458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.942153 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-data\") pod \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\" (UID: \"8a0adf45-f6fc-41c4-9c8b-142f595fd03f\") " Oct 30 05:12:35.943458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.943275 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-podinfo" pod "8a0adf45-f6fc-41c4-9c8b-142f595fd03f" (UID: "8a0adf45-f6fc-41c4-9c8b-142f595fd03f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:12:35.951277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.944736 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istiod-ca-cert" pod "8a0adf45-f6fc-41c4-9c8b-142f595fd03f" (UID: "8a0adf45-f6fc-41c4-9c8b-142f595fd03f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:35.951277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.945020 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-certs" pod "8a0adf45-f6fc-41c4-9c8b-142f595fd03f" (UID: "8a0adf45-f6fc-41c4-9c8b-142f595fd03f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:35.951277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.945208 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-socket" pod "8a0adf45-f6fc-41c4-9c8b-142f595fd03f" (UID: "8a0adf45-f6fc-41c4-9c8b-142f595fd03f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:35.951277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.946435 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-credential-socket" pod "8a0adf45-f6fc-41c4-9c8b-142f595fd03f" (UID: "8a0adf45-f6fc-41c4-9c8b-142f595fd03f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:35.951277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.946716 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-data" pod "8a0adf45-f6fc-41c4-9c8b-142f595fd03f" (UID: "8a0adf45-f6fc-41c4-9c8b-142f595fd03f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:35.951277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.948850 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-kube-api-access-szslv" pod "8a0adf45-f6fc-41c4-9c8b-142f595fd03f" (UID: "8a0adf45-f6fc-41c4-9c8b-142f595fd03f"). InnerVolumeSpecName "kube-api-access-szslv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:35.960430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.955740 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-envoy" pod "8a0adf45-f6fc-41c4-9c8b-142f595fd03f" (UID: "8a0adf45-f6fc-41c4-9c8b-142f595fd03f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:35.963275 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.955884 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-token" pod "8a0adf45-f6fc-41c4-9c8b-142f595fd03f" (UID: "8a0adf45-f6fc-41c4-9c8b-142f595fd03f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:35.975597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.975568 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.975597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.975600 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.975820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.975614 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.975820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.975628 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.975820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.975639 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.975820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.975651 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.975820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.975662 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.975820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.975676 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szslv\" (UniqueName: \"kubernetes.io/projected/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-kube-api-access-szslv\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:35.975820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:35.975689 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0adf45-f6fc-41c4-9c8b-142f595fd03f-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:36.574048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.574016 6587 generic.go:356] "Generic (PLEG): container finished" podID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerID="80be2ce5cb3062cc402f516986a1e6ecb12bf821af7d663da972ac1093bdc3ab" exitCode=0 Oct 30 05:12:36.574684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.574066 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" Oct 30 05:12:36.574684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.574104 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" event={"ID":"8a0adf45-f6fc-41c4-9c8b-142f595fd03f","Type":"ContainerDied","Data":"80be2ce5cb3062cc402f516986a1e6ecb12bf821af7d663da972ac1093bdc3ab"} Oct 30 05:12:36.574684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.574150 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g" event={"ID":"8a0adf45-f6fc-41c4-9c8b-142f595fd03f","Type":"ContainerDied","Data":"d8f3cf73fabced40c34aaebc2469c0bf39cd032e7bb7fed0cb776967246d4f27"} Oct 30 05:12:36.574684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.574178 6587 scope.go:122] "RemoveContainer" containerID="80be2ce5cb3062cc402f516986a1e6ecb12bf821af7d663da972ac1093bdc3ab" Oct 30 05:12:36.576349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.576324 6587 generic.go:356] "Generic (PLEG): container finished" podID="97f8e41c-430f-49a5-ac42-ac192db72560" containerID="b8a3c16ed187731d5aa692196e6bc0e9a7cb99bd43e0b754ebad91d65417f976" exitCode=0 Oct 30 05:12:36.576508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.576420 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" event={"ID":"97f8e41c-430f-49a5-ac42-ac192db72560","Type":"ContainerDied","Data":"b8a3c16ed187731d5aa692196e6bc0e9a7cb99bd43e0b754ebad91d65417f976"} Oct 30 05:12:36.594018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.593991 6587 scope.go:122] "RemoveContainer" containerID="98dd0487492a898bea84251bab5e1fb2938a4491fc2a5c450e55dc5296df4501" Oct 30 05:12:36.604454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.604375 6587 scope.go:122] "RemoveContainer" containerID="80be2ce5cb3062cc402f516986a1e6ecb12bf821af7d663da972ac1093bdc3ab" Oct 30 05:12:36.604713 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:12:36.604689 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80be2ce5cb3062cc402f516986a1e6ecb12bf821af7d663da972ac1093bdc3ab\": container with ID starting with 80be2ce5cb3062cc402f516986a1e6ecb12bf821af7d663da972ac1093bdc3ab not found: ID does not exist" containerID="80be2ce5cb3062cc402f516986a1e6ecb12bf821af7d663da972ac1093bdc3ab" Oct 30 05:12:36.604775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.604728 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80be2ce5cb3062cc402f516986a1e6ecb12bf821af7d663da972ac1093bdc3ab"} err="failed to get container status \"80be2ce5cb3062cc402f516986a1e6ecb12bf821af7d663da972ac1093bdc3ab\": rpc error: code = NotFound desc = could not find container \"80be2ce5cb3062cc402f516986a1e6ecb12bf821af7d663da972ac1093bdc3ab\": container with ID starting with 80be2ce5cb3062cc402f516986a1e6ecb12bf821af7d663da972ac1093bdc3ab not found: ID does not exist" Oct 30 05:12:36.604775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.604752 6587 scope.go:122] "RemoveContainer" containerID="98dd0487492a898bea84251bab5e1fb2938a4491fc2a5c450e55dc5296df4501" Oct 30 05:12:36.605029 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:12:36.605012 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98dd0487492a898bea84251bab5e1fb2938a4491fc2a5c450e55dc5296df4501\": container with ID starting with 98dd0487492a898bea84251bab5e1fb2938a4491fc2a5c450e55dc5296df4501 not found: ID does not exist" containerID="98dd0487492a898bea84251bab5e1fb2938a4491fc2a5c450e55dc5296df4501" Oct 30 05:12:36.605069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.605032 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98dd0487492a898bea84251bab5e1fb2938a4491fc2a5c450e55dc5296df4501"} err="failed to get container status \"98dd0487492a898bea84251bab5e1fb2938a4491fc2a5c450e55dc5296df4501\": rpc error: code = NotFound desc = could not find container \"98dd0487492a898bea84251bab5e1fb2938a4491fc2a5c450e55dc5296df4501\": container with ID starting with 98dd0487492a898bea84251bab5e1fb2938a4491fc2a5c450e55dc5296df4501 not found: ID does not exist" Oct 30 05:12:36.737899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.737866 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g"] Oct 30 05:12:36.747729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:36.747699 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g"] Oct 30 05:12:37.587461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:37.587423 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" path="/var/lib/kubelet/pods/484fe4bc-1006-4745-ac99-408b72f728d8/volumes" Oct 30 05:12:37.588115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:37.588094 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" path="/var/lib/kubelet/pods/8a0adf45-f6fc-41c4-9c8b-142f595fd03f/volumes" Oct 30 05:12:37.588580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:37.588564 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99d8290-5005-4241-8d45-8791f5640d58" path="/var/lib/kubelet/pods/f99d8290-5005-4241-8d45-8791f5640d58/volumes" Oct 30 05:12:37.785237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:37.785208 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" Oct 30 05:12:37.908112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:37.908027 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-xks-ca\") pod \"97f8e41c-430f-49a5-ac42-ac192db72560\" (UID: \"97f8e41c-430f-49a5-ac42-ac192db72560\") " Oct 30 05:12:37.908269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:37.908175 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f8e41c-430f-49a5-ac42-ac192db72560-kube-api-access-bnqjz\" (UniqueName: \"kubernetes.io/projected/97f8e41c-430f-49a5-ac42-ac192db72560-kube-api-access-bnqjz\") pod \"97f8e41c-430f-49a5-ac42-ac192db72560\" (UID: \"97f8e41c-430f-49a5-ac42-ac192db72560\") " Oct 30 05:12:37.908269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:37.908248 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-openshift-ca\") pod \"97f8e41c-430f-49a5-ac42-ac192db72560\" (UID: \"97f8e41c-430f-49a5-ac42-ac192db72560\") " Oct 30 05:12:37.908886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:37.908855 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-openshift-ca" pod "97f8e41c-430f-49a5-ac42-ac192db72560" (UID: "97f8e41c-430f-49a5-ac42-ac192db72560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:37.911233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:37.911208 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f8e41c-430f-49a5-ac42-ac192db72560-kube-api-access-bnqjz" pod "97f8e41c-430f-49a5-ac42-ac192db72560" (UID: "97f8e41c-430f-49a5-ac42-ac192db72560"). InnerVolumeSpecName "kube-api-access-bnqjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:37.911406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:37.911378 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-xks-ca" pod "97f8e41c-430f-49a5-ac42-ac192db72560" (UID: "97f8e41c-430f-49a5-ac42-ac192db72560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:12:38.010688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:38.010654 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:38.010688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:38.010686 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f8e41c-430f-49a5-ac42-ac192db72560-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:38.010688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:38.010696 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnqjz\" (UniqueName: \"kubernetes.io/projected/97f8e41c-430f-49a5-ac42-ac192db72560-kube-api-access-bnqjz\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:38.609862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:38.609823 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" Oct 30 05:12:38.610311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:38.610065 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t" event={"ID":"97f8e41c-430f-49a5-ac42-ac192db72560","Type":"ContainerDied","Data":"2847c484311a2db1dab23fd24be527a07a2120f9fd68246ee5a603cfd3932ab4"} Oct 30 05:12:38.610311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:38.610136 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2847c484311a2db1dab23fd24be527a07a2120f9fd68246ee5a603cfd3932ab4" Oct 30 05:12:41.041574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:41.041536 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8"] Oct 30 05:12:43.716036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:43.715999 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt"] Oct 30 05:12:43.716477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:43.716233 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" containerID="cri-o://51ced6ec618ad6680dce6b504f410d5f891154acd39c011019adb6fd27936924" gracePeriod=30 Oct 30 05:12:44.036335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.036235 6587 patch_prober.go:29] interesting pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:12:44.036665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.036625 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:12:44.201350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.201299 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss"] Oct 30 05:12:44.202247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202222 6587 cpu_manager.go:414] "containerMap: removing container" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" Oct 30 05:12:44.202247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202245 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" Oct 30 05:12:44.202509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202277 6587 cpu_manager.go:414] "containerMap: removing container" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" Oct 30 05:12:44.202509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202286 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" Oct 30 05:12:44.202509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202323 6587 cpu_manager.go:414] "containerMap: removing container" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" Oct 30 05:12:44.202509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202331 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" Oct 30 05:12:44.202509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202372 6587 cpu_manager.go:414] "containerMap: removing container" podUID="97f8e41c-430f-49a5-ac42-ac192db72560" containerName="revoke-keys" Oct 30 05:12:44.202509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202380 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f8e41c-430f-49a5-ac42-ac192db72560" containerName="revoke-keys" Oct 30 05:12:44.202509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202396 6587 cpu_manager.go:414] "containerMap: removing container" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" Oct 30 05:12:44.202509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202403 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" Oct 30 05:12:44.202923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202675 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" Oct 30 05:12:44.202923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202697 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f8e41c-430f-49a5-ac42-ac192db72560" containerName="revoke-keys" Oct 30 05:12:44.202923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202708 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a0adf45-f6fc-41c4-9c8b-142f595fd03f" containerName="istio-proxy" Oct 30 05:12:44.202923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202727 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" Oct 30 05:12:44.202923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.202742 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="484fe4bc-1006-4745-ac99-408b72f728d8" containerName="istio-proxy" Oct 30 05:12:44.208239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.208210 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" Oct 30 05:12:44.228929 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.224211 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss"] Oct 30 05:12:44.229430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.229367 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:12:44.229934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.229822 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:12:44.229934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.229908 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:12:44.230221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.230188 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:12:44.247451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.246445 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pmwh\" (UniqueName: \"kubernetes.io/projected/b3baff95-ae1b-4863-8516-b566fdbdf1b8-kube-api-access-7pmwh\") pod \"maas-api-revoke-keys-e2e-api-b-47af37-g87ss\" (UID: \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" Oct 30 05:12:44.247451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.246612 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47af37-g87ss\" (UID: \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" Oct 30 05:12:44.247451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.246670 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47af37-g87ss\" (UID: \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" Oct 30 05:12:44.348618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.348583 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47af37-g87ss\" (UID: \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" Oct 30 05:12:44.348837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.348679 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47af37-g87ss\" (UID: \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" Oct 30 05:12:44.348911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.348898 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmwh\" (UniqueName: \"kubernetes.io/projected/b3baff95-ae1b-4863-8516-b566fdbdf1b8-kube-api-access-7pmwh\") pod \"maas-api-revoke-keys-e2e-api-b-47af37-g87ss\" (UID: \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" Oct 30 05:12:44.349421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.349396 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47af37-g87ss\" (UID: \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" Oct 30 05:12:44.351159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.351131 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47af37-g87ss\" (UID: \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" Oct 30 05:12:44.374728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.374696 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmwh\" (UniqueName: \"kubernetes.io/projected/b3baff95-ae1b-4863-8516-b566fdbdf1b8-kube-api-access-7pmwh\") pod \"maas-api-revoke-keys-e2e-api-b-47af37-g87ss\" (UID: \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" Oct 30 05:12:44.528730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.528697 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" Oct 30 05:12:44.980917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:44.953418 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss"] Oct 30 05:12:45.520102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.520024 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4"] Oct 30 05:12:45.524944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.524918 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" Oct 30 05:12:45.555199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.547229 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4"] Oct 30 05:12:45.617877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.617830 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72a37b86-nmsp4\" (UID: \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" Oct 30 05:12:45.618066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.618018 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72a37b86-nmsp4\" (UID: \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" Oct 30 05:12:45.618131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.618111 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8qc6\" (UniqueName: \"kubernetes.io/projected/10bbc35b-d12d-41c4-acd9-6d3a484459a8-kube-api-access-f8qc6\") pod \"maas-api-revoke-sub-72a37b86-nmsp4\" (UID: \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" Oct 30 05:12:45.687263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.687222 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" event={"ID":"b3baff95-ae1b-4863-8516-b566fdbdf1b8","Type":"ContainerStarted","Data":"8e66baba563954f8df31237655ef91a8d3b03a492c1d6b5a1f6add54d53d2ddf"} Oct 30 05:12:45.687446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.687278 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" event={"ID":"b3baff95-ae1b-4863-8516-b566fdbdf1b8","Type":"ContainerStarted","Data":"1b4d6db259cb710d34fcddd4c2a72f9017d1ef96b23d8196e4ec33640d4f8e7e"} Oct 30 05:12:45.719405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.719371 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8qc6\" (UniqueName: \"kubernetes.io/projected/10bbc35b-d12d-41c4-acd9-6d3a484459a8-kube-api-access-f8qc6\") pod \"maas-api-revoke-sub-72a37b86-nmsp4\" (UID: \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" Oct 30 05:12:45.719637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.719616 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72a37b86-nmsp4\" (UID: \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" Oct 30 05:12:45.719890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.719805 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72a37b86-nmsp4\" (UID: \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" Oct 30 05:12:45.733076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.733012 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" podStartSLOduration=1.732993939 podStartE2EDuration="1.732993939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:45.729760217 +0000 UTC m=+1690.257527005" watchObservedRunningTime="2026-10-30 05:12:45.732993939 +0000 UTC m=+1690.260760726" Oct 30 05:12:45.739900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.739869 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72a37b86-nmsp4\" (UID: \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" Oct 30 05:12:45.741399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.741368 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72a37b86-nmsp4\" (UID: \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" Oct 30 05:12:45.743562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.743540 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8qc6\" (UniqueName: \"kubernetes.io/projected/10bbc35b-d12d-41c4-acd9-6d3a484459a8-kube-api-access-f8qc6\") pod \"maas-api-revoke-sub-72a37b86-nmsp4\" (UID: \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" Oct 30 05:12:45.842712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:45.842670 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" Oct 30 05:12:46.047567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.045417 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" podUID="2a7858bf-8a5c-44d6-ba1d-f37f7e567124" containerName="payload-processing" containerID="cri-o://3136733369e648d24a0fb2d37b0d89e8ce45361f86d10a0cae6b5708a89106dd" gracePeriod=25 Oct 30 05:12:46.221573 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:12:46.221534 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10bbc35b_d12d_41c4_acd9_6d3a484459a8.slice/crio-43b3a5a855c997ae7eaf9de65af7cc186cb1bb4dee1722e18d6d7cd15fb46032 WatchSource:0}: Error finding container 43b3a5a855c997ae7eaf9de65af7cc186cb1bb4dee1722e18d6d7cd15fb46032: Status 404 returned error can't find the container with id 43b3a5a855c997ae7eaf9de65af7cc186cb1bb4dee1722e18d6d7cd15fb46032 Oct 30 05:12:46.225464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.225436 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4"] Oct 30 05:12:46.349734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.349702 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" Oct 30 05:12:46.470408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.469260 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-plugins-config-volume\") pod \"2a7858bf-8a5c-44d6-ba1d-f37f7e567124\" (UID: \"2a7858bf-8a5c-44d6-ba1d-f37f7e567124\") " Oct 30 05:12:46.470408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.469642 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-kube-api-access-lcc69\" (UniqueName: \"kubernetes.io/projected/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-kube-api-access-lcc69\") pod \"2a7858bf-8a5c-44d6-ba1d-f37f7e567124\" (UID: \"2a7858bf-8a5c-44d6-ba1d-f37f7e567124\") " Oct 30 05:12:46.470408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.469868 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-plugins-config-volume" pod "2a7858bf-8a5c-44d6-ba1d-f37f7e567124" (UID: "2a7858bf-8a5c-44d6-ba1d-f37f7e567124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:46.470408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.470379 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:46.482111 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.482061 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-kube-api-access-lcc69" pod "2a7858bf-8a5c-44d6-ba1d-f37f7e567124" (UID: "2a7858bf-8a5c-44d6-ba1d-f37f7e567124"). InnerVolumeSpecName "kube-api-access-lcc69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:46.571530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.571487 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcc69\" (UniqueName: \"kubernetes.io/projected/2a7858bf-8a5c-44d6-ba1d-f37f7e567124-kube-api-access-lcc69\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:46.696520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.696436 6587 generic.go:356] "Generic (PLEG): container finished" podID="2a7858bf-8a5c-44d6-ba1d-f37f7e567124" containerID="3136733369e648d24a0fb2d37b0d89e8ce45361f86d10a0cae6b5708a89106dd" exitCode=0 Oct 30 05:12:46.696520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.696495 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" Oct 30 05:12:46.696520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.696524 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" event={"ID":"2a7858bf-8a5c-44d6-ba1d-f37f7e567124","Type":"ContainerDied","Data":"3136733369e648d24a0fb2d37b0d89e8ce45361f86d10a0cae6b5708a89106dd"} Oct 30 05:12:46.696813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.696584 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8" event={"ID":"2a7858bf-8a5c-44d6-ba1d-f37f7e567124","Type":"ContainerDied","Data":"c62285268e33bca5031a4fc3bc14a92468a7e29d3bfed40626ea124b6d2747b2"} Oct 30 05:12:46.696813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.696608 6587 scope.go:122] "RemoveContainer" containerID="3136733369e648d24a0fb2d37b0d89e8ce45361f86d10a0cae6b5708a89106dd" Oct 30 05:12:46.698631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.698603 6587 generic.go:356] "Generic (PLEG): container finished" podID="b3baff95-ae1b-4863-8516-b566fdbdf1b8" containerID="8e66baba563954f8df31237655ef91a8d3b03a492c1d6b5a1f6add54d53d2ddf" exitCode=0 Oct 30 05:12:46.698784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.698691 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" event={"ID":"b3baff95-ae1b-4863-8516-b566fdbdf1b8","Type":"ContainerDied","Data":"8e66baba563954f8df31237655ef91a8d3b03a492c1d6b5a1f6add54d53d2ddf"} Oct 30 05:12:46.700234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.700210 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" event={"ID":"10bbc35b-d12d-41c4-acd9-6d3a484459a8","Type":"ContainerStarted","Data":"43b3a5a855c997ae7eaf9de65af7cc186cb1bb4dee1722e18d6d7cd15fb46032"} Oct 30 05:12:46.710831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.710793 6587 scope.go:122] "RemoveContainer" containerID="3136733369e648d24a0fb2d37b0d89e8ce45361f86d10a0cae6b5708a89106dd" Oct 30 05:12:46.711807 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:12:46.711770 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3136733369e648d24a0fb2d37b0d89e8ce45361f86d10a0cae6b5708a89106dd\": container with ID starting with 3136733369e648d24a0fb2d37b0d89e8ce45361f86d10a0cae6b5708a89106dd not found: ID does not exist" containerID="3136733369e648d24a0fb2d37b0d89e8ce45361f86d10a0cae6b5708a89106dd" Oct 30 05:12:46.711896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.711813 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3136733369e648d24a0fb2d37b0d89e8ce45361f86d10a0cae6b5708a89106dd"} err="failed to get container status \"3136733369e648d24a0fb2d37b0d89e8ce45361f86d10a0cae6b5708a89106dd\": rpc error: code = NotFound desc = could not find container \"3136733369e648d24a0fb2d37b0d89e8ce45361f86d10a0cae6b5708a89106dd\": container with ID starting with 3136733369e648d24a0fb2d37b0d89e8ce45361f86d10a0cae6b5708a89106dd not found: ID does not exist" Oct 30 05:12:46.758424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.758390 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8"] Oct 30 05:12:46.770529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:46.770500 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8"] Oct 30 05:12:47.585901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:47.585863 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7858bf-8a5c-44d6-ba1d-f37f7e567124" path="/var/lib/kubelet/pods/2a7858bf-8a5c-44d6-ba1d-f37f7e567124/volumes" Oct 30 05:12:47.711936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:47.711844 6587 generic.go:356] "Generic (PLEG): container finished" podID="10bbc35b-d12d-41c4-acd9-6d3a484459a8" containerID="af3a7576ffd4d4081caa1b0fc351b9e13c74ca1d91fc8e8ce7592fd53572f5a4" exitCode=0 Oct 30 05:12:47.712080 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:47.711936 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" event={"ID":"10bbc35b-d12d-41c4-acd9-6d3a484459a8","Type":"ContainerDied","Data":"af3a7576ffd4d4081caa1b0fc351b9e13c74ca1d91fc8e8ce7592fd53572f5a4"} Oct 30 05:12:47.932897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:47.932867 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" Oct 30 05:12:48.012167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:48.012075 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-xks-ca\") pod \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\" (UID: \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\") " Oct 30 05:12:48.012167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:48.012155 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3baff95-ae1b-4863-8516-b566fdbdf1b8-kube-api-access-7pmwh\" (UniqueName: \"kubernetes.io/projected/b3baff95-ae1b-4863-8516-b566fdbdf1b8-kube-api-access-7pmwh\") pod \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\" (UID: \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\") " Oct 30 05:12:48.012507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:48.012466 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-openshift-ca\") pod \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\" (UID: \"b3baff95-ae1b-4863-8516-b566fdbdf1b8\") " Oct 30 05:12:48.013061 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:48.013031 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-openshift-ca" pod "b3baff95-ae1b-4863-8516-b566fdbdf1b8" (UID: "b3baff95-ae1b-4863-8516-b566fdbdf1b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:48.013519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:48.013494 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:48.014600 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:48.014571 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-xks-ca" pod "b3baff95-ae1b-4863-8516-b566fdbdf1b8" (UID: "b3baff95-ae1b-4863-8516-b566fdbdf1b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:12:48.015126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:48.015100 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3baff95-ae1b-4863-8516-b566fdbdf1b8-kube-api-access-7pmwh" pod "b3baff95-ae1b-4863-8516-b566fdbdf1b8" (UID: "b3baff95-ae1b-4863-8516-b566fdbdf1b8"). InnerVolumeSpecName "kube-api-access-7pmwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:48.115323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:48.115290 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3baff95-ae1b-4863-8516-b566fdbdf1b8-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:48.115323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:48.115318 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pmwh\" (UniqueName: \"kubernetes.io/projected/b3baff95-ae1b-4863-8516-b566fdbdf1b8-kube-api-access-7pmwh\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:48.726173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:48.726141 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" Oct 30 05:12:48.726173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:48.726148 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss" event={"ID":"b3baff95-ae1b-4863-8516-b566fdbdf1b8","Type":"ContainerDied","Data":"1b4d6db259cb710d34fcddd4c2a72f9017d1ef96b23d8196e4ec33640d4f8e7e"} Oct 30 05:12:48.726657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:48.726216 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b4d6db259cb710d34fcddd4c2a72f9017d1ef96b23d8196e4ec33640d4f8e7e" Oct 30 05:12:48.957964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:48.957937 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" Oct 30 05:12:49.047592 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.047544 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-openshift-ca\") pod \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\" (UID: \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\") " Oct 30 05:12:49.049576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.047699 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-xks-ca\") pod \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\" (UID: \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\") " Oct 30 05:12:49.049576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.047834 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10bbc35b-d12d-41c4-acd9-6d3a484459a8-kube-api-access-f8qc6\" (UniqueName: \"kubernetes.io/projected/10bbc35b-d12d-41c4-acd9-6d3a484459a8-kube-api-access-f8qc6\") pod \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\" (UID: \"10bbc35b-d12d-41c4-acd9-6d3a484459a8\") " Oct 30 05:12:49.049576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.048105 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-openshift-ca" pod "10bbc35b-d12d-41c4-acd9-6d3a484459a8" (UID: "10bbc35b-d12d-41c4-acd9-6d3a484459a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:49.049576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.048684 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:49.052831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.051603 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bbc35b-d12d-41c4-acd9-6d3a484459a8-kube-api-access-f8qc6" pod "10bbc35b-d12d-41c4-acd9-6d3a484459a8" (UID: "10bbc35b-d12d-41c4-acd9-6d3a484459a8"). InnerVolumeSpecName "kube-api-access-f8qc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:49.055502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.055473 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-xks-ca" pod "10bbc35b-d12d-41c4-acd9-6d3a484459a8" (UID: "10bbc35b-d12d-41c4-acd9-6d3a484459a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:12:49.062776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.062625 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.153893 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-kube-api-access-kn47c\" (UniqueName: \"kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-kube-api-access-kn47c\") pod \"793b0190-5368-47f1-ae26-e6f736e9e878\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.153978 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-socket\") pod \"793b0190-5368-47f1-ae26-e6f736e9e878\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.154008 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-certs\") pod \"793b0190-5368-47f1-ae26-e6f736e9e878\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.154332 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/793b0190-5368-47f1-ae26-e6f736e9e878-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793b0190-5368-47f1-ae26-e6f736e9e878-istio-podinfo\") pod \"793b0190-5368-47f1-ae26-e6f736e9e878\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.154411 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-credential-socket\") pod \"793b0190-5368-47f1-ae26-e6f736e9e878\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.154470 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-istio-token\" (UniqueName: \"kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-istio-token\") pod \"793b0190-5368-47f1-ae26-e6f736e9e878\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.154506 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/793b0190-5368-47f1-ae26-e6f736e9e878-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793b0190-5368-47f1-ae26-e6f736e9e878-istiod-ca-cert\") pod \"793b0190-5368-47f1-ae26-e6f736e9e878\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.154535 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-data\") pod \"793b0190-5368-47f1-ae26-e6f736e9e878\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.154571 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-envoy\") pod \"793b0190-5368-47f1-ae26-e6f736e9e878\" (UID: \"793b0190-5368-47f1-ae26-e6f736e9e878\") " Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.155289 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bbc35b-d12d-41c4-acd9-6d3a484459a8-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.155313 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8qc6\" (UniqueName: \"kubernetes.io/projected/10bbc35b-d12d-41c4-acd9-6d3a484459a8-kube-api-access-f8qc6\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.156612 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-socket" pod "793b0190-5368-47f1-ae26-e6f736e9e878" (UID: "793b0190-5368-47f1-ae26-e6f736e9e878"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.157432 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793b0190-5368-47f1-ae26-e6f736e9e878-istiod-ca-cert" pod "793b0190-5368-47f1-ae26-e6f736e9e878" (UID: "793b0190-5368-47f1-ae26-e6f736e9e878"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.157686 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-data" pod "793b0190-5368-47f1-ae26-e6f736e9e878" (UID: "793b0190-5368-47f1-ae26-e6f736e9e878"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.157793 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-credential-socket" pod "793b0190-5368-47f1-ae26-e6f736e9e878" (UID: "793b0190-5368-47f1-ae26-e6f736e9e878"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:49.158265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.158052 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-certs" pod "793b0190-5368-47f1-ae26-e6f736e9e878" (UID: "793b0190-5368-47f1-ae26-e6f736e9e878"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:49.164967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.164899 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-envoy" pod "793b0190-5368-47f1-ae26-e6f736e9e878" (UID: "793b0190-5368-47f1-ae26-e6f736e9e878"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:49.165100 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.164977 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-istio-token" pod "793b0190-5368-47f1-ae26-e6f736e9e878" (UID: "793b0190-5368-47f1-ae26-e6f736e9e878"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:49.165278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.165247 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/793b0190-5368-47f1-ae26-e6f736e9e878-istio-podinfo" pod "793b0190-5368-47f1-ae26-e6f736e9e878" (UID: "793b0190-5368-47f1-ae26-e6f736e9e878"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:12:49.167139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.167116 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-kube-api-access-kn47c" pod "793b0190-5368-47f1-ae26-e6f736e9e878" (UID: "793b0190-5368-47f1-ae26-e6f736e9e878"). InnerVolumeSpecName "kube-api-access-kn47c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:49.256949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.256912 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:49.256949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.256941 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:49.256949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.256951 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793b0190-5368-47f1-ae26-e6f736e9e878-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:49.257194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.256960 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:49.257194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.256968 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:49.257194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.256977 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn47c\" (UniqueName: \"kubernetes.io/projected/793b0190-5368-47f1-ae26-e6f736e9e878-kube-api-access-kn47c\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:49.257194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.256987 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:49.257194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.256995 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793b0190-5368-47f1-ae26-e6f736e9e878-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:49.257194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.257003 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793b0190-5368-47f1-ae26-e6f736e9e878-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:12:49.739285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.739248 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" Oct 30 05:12:49.739285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.739273 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4" event={"ID":"10bbc35b-d12d-41c4-acd9-6d3a484459a8","Type":"ContainerDied","Data":"43b3a5a855c997ae7eaf9de65af7cc186cb1bb4dee1722e18d6d7cd15fb46032"} Oct 30 05:12:49.739839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.739329 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43b3a5a855c997ae7eaf9de65af7cc186cb1bb4dee1722e18d6d7cd15fb46032" Oct 30 05:12:49.741865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.741836 6587 generic.go:356] "Generic (PLEG): container finished" podID="793b0190-5368-47f1-ae26-e6f736e9e878" containerID="51ced6ec618ad6680dce6b504f410d5f891154acd39c011019adb6fd27936924" exitCode=0 Oct 30 05:12:49.742000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.741878 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" Oct 30 05:12:49.742000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.741913 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" event={"ID":"793b0190-5368-47f1-ae26-e6f736e9e878","Type":"ContainerDied","Data":"51ced6ec618ad6680dce6b504f410d5f891154acd39c011019adb6fd27936924"} Oct 30 05:12:49.742000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.741965 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt" event={"ID":"793b0190-5368-47f1-ae26-e6f736e9e878","Type":"ContainerDied","Data":"0ceb167894b27ee48ed475edf607d35d647a34fd400e02fd5b7f80f30bf53b6f"} Oct 30 05:12:49.742000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.741998 6587 scope.go:122] "RemoveContainer" containerID="51ced6ec618ad6680dce6b504f410d5f891154acd39c011019adb6fd27936924" Oct 30 05:12:49.765586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.765566 6587 scope.go:122] "RemoveContainer" containerID="37e7d3b9792a464344d3b75dfd735121bd8ef138dfed4e4e7db6d480e2c5e16c" Oct 30 05:12:49.775959 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.775939 6587 scope.go:122] "RemoveContainer" containerID="51ced6ec618ad6680dce6b504f410d5f891154acd39c011019adb6fd27936924" Oct 30 05:12:49.776212 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:12:49.776192 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ced6ec618ad6680dce6b504f410d5f891154acd39c011019adb6fd27936924\": container with ID starting with 51ced6ec618ad6680dce6b504f410d5f891154acd39c011019adb6fd27936924 not found: ID does not exist" containerID="51ced6ec618ad6680dce6b504f410d5f891154acd39c011019adb6fd27936924" Oct 30 05:12:49.776284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.776225 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ced6ec618ad6680dce6b504f410d5f891154acd39c011019adb6fd27936924"} err="failed to get container status \"51ced6ec618ad6680dce6b504f410d5f891154acd39c011019adb6fd27936924\": rpc error: code = NotFound desc = could not find container \"51ced6ec618ad6680dce6b504f410d5f891154acd39c011019adb6fd27936924\": container with ID starting with 51ced6ec618ad6680dce6b504f410d5f891154acd39c011019adb6fd27936924 not found: ID does not exist" Oct 30 05:12:49.776284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.776249 6587 scope.go:122] "RemoveContainer" containerID="37e7d3b9792a464344d3b75dfd735121bd8ef138dfed4e4e7db6d480e2c5e16c" Oct 30 05:12:49.776548 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:12:49.776519 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e7d3b9792a464344d3b75dfd735121bd8ef138dfed4e4e7db6d480e2c5e16c\": container with ID starting with 37e7d3b9792a464344d3b75dfd735121bd8ef138dfed4e4e7db6d480e2c5e16c not found: ID does not exist" containerID="37e7d3b9792a464344d3b75dfd735121bd8ef138dfed4e4e7db6d480e2c5e16c" Oct 30 05:12:49.776655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.776545 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e7d3b9792a464344d3b75dfd735121bd8ef138dfed4e4e7db6d480e2c5e16c"} err="failed to get container status \"37e7d3b9792a464344d3b75dfd735121bd8ef138dfed4e4e7db6d480e2c5e16c\": rpc error: code = NotFound desc = could not find container \"37e7d3b9792a464344d3b75dfd735121bd8ef138dfed4e4e7db6d480e2c5e16c\": container with ID starting with 37e7d3b9792a464344d3b75dfd735121bd8ef138dfed4e4e7db6d480e2c5e16c not found: ID does not exist" Oct 30 05:12:49.816397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.816372 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt"] Oct 30 05:12:49.833595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:49.833572 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt"] Oct 30 05:12:50.089577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:50.089513 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c"] Oct 30 05:12:50.095898 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:50.095859 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce054d4-kk55c"] Oct 30 05:12:51.583084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:51.583050 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" path="/var/lib/kubelet/pods/793b0190-5368-47f1-ae26-e6f736e9e878/volumes" Oct 30 05:12:51.583635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:12:51.583618 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c981fbe5-95c5-4a9c-94da-cfc66d8665fc" path="/var/lib/kubelet/pods/c981fbe5-95c5-4a9c-94da-cfc66d8665fc/volumes" Oct 30 05:12:55.990678 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:12:55.990638 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:13:09.033762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.033710 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8"] Oct 30 05:13:09.034933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.034904 6587 cpu_manager.go:414] "containerMap: removing container" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" Oct 30 05:13:09.034933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.034931 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" Oct 30 05:13:09.035635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.034989 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2a7858bf-8a5c-44d6-ba1d-f37f7e567124" containerName="payload-processing" Oct 30 05:13:09.035635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.034997 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7858bf-8a5c-44d6-ba1d-f37f7e567124" containerName="payload-processing" Oct 30 05:13:09.035635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.035011 6587 cpu_manager.go:414] "containerMap: removing container" podUID="10bbc35b-d12d-41c4-acd9-6d3a484459a8" containerName="revoke-keys" Oct 30 05:13:09.035635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.035019 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10bbc35b-d12d-41c4-acd9-6d3a484459a8" containerName="revoke-keys" Oct 30 05:13:09.035635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.035064 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b3baff95-ae1b-4863-8516-b566fdbdf1b8" containerName="revoke-keys" Oct 30 05:13:09.035635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.035072 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3baff95-ae1b-4863-8516-b566fdbdf1b8" containerName="revoke-keys" Oct 30 05:13:09.035635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.035292 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" Oct 30 05:13:09.035635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.035316 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7858bf-8a5c-44d6-ba1d-f37f7e567124" containerName="payload-processing" Oct 30 05:13:09.035635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.035325 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="10bbc35b-d12d-41c4-acd9-6d3a484459a8" containerName="revoke-keys" Oct 30 05:13:09.035635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.035339 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3baff95-ae1b-4863-8516-b566fdbdf1b8" containerName="revoke-keys" Oct 30 05:13:09.039888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.039862 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:09.056318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.056291 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8"] Oct 30 05:13:09.064575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.064551 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:13:09.064865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.064845 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:13:09.064977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.064895 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:13:09.065493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.065465 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:13:09.106720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.106606 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8\" (UID: \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:09.106922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.106753 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8\" (UID: \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:09.106922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.106896 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk546\" (UniqueName: \"kubernetes.io/projected/f45e9047-555f-47c2-b65f-f920bfc5cf1c-kube-api-access-hk546\") pod \"maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8\" (UID: \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:09.208395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.208331 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8\" (UID: \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:09.208576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.208535 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk546\" (UniqueName: \"kubernetes.io/projected/f45e9047-555f-47c2-b65f-f920bfc5cf1c-kube-api-access-hk546\") pod \"maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8\" (UID: \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:09.208730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.208715 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8\" (UID: \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:09.209176 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.209147 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8\" (UID: \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:09.211183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.211162 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8\" (UID: \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:09.232880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.232853 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk546\" (UniqueName: \"kubernetes.io/projected/f45e9047-555f-47c2-b65f-f920bfc5cf1c-kube-api-access-hk546\") pod \"maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8\" (UID: \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:09.358514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.358476 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:09.671680 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:13:09.671646 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45e9047_555f_47c2_b65f_f920bfc5cf1c.slice/crio-2f53aea8dadf4cf322084d7dfd5d4411878c4bbe1edcdda8fd8b10e44e8da576 WatchSource:0}: Error finding container 2f53aea8dadf4cf322084d7dfd5d4411878c4bbe1edcdda8fd8b10e44e8da576: Status 404 returned error can't find the container with id 2f53aea8dadf4cf322084d7dfd5d4411878c4bbe1edcdda8fd8b10e44e8da576 Oct 30 05:13:09.673824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.673798 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8"] Oct 30 05:13:09.971438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:09.971307 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" event={"ID":"f45e9047-555f-47c2-b65f-f920bfc5cf1c","Type":"ContainerStarted","Data":"2f53aea8dadf4cf322084d7dfd5d4411878c4bbe1edcdda8fd8b10e44e8da576"} Oct 30 05:13:10.980763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:10.980729 6587 generic.go:356] "Generic (PLEG): container finished" podID="f45e9047-555f-47c2-b65f-f920bfc5cf1c" containerID="ab1d4a55e7a0f7bfc14dc709b70e8e68f4a56e09eeac3c75d9dcd8c49011de63" exitCode=0 Oct 30 05:13:10.981220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:10.980797 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" event={"ID":"f45e9047-555f-47c2-b65f-f920bfc5cf1c","Type":"ContainerDied","Data":"ab1d4a55e7a0f7bfc14dc709b70e8e68f4a56e09eeac3c75d9dcd8c49011de63"} Oct 30 05:13:12.170671 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.170644 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:12.265266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.265169 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-xks-ca\") pod \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\" (UID: \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\") " Oct 30 05:13:12.265455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.265275 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45e9047-555f-47c2-b65f-f920bfc5cf1c-kube-api-access-hk546\" (UniqueName: \"kubernetes.io/projected/f45e9047-555f-47c2-b65f-f920bfc5cf1c-kube-api-access-hk546\") pod \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\" (UID: \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\") " Oct 30 05:13:12.265656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.265634 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-openshift-ca\") pod \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\" (UID: \"f45e9047-555f-47c2-b65f-f920bfc5cf1c\") " Oct 30 05:13:12.266378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.266039 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-openshift-ca" pod "f45e9047-555f-47c2-b65f-f920bfc5cf1c" (UID: "f45e9047-555f-47c2-b65f-f920bfc5cf1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:12.268309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.268278 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45e9047-555f-47c2-b65f-f920bfc5cf1c-kube-api-access-hk546" pod "f45e9047-555f-47c2-b65f-f920bfc5cf1c" (UID: "f45e9047-555f-47c2-b65f-f920bfc5cf1c"). InnerVolumeSpecName "kube-api-access-hk546". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:12.283501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.283465 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-xks-ca" pod "f45e9047-555f-47c2-b65f-f920bfc5cf1c" (UID: "f45e9047-555f-47c2-b65f-f920bfc5cf1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:13:12.368022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.367990 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:13:12.368022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.368019 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45e9047-555f-47c2-b65f-f920bfc5cf1c-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:13:12.368022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.368029 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk546\" (UniqueName: \"kubernetes.io/projected/f45e9047-555f-47c2-b65f-f920bfc5cf1c-kube-api-access-hk546\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:13:12.588831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.588796 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk"] Oct 30 05:13:12.590050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.590026 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f45e9047-555f-47c2-b65f-f920bfc5cf1c" containerName="revoke-keys" Oct 30 05:13:12.590158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.590052 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45e9047-555f-47c2-b65f-f920bfc5cf1c" containerName="revoke-keys" Oct 30 05:13:12.590158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.590085 6587 cpu_manager.go:414] "containerMap: removing container" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" Oct 30 05:13:12.590158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.590094 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" Oct 30 05:13:12.590297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.590284 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45e9047-555f-47c2-b65f-f920bfc5cf1c" containerName="revoke-keys" Oct 30 05:13:12.590330 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.590306 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="793b0190-5368-47f1-ae26-e6f736e9e878" containerName="istio-proxy" Oct 30 05:13:12.595015 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.594994 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.617884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.617852 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk"] Oct 30 05:13:12.626407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.624772 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-453511-openshift-default-dockercfg-j7lkc\"" Oct 30 05:13:12.673154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.673122 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-certs\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.673383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.673343 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-istio-token\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.673496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.673473 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkspw\" (UniqueName: \"kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-kube-api-access-dkspw\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.673534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.673515 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-envoy\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.673575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.673537 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548909cc-6196-4b54-9372-e9f61190ddce-istio-podinfo\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.673698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.673679 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-data\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.673752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.673738 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548909cc-6196-4b54-9372-e9f61190ddce-istiod-ca-cert\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.673920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.673900 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-socket\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.673963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.673946 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-credential-socket\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.775533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.775094 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-certs\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.775533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.775256 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-istio-token\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.775533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.775314 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkspw\" (UniqueName: \"kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-kube-api-access-dkspw\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.775533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.775342 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-envoy\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.775533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.775414 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548909cc-6196-4b54-9372-e9f61190ddce-istio-podinfo\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.775533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.775499 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-data\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.776023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.775569 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548909cc-6196-4b54-9372-e9f61190ddce-istiod-ca-cert\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.776023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.775707 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-socket\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.776023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.775745 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-credential-socket\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.776978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.776216 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-credential-socket\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.776978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.776317 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-certs\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.776978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.776692 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-data\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.777724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.777699 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-socket\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.778043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.778017 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548909cc-6196-4b54-9372-e9f61190ddce-istiod-ca-cert\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.795384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.790020 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-envoy\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.795384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.790045 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548909cc-6196-4b54-9372-e9f61190ddce-istio-podinfo\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.856442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.843513 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkspw\" (UniqueName: \"kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-kube-api-access-dkspw\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.856442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.843833 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-istio-token\") pod \"e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:12.922379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:12.913863 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:13.093630 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:13.092957 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:13.093630 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:13.092987 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" event={"ID":"f45e9047-555f-47c2-b65f-f920bfc5cf1c","Type":"ContainerDied","Data":"2f53aea8dadf4cf322084d7dfd5d4411878c4bbe1edcdda8fd8b10e44e8da576"} Oct 30 05:13:13.093630 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:13.093036 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f53aea8dadf4cf322084d7dfd5d4411878c4bbe1edcdda8fd8b10e44e8da576" Oct 30 05:13:13.408419 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:13:13.408349 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548909cc_6196_4b54_9372_e9f61190ddce.slice/crio-67bb309ef212bcaa1e874e2065d0ca61334e2f64b50f1a7a00ac6d6dff415e37 WatchSource:0}: Error finding container 67bb309ef212bcaa1e874e2065d0ca61334e2f64b50f1a7a00ac6d6dff415e37: Status 404 returned error can't find the container with id 67bb309ef212bcaa1e874e2065d0ca61334e2f64b50f1a7a00ac6d6dff415e37 Oct 30 05:13:13.409907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:13.409882 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk"] Oct 30 05:13:13.411215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:13.411187 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:13.411263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:13.411248 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:13.411307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:13.411277 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:14.103572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:14.103530 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" event={"ID":"548909cc-6196-4b54-9372-e9f61190ddce","Type":"ContainerStarted","Data":"145cc54dd3354c34a60099025653385ed8c73ed200d9a158cae9c68ebba6580b"} Oct 30 05:13:14.103774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:14.103622 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" event={"ID":"548909cc-6196-4b54-9372-e9f61190ddce","Type":"ContainerStarted","Data":"67bb309ef212bcaa1e874e2065d0ca61334e2f64b50f1a7a00ac6d6dff415e37"} Oct 30 05:13:14.179915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:14.179856 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" podStartSLOduration=2.179842663 podStartE2EDuration="2.179842663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:14.178316406 +0000 UTC m=+1718.706083193" watchObservedRunningTime="2026-10-30 05:13:14.179842663 +0000 UTC m=+1718.707609448" Oct 30 05:13:14.915057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:14.915022 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:14.922813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:14.922785 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:15.113081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:15.113045 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:15.114056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:15.114031 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:13:18.492702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.492627 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2"] Oct 30 05:13:18.498162 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.498130 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" Oct 30 05:13:18.516868 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.516837 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2"] Oct 30 05:13:18.524477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.524438 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-453511\"" Oct 30 05:13:18.563711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.563677 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t"] Oct 30 05:13:18.569436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.569402 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" Oct 30 05:13:18.587433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.587398 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-453511-dockercfg-dvvlr\"" Oct 30 05:13:18.589222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.588626 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t"] Oct 30 05:13:18.606560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.606527 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472f9495-f7da-43a3-a891-83e3b4894cdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t\" (UID: \"472f9495-f7da-43a3-a891-83e3b4894cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" Oct 30 05:13:18.606748 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.606661 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c88hk\" (UniqueName: \"kubernetes.io/projected/472f9495-f7da-43a3-a891-83e3b4894cdd-kube-api-access-c88hk\") pod \"payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t\" (UID: \"472f9495-f7da-43a3-a891-83e3b4894cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" Oct 30 05:13:18.606825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.606789 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5f2f\" (UniqueName: \"kubernetes.io/projected/e2e647db-b480-43b6-a65c-aaa6352cd512-kube-api-access-t5f2f\") pod \"payload-processing-e2e-inf-b-453511-5d55648d-c4bx2\" (UID: \"e2e647db-b480-43b6-a65c-aaa6352cd512\") " pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" Oct 30 05:13:18.606908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.606891 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e647db-b480-43b6-a65c-aaa6352cd512-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-453511-5d55648d-c4bx2\" (UID: \"e2e647db-b480-43b6-a65c-aaa6352cd512\") " pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" Oct 30 05:13:18.715575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.715531 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5f2f\" (UniqueName: \"kubernetes.io/projected/e2e647db-b480-43b6-a65c-aaa6352cd512-kube-api-access-t5f2f\") pod \"payload-processing-e2e-inf-b-453511-5d55648d-c4bx2\" (UID: \"e2e647db-b480-43b6-a65c-aaa6352cd512\") " pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" Oct 30 05:13:18.732396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.731449 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e647db-b480-43b6-a65c-aaa6352cd512-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-453511-5d55648d-c4bx2\" (UID: \"e2e647db-b480-43b6-a65c-aaa6352cd512\") " pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" Oct 30 05:13:18.732396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.731544 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472f9495-f7da-43a3-a891-83e3b4894cdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t\" (UID: \"472f9495-f7da-43a3-a891-83e3b4894cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" Oct 30 05:13:18.732396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.731707 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c88hk\" (UniqueName: \"kubernetes.io/projected/472f9495-f7da-43a3-a891-83e3b4894cdd-kube-api-access-c88hk\") pod \"payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t\" (UID: \"472f9495-f7da-43a3-a891-83e3b4894cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" Oct 30 05:13:18.733178 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.732923 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e647db-b480-43b6-a65c-aaa6352cd512-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-453511-5d55648d-c4bx2\" (UID: \"e2e647db-b480-43b6-a65c-aaa6352cd512\") " pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" Oct 30 05:13:18.735669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.733674 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472f9495-f7da-43a3-a891-83e3b4894cdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t\" (UID: \"472f9495-f7da-43a3-a891-83e3b4894cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" Oct 30 05:13:18.799175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.783303 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5f2f\" (UniqueName: \"kubernetes.io/projected/e2e647db-b480-43b6-a65c-aaa6352cd512-kube-api-access-t5f2f\") pod \"payload-processing-e2e-inf-b-453511-5d55648d-c4bx2\" (UID: \"e2e647db-b480-43b6-a65c-aaa6352cd512\") " pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" Oct 30 05:13:18.822933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.817033 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" Oct 30 05:13:18.824274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.824245 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c88hk\" (UniqueName: \"kubernetes.io/projected/472f9495-f7da-43a3-a891-83e3b4894cdd-kube-api-access-c88hk\") pod \"payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t\" (UID: \"472f9495-f7da-43a3-a891-83e3b4894cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" Oct 30 05:13:18.898407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:18.894848 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" Oct 30 05:13:19.302900 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:13:19.302828 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e647db_b480_43b6_a65c_aaa6352cd512.slice/crio-34a0efdc41b6ed6942d89276c56afd32987dc2d601976d7eaddb474333667805 WatchSource:0}: Error finding container 34a0efdc41b6ed6942d89276c56afd32987dc2d601976d7eaddb474333667805: Status 404 returned error can't find the container with id 34a0efdc41b6ed6942d89276c56afd32987dc2d601976d7eaddb474333667805 Oct 30 05:13:19.304724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:19.304694 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2"] Oct 30 05:13:19.347762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:19.347599 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t"] Oct 30 05:13:19.601496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:19.598512 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl"] Oct 30 05:13:19.639744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:19.639706 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch"] Oct 30 05:13:20.191381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:20.179228 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" event={"ID":"e2e647db-b480-43b6-a65c-aaa6352cd512","Type":"ContainerStarted","Data":"afba1ab5551d65485e135852974c5a427b9075f653b34bdbb2b54e53d02442e1"} Oct 30 05:13:20.191381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:20.179412 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" event={"ID":"e2e647db-b480-43b6-a65c-aaa6352cd512","Type":"ContainerStarted","Data":"34a0efdc41b6ed6942d89276c56afd32987dc2d601976d7eaddb474333667805"} Oct 30 05:13:20.191381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:20.181694 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" event={"ID":"472f9495-f7da-43a3-a891-83e3b4894cdd","Type":"ContainerStarted","Data":"08470756e09a4ff18166c9f3f3ea996052d7710b13902c9b52049f304009e8b3"} Oct 30 05:13:20.191381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:20.181767 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" event={"ID":"472f9495-f7da-43a3-a891-83e3b4894cdd","Type":"ContainerStarted","Data":"7077de0185ed2484ff782909c49e532eda224bad44fe46d633c0a14a57457ac3"} Oct 30 05:13:20.191381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:20.181823 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" Oct 30 05:13:20.280819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:20.266696 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" podStartSLOduration=2.266675633 podStartE2EDuration="2.266675633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:20.232739144 +0000 UTC m=+1724.760505931" watchObservedRunningTime="2026-10-30 05:13:20.266675633 +0000 UTC m=+1724.794442420" Oct 30 05:13:20.312690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:20.312620 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" podStartSLOduration=2.312595122 podStartE2EDuration="2.312595122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:20.297818359 +0000 UTC m=+1724.825585145" watchObservedRunningTime="2026-10-30 05:13:20.312595122 +0000 UTC m=+1724.840361909" Oct 30 05:13:21.192225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:21.192189 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" Oct 30 05:13:23.045768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:23.045731 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt"] Oct 30 05:13:23.050035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:23.050006 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a43fea-xlkkt"] Oct 30 05:13:23.583191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:23.583146 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5641a03d-3b4b-4372-8dd8-6d508a0e4ec5" path="/var/lib/kubelet/pods/5641a03d-3b4b-4372-8dd8-6d508a0e4ec5/volumes" Oct 30 05:13:24.619480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:24.619437 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" podUID="07ecdfee-0acb-45f5-b05b-0c5da48c4260" containerName="payload-processing" containerID="cri-o://4871878ef909b3425c5c353ad956f6fc515542490381158a74bbfb7249804f6b" gracePeriod=25 Oct 30 05:13:24.640870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:24.640826 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" podUID="30889d31-1669-4b3a-885f-c0ae680dcacf" containerName="payload-pre-processing" containerID="cri-o://9c09efd7c4eaba7ec276a8ef90cf1adc4ff1fca0c941c9bbb3a98e9b4b2f4b5e" gracePeriod=25 Oct 30 05:13:25.988647 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:13:25.988612 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:13:28.530859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.530823 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn"] Oct 30 05:13:28.536762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.536725 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.569663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.569541 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-36fd3c-openshift-default-dockercfg-5l4n9\"" Oct 30 05:13:28.571121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.571094 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn"] Oct 30 05:13:28.692985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.692948 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-socket\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.692985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.692986 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-token\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.693205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.693073 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istiod-ca-cert\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.693205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.693136 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-certs\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.693291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.693233 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-credential-socket\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.693291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.693264 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-podinfo\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.693369 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.693330 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-data\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.693498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.693483 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrxq\" (UniqueName: \"kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-kube-api-access-4jrxq\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.693539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.693519 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-envoy\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.805466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.796492 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrxq\" (UniqueName: \"kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-kube-api-access-4jrxq\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.805466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.796555 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-envoy\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.805466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.796760 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-socket\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.805466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.796786 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-token\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.805466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.796842 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istiod-ca-cert\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.805466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.796897 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-certs\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.805466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.796953 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-credential-socket\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.805466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.796975 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-podinfo\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.805466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.797028 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-data\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.805466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.797657 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-data\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.805466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.798867 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-credential-socket\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.805466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.799103 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-certs\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.806255 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.799345 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-socket\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.806374 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.806332 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istiod-ca-cert\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.806657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.806606 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-envoy\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.846305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.845739 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-podinfo\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.866998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.864006 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-token\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:28.868265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:28.868238 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrxq\" (UniqueName: \"kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-kube-api-access-4jrxq\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:29.168520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:29.168431 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:29.274194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:29.274152 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" podUID="07ecdfee-0acb-45f5-b05b-0c5da48c4260" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:13:29.274622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:29.274587 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" podUID="30889d31-1669-4b3a-885f-c0ae680dcacf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:13:29.488688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:29.488658 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn"] Oct 30 05:13:29.488952 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:13:29.488929 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb57c417_36f4_4a6b_9521_9bcfd52abf8c.slice/crio-f28837ee75ac00f5847cb7fc0f5da6cb8b35e80dcfac436b864fc8814f0e4192 WatchSource:0}: Error finding container f28837ee75ac00f5847cb7fc0f5da6cb8b35e80dcfac436b864fc8814f0e4192: Status 404 returned error can't find the container with id f28837ee75ac00f5847cb7fc0f5da6cb8b35e80dcfac436b864fc8814f0e4192 Oct 30 05:13:29.492096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:29.492064 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:29.492182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:29.492129 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:29.492182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:29.492158 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:30.279746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:30.279712 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" event={"ID":"cb57c417-36f4-4a6b-9521-9bcfd52abf8c","Type":"ContainerStarted","Data":"790730cb8d823c19a885dcfe6b420dfb145711d6756de4efe9584410072d187e"} Oct 30 05:13:30.280175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:30.279763 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" event={"ID":"cb57c417-36f4-4a6b-9521-9bcfd52abf8c","Type":"ContainerStarted","Data":"f28837ee75ac00f5847cb7fc0f5da6cb8b35e80dcfac436b864fc8814f0e4192"} Oct 30 05:13:30.366812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:30.366728 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" podStartSLOduration=2.366707345 podStartE2EDuration="2.366707345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:30.339477288 +0000 UTC m=+1734.867244075" watchObservedRunningTime="2026-10-30 05:13:30.366707345 +0000 UTC m=+1734.894474133" Oct 30 05:13:31.169038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:31.169003 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:31.175243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:31.175217 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:31.194955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:31.194933 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" Oct 30 05:13:31.291149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:31.291116 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:31.292784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:31.292756 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:13:32.202963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:32.202938 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" Oct 30 05:13:32.988157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:32.988075 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj"] Oct 30 05:13:32.994063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:32.994035 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" Oct 30 05:13:33.013434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.013408 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj"] Oct 30 05:13:33.022643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.022601 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:13:33.022799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.022674 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:13:33.022799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.022692 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:13:33.025690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.025669 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:13:33.079726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.079695 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt7wv\" (UniqueName: \"kubernetes.io/projected/7458a4c6-ac23-492d-9256-bc6f75b52474-kube-api-access-wt7wv\") pod \"maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj\" (UID: \"7458a4c6-ac23-492d-9256-bc6f75b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" Oct 30 05:13:33.079933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.079911 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj\" (UID: \"7458a4c6-ac23-492d-9256-bc6f75b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" Oct 30 05:13:33.079999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.079985 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj\" (UID: \"7458a4c6-ac23-492d-9256-bc6f75b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" Oct 30 05:13:33.181755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.181717 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj\" (UID: \"7458a4c6-ac23-492d-9256-bc6f75b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" Oct 30 05:13:33.181979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.181833 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj\" (UID: \"7458a4c6-ac23-492d-9256-bc6f75b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" Oct 30 05:13:33.182051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.182038 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt7wv\" (UniqueName: \"kubernetes.io/projected/7458a4c6-ac23-492d-9256-bc6f75b52474-kube-api-access-wt7wv\") pod \"maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj\" (UID: \"7458a4c6-ac23-492d-9256-bc6f75b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" Oct 30 05:13:33.182659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.182629 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj\" (UID: \"7458a4c6-ac23-492d-9256-bc6f75b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" Oct 30 05:13:33.184479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.184458 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj\" (UID: \"7458a4c6-ac23-492d-9256-bc6f75b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" Oct 30 05:13:33.203395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.203349 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt7wv\" (UniqueName: \"kubernetes.io/projected/7458a4c6-ac23-492d-9256-bc6f75b52474-kube-api-access-wt7wv\") pod \"maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj\" (UID: \"7458a4c6-ac23-492d-9256-bc6f75b52474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" Oct 30 05:13:33.376379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.376239 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" Oct 30 05:13:33.834074 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:13:33.834032 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7458a4c6_ac23_492d_9256_bc6f75b52474.slice/crio-c00b64fe256f0dc60e801c3d4512386f656a7108026bc61d1c2fd3955e0ef96b WatchSource:0}: Error finding container c00b64fe256f0dc60e801c3d4512386f656a7108026bc61d1c2fd3955e0ef96b: Status 404 returned error can't find the container with id c00b64fe256f0dc60e801c3d4512386f656a7108026bc61d1c2fd3955e0ef96b Oct 30 05:13:33.838059 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:33.838029 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj"] Oct 30 05:13:34.531492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:34.531397 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" event={"ID":"7458a4c6-ac23-492d-9256-bc6f75b52474","Type":"ContainerStarted","Data":"f78f3c2d2abc54711b6bc1c9ead1a801bb6dada84c1979caa44e454e945d78f7"} Oct 30 05:13:34.531492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:34.531476 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" event={"ID":"7458a4c6-ac23-492d-9256-bc6f75b52474","Type":"ContainerStarted","Data":"c00b64fe256f0dc60e801c3d4512386f656a7108026bc61d1c2fd3955e0ef96b"} Oct 30 05:13:34.575048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:34.574990 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" podStartSLOduration=2.574975746 podStartE2EDuration="2.574975746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:34.572930754 +0000 UTC m=+1739.100697540" watchObservedRunningTime="2026-10-30 05:13:34.574975746 +0000 UTC m=+1739.102742531" Oct 30 05:13:34.765328 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:34.765291 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf"] Oct 30 05:13:34.770314 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:34.770292 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" Oct 30 05:13:34.787194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:34.787132 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf"] Oct 30 05:13:34.864618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:34.864581 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2dth\" (UniqueName: \"kubernetes.io/projected/6dc48b09-eba2-4237-be66-16087f00866e-kube-api-access-q2dth\") pod \"payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf\" (UID: \"6dc48b09-eba2-4237-be66-16087f00866e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" Oct 30 05:13:34.864809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:34.864748 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc48b09-eba2-4237-be66-16087f00866e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf\" (UID: \"6dc48b09-eba2-4237-be66-16087f00866e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" Oct 30 05:13:34.917428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:34.904578 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-36fd3c\"" Oct 30 05:13:34.974397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:34.973554 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc48b09-eba2-4237-be66-16087f00866e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf\" (UID: \"6dc48b09-eba2-4237-be66-16087f00866e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" Oct 30 05:13:34.974397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:34.974037 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2dth\" (UniqueName: \"kubernetes.io/projected/6dc48b09-eba2-4237-be66-16087f00866e-kube-api-access-q2dth\") pod \"payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf\" (UID: \"6dc48b09-eba2-4237-be66-16087f00866e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" Oct 30 05:13:34.976352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:34.975469 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc48b09-eba2-4237-be66-16087f00866e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf\" (UID: \"6dc48b09-eba2-4237-be66-16087f00866e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" Oct 30 05:13:35.058499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:35.057909 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2dth\" (UniqueName: \"kubernetes.io/projected/6dc48b09-eba2-4237-be66-16087f00866e-kube-api-access-q2dth\") pod \"payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf\" (UID: \"6dc48b09-eba2-4237-be66-16087f00866e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" Oct 30 05:13:35.092383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:35.088691 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" Oct 30 05:13:35.544486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:35.544455 6587 generic.go:356] "Generic (PLEG): container finished" podID="7458a4c6-ac23-492d-9256-bc6f75b52474" containerID="f78f3c2d2abc54711b6bc1c9ead1a801bb6dada84c1979caa44e454e945d78f7" exitCode=0 Oct 30 05:13:35.544892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:35.544537 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" event={"ID":"7458a4c6-ac23-492d-9256-bc6f75b52474","Type":"ContainerDied","Data":"f78f3c2d2abc54711b6bc1c9ead1a801bb6dada84c1979caa44e454e945d78f7"} Oct 30 05:13:35.605143 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:13:35.605079 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc48b09_eba2_4237_be66_16087f00866e.slice/crio-326f0d37ece95b88def8616c2fdfd376339a419c40a23cd36a6747661a0a0717 WatchSource:0}: Error finding container 326f0d37ece95b88def8616c2fdfd376339a419c40a23cd36a6747661a0a0717: Status 404 returned error can't find the container with id 326f0d37ece95b88def8616c2fdfd376339a419c40a23cd36a6747661a0a0717 Oct 30 05:13:35.608856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:35.608829 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf"] Oct 30 05:13:36.573384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:36.570716 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" event={"ID":"6dc48b09-eba2-4237-be66-16087f00866e","Type":"ContainerStarted","Data":"b93e01627e545b554ce94ce2825de1078d9c5ea22d1e77ed00120dd3fef55934"} Oct 30 05:13:36.573384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:36.570788 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" event={"ID":"6dc48b09-eba2-4237-be66-16087f00866e","Type":"ContainerStarted","Data":"326f0d37ece95b88def8616c2fdfd376339a419c40a23cd36a6747661a0a0717"} Oct 30 05:13:36.573384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:36.570952 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" Oct 30 05:13:36.615707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:36.615638 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" podStartSLOduration=2.615617228 podStartE2EDuration="2.615617228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:36.613917305 +0000 UTC m=+1741.141684088" watchObservedRunningTime="2026-10-30 05:13:36.615617228 +0000 UTC m=+1741.143384015" Oct 30 05:13:36.830860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:36.830787 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" Oct 30 05:13:36.943934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:36.943890 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-openshift-ca\") pod \"7458a4c6-ac23-492d-9256-bc6f75b52474\" (UID: \"7458a4c6-ac23-492d-9256-bc6f75b52474\") " Oct 30 05:13:36.944140 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:36.943983 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-xks-ca\") pod \"7458a4c6-ac23-492d-9256-bc6f75b52474\" (UID: \"7458a4c6-ac23-492d-9256-bc6f75b52474\") " Oct 30 05:13:36.944236 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:36.944219 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7458a4c6-ac23-492d-9256-bc6f75b52474-kube-api-access-wt7wv\" (UniqueName: \"kubernetes.io/projected/7458a4c6-ac23-492d-9256-bc6f75b52474-kube-api-access-wt7wv\") pod \"7458a4c6-ac23-492d-9256-bc6f75b52474\" (UID: \"7458a4c6-ac23-492d-9256-bc6f75b52474\") " Oct 30 05:13:36.945352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:36.945320 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-openshift-ca" pod "7458a4c6-ac23-492d-9256-bc6f75b52474" (UID: "7458a4c6-ac23-492d-9256-bc6f75b52474"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:36.953982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:36.951410 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7458a4c6-ac23-492d-9256-bc6f75b52474-kube-api-access-wt7wv" pod "7458a4c6-ac23-492d-9256-bc6f75b52474" (UID: "7458a4c6-ac23-492d-9256-bc6f75b52474"). InnerVolumeSpecName "kube-api-access-wt7wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:36.953982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:36.951547 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-xks-ca" pod "7458a4c6-ac23-492d-9256-bc6f75b52474" (UID: "7458a4c6-ac23-492d-9256-bc6f75b52474"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:13:37.062383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:37.051830 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:13:37.062383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:37.051874 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7458a4c6-ac23-492d-9256-bc6f75b52474-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:13:37.062383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:37.051890 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt7wv\" (UniqueName: \"kubernetes.io/projected/7458a4c6-ac23-492d-9256-bc6f75b52474-kube-api-access-wt7wv\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:13:37.584820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:37.584792 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" Oct 30 05:13:37.591096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:37.591062 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj" event={"ID":"7458a4c6-ac23-492d-9256-bc6f75b52474","Type":"ContainerDied","Data":"c00b64fe256f0dc60e801c3d4512386f656a7108026bc61d1c2fd3955e0ef96b"} Oct 30 05:13:37.591251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:37.591120 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c00b64fe256f0dc60e801c3d4512386f656a7108026bc61d1c2fd3955e0ef96b" Oct 30 05:13:38.956378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:38.956125 6587 scope.go:122] "RemoveContainer" containerID="cd17e506d6545e4aa6295e4bf0157f90b23814be0b69c4d16755a2cbebc15d08" Oct 30 05:13:39.274211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:39.274171 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" podUID="30889d31-1669-4b3a-885f-c0ae680dcacf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:13:39.275401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:39.275324 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" podUID="07ecdfee-0acb-45f5-b05b-0c5da48c4260" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:13:44.801493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:44.801465 6587 generic.go:356] "Generic (PLEG): container finished" podID="07ecdfee-0acb-45f5-b05b-0c5da48c4260" containerID="4871878ef909b3425c5c353ad956f6fc515542490381158a74bbfb7249804f6b" exitCode=0 Oct 30 05:13:44.801889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:44.801547 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" event={"ID":"07ecdfee-0acb-45f5-b05b-0c5da48c4260","Type":"ContainerDied","Data":"4871878ef909b3425c5c353ad956f6fc515542490381158a74bbfb7249804f6b"} Oct 30 05:13:45.282171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.282037 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" Oct 30 05:13:45.368021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.317000 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ecdfee-0acb-45f5-b05b-0c5da48c4260-kube-api-access-lnfs9\" (UniqueName: \"kubernetes.io/projected/07ecdfee-0acb-45f5-b05b-0c5da48c4260-kube-api-access-lnfs9\") pod \"07ecdfee-0acb-45f5-b05b-0c5da48c4260\" (UID: \"07ecdfee-0acb-45f5-b05b-0c5da48c4260\") " Oct 30 05:13:45.381296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.381264 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ecdfee-0acb-45f5-b05b-0c5da48c4260-kube-api-access-lnfs9" pod "07ecdfee-0acb-45f5-b05b-0c5da48c4260" (UID: "07ecdfee-0acb-45f5-b05b-0c5da48c4260"). InnerVolumeSpecName "kube-api-access-lnfs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:45.399798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.399773 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" Oct 30 05:13:45.449128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.449090 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ecdfee-0acb-45f5-b05b-0c5da48c4260-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ecdfee-0acb-45f5-b05b-0c5da48c4260-plugins-config-volume\") pod \"07ecdfee-0acb-45f5-b05b-0c5da48c4260\" (UID: \"07ecdfee-0acb-45f5-b05b-0c5da48c4260\") " Oct 30 05:13:45.452416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.449998 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnfs9\" (UniqueName: \"kubernetes.io/projected/07ecdfee-0acb-45f5-b05b-0c5da48c4260-kube-api-access-lnfs9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:13:45.452416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.450425 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ecdfee-0acb-45f5-b05b-0c5da48c4260-plugins-config-volume" pod "07ecdfee-0acb-45f5-b05b-0c5da48c4260" (UID: "07ecdfee-0acb-45f5-b05b-0c5da48c4260"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:45.553006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.552966 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30889d31-1669-4b3a-885f-c0ae680dcacf-kube-api-access-9msr9\" (UniqueName: \"kubernetes.io/projected/30889d31-1669-4b3a-885f-c0ae680dcacf-kube-api-access-9msr9\") pod \"30889d31-1669-4b3a-885f-c0ae680dcacf\" (UID: \"30889d31-1669-4b3a-885f-c0ae680dcacf\") " Oct 30 05:13:45.553301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.553048 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30889d31-1669-4b3a-885f-c0ae680dcacf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30889d31-1669-4b3a-885f-c0ae680dcacf-plugins-config-volume\") pod \"30889d31-1669-4b3a-885f-c0ae680dcacf\" (UID: \"30889d31-1669-4b3a-885f-c0ae680dcacf\") " Oct 30 05:13:45.553656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.553625 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30889d31-1669-4b3a-885f-c0ae680dcacf-plugins-config-volume" pod "30889d31-1669-4b3a-885f-c0ae680dcacf" (UID: "30889d31-1669-4b3a-885f-c0ae680dcacf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:45.556064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.555794 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30889d31-1669-4b3a-885f-c0ae680dcacf-kube-api-access-9msr9" pod "30889d31-1669-4b3a-885f-c0ae680dcacf" (UID: "30889d31-1669-4b3a-885f-c0ae680dcacf"). InnerVolumeSpecName "kube-api-access-9msr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:45.556064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.556001 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ecdfee-0acb-45f5-b05b-0c5da48c4260-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:13:45.556064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.556028 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9msr9\" (UniqueName: \"kubernetes.io/projected/30889d31-1669-4b3a-885f-c0ae680dcacf-kube-api-access-9msr9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:13:45.556064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.556043 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30889d31-1669-4b3a-885f-c0ae680dcacf-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:13:45.812903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.812808 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" event={"ID":"07ecdfee-0acb-45f5-b05b-0c5da48c4260","Type":"ContainerDied","Data":"2d8a6c285985ab3f8f8e04af3a6fcacbbaec209dd68962be7bf2bfb4fdb26c59"} Oct 30 05:13:45.812903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.812829 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl" Oct 30 05:13:45.812903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.812882 6587 scope.go:122] "RemoveContainer" containerID="4871878ef909b3425c5c353ad956f6fc515542490381158a74bbfb7249804f6b" Oct 30 05:13:45.827141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.827103 6587 generic.go:356] "Generic (PLEG): container finished" podID="30889d31-1669-4b3a-885f-c0ae680dcacf" containerID="9c09efd7c4eaba7ec276a8ef90cf1adc4ff1fca0c941c9bbb3a98e9b4b2f4b5e" exitCode=0 Oct 30 05:13:45.827280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.827246 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" event={"ID":"30889d31-1669-4b3a-885f-c0ae680dcacf","Type":"ContainerDied","Data":"9c09efd7c4eaba7ec276a8ef90cf1adc4ff1fca0c941c9bbb3a98e9b4b2f4b5e"} Oct 30 05:13:45.827352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.827293 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" event={"ID":"30889d31-1669-4b3a-885f-c0ae680dcacf","Type":"ContainerDied","Data":"4a7982c6ea864cfc5f777c7005d90deda9a27306602317d864929c369b2845e2"} Oct 30 05:13:45.827464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.827387 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch" Oct 30 05:13:45.838505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.838483 6587 scope.go:122] "RemoveContainer" containerID="9c09efd7c4eaba7ec276a8ef90cf1adc4ff1fca0c941c9bbb3a98e9b4b2f4b5e" Oct 30 05:13:45.852489 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.852465 6587 scope.go:122] "RemoveContainer" containerID="9c09efd7c4eaba7ec276a8ef90cf1adc4ff1fca0c941c9bbb3a98e9b4b2f4b5e" Oct 30 05:13:45.852953 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:13:45.852925 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c09efd7c4eaba7ec276a8ef90cf1adc4ff1fca0c941c9bbb3a98e9b4b2f4b5e\": container with ID starting with 9c09efd7c4eaba7ec276a8ef90cf1adc4ff1fca0c941c9bbb3a98e9b4b2f4b5e not found: ID does not exist" containerID="9c09efd7c4eaba7ec276a8ef90cf1adc4ff1fca0c941c9bbb3a98e9b4b2f4b5e" Oct 30 05:13:45.853102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.853080 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c09efd7c4eaba7ec276a8ef90cf1adc4ff1fca0c941c9bbb3a98e9b4b2f4b5e"} err="failed to get container status \"9c09efd7c4eaba7ec276a8ef90cf1adc4ff1fca0c941c9bbb3a98e9b4b2f4b5e\": rpc error: code = NotFound desc = could not find container \"9c09efd7c4eaba7ec276a8ef90cf1adc4ff1fca0c941c9bbb3a98e9b4b2f4b5e\": container with ID starting with 9c09efd7c4eaba7ec276a8ef90cf1adc4ff1fca0c941c9bbb3a98e9b4b2f4b5e not found: ID does not exist" Oct 30 05:13:45.891312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.891281 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl"] Oct 30 05:13:45.904517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.904490 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl"] Oct 30 05:13:45.933304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.933232 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch"] Oct 30 05:13:45.939596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:45.939563 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch"] Oct 30 05:13:47.586610 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:47.586570 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ecdfee-0acb-45f5-b05b-0c5da48c4260" path="/var/lib/kubelet/pods/07ecdfee-0acb-45f5-b05b-0c5da48c4260/volumes" Oct 30 05:13:47.587230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:47.587207 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30889d31-1669-4b3a-885f-c0ae680dcacf" path="/var/lib/kubelet/pods/30889d31-1669-4b3a-885f-c0ae680dcacf/volumes" Oct 30 05:13:47.589081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:47.589062 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" Oct 30 05:13:50.504526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:50.504481 6587 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:13:50.520440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:50.504552 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:50.767716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:50.767639 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n"] Oct 30 05:13:50.776826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:50.776784 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d55bd-x9b9n"] Oct 30 05:13:50.891890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:50.891860 6587 generic.go:356] "Generic (PLEG): container finished" podID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerID="d33226465478fffc519b132e3e3baa41f9ae6b147d6c9a5af221d3ab12b9824f" exitCode=0 Oct 30 05:13:50.892085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:50.891944 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" event={"ID":"2ff795b5-eea0-47c6-acba-5aea88e9fea4","Type":"ContainerDied","Data":"d33226465478fffc519b132e3e3baa41f9ae6b147d6c9a5af221d3ab12b9824f"} Oct 30 05:13:50.892632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:50.892607 6587 scope.go:122] "RemoveContainer" containerID="d33226465478fffc519b132e3e3baa41f9ae6b147d6c9a5af221d3ab12b9824f" Oct 30 05:13:50.894899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:50.894871 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:50.894993 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:50.894938 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:50.894993 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:50.894968 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:51.583499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.583463 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555589a4-8974-4af2-a29a-c9b77a664020" path="/var/lib/kubelet/pods/555589a4-8974-4af2-a29a-c9b77a664020/volumes" Oct 30 05:13:51.621763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.621732 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz"] Oct 30 05:13:51.622554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.622537 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7458a4c6-ac23-492d-9256-bc6f75b52474" containerName="revoke-keys" Oct 30 05:13:51.622554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.622554 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7458a4c6-ac23-492d-9256-bc6f75b52474" containerName="revoke-keys" Oct 30 05:13:51.622680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.622594 6587 cpu_manager.go:414] "containerMap: removing container" podUID="07ecdfee-0acb-45f5-b05b-0c5da48c4260" containerName="payload-processing" Oct 30 05:13:51.622680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.622600 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ecdfee-0acb-45f5-b05b-0c5da48c4260" containerName="payload-processing" Oct 30 05:13:51.622680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.622617 6587 cpu_manager.go:414] "containerMap: removing container" podUID="30889d31-1669-4b3a-885f-c0ae680dcacf" containerName="payload-pre-processing" Oct 30 05:13:51.622680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.622622 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30889d31-1669-4b3a-885f-c0ae680dcacf" containerName="payload-pre-processing" Oct 30 05:13:51.622809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.622718 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7458a4c6-ac23-492d-9256-bc6f75b52474" containerName="revoke-keys" Oct 30 05:13:51.622809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.622735 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ecdfee-0acb-45f5-b05b-0c5da48c4260" containerName="payload-processing" Oct 30 05:13:51.622809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.622743 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="30889d31-1669-4b3a-885f-c0ae680dcacf" containerName="payload-pre-processing" Oct 30 05:13:51.657791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.641103 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.671460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.671428 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz"] Oct 30 05:13:51.687284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.687253 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-data\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.687464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.687304 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58ec340-b565-4048-83a2-342485b1a68d-istio-podinfo\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.687464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.687370 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.687464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.687434 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-istio-token\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.687648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.687626 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-credential-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.692058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.687739 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-envoy\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.692058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.687783 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-certs\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.692058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.687850 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twnff\" (UniqueName: \"kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-kube-api-access-twnff\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.692058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.687981 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58ec340-b565-4048-83a2-342485b1a68d-istiod-ca-cert\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.790189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.790139 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58ec340-b565-4048-83a2-342485b1a68d-istiod-ca-cert\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.790437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.790257 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-data\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.790437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.790281 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58ec340-b565-4048-83a2-342485b1a68d-istio-podinfo\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.790437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.790323 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.790437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.790393 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-istio-token\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.790656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.790482 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-credential-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.790656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.790570 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-envoy\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.790656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.790601 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-certs\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.790803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.790659 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twnff\" (UniqueName: \"kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-kube-api-access-twnff\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.792134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.791567 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-credential-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.792134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.791569 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.792343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.792297 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-certs\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.792696 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.792672 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-data\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.793225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.793112 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58ec340-b565-4048-83a2-342485b1a68d-istiod-ca-cert\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.794504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.794481 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-envoy\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.794656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.794637 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58ec340-b565-4048-83a2-342485b1a68d-istio-podinfo\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.812256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.812230 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-istio-token\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.813067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.813044 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twnff\" (UniqueName: \"kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-kube-api-access-twnff\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:51.902166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.902085 6587 generic.go:356] "Generic (PLEG): container finished" podID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerID="b34cc9f6e511e072b08ead79fd4c498b10b19a27db5226970364e2e6e7d8af55" exitCode=0 Oct 30 05:13:51.902323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.902165 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" event={"ID":"a2fb6f3a-1091-4baa-be60-aab470fa8e84","Type":"ContainerDied","Data":"b34cc9f6e511e072b08ead79fd4c498b10b19a27db5226970364e2e6e7d8af55"} Oct 30 05:13:51.902776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.902758 6587 scope.go:122] "RemoveContainer" containerID="b34cc9f6e511e072b08ead79fd4c498b10b19a27db5226970364e2e6e7d8af55" Oct 30 05:13:51.904412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.904387 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" event={"ID":"2ff795b5-eea0-47c6-acba-5aea88e9fea4","Type":"ContainerStarted","Data":"98c167779fff28559d2cd8c8729bf50ea395c93b3b5ed7d4e58f502caf1b21dd"} Oct 30 05:13:51.904831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.904801 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:51.904942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.904887 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:51.904942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:51.904937 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:52.012889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:52.012846 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:52.196209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:52.196070 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:13:52.196410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:52.196318 6587 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 05:13:52.196497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:52.196409 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 05:13:52.461566 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:13:52.460498 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58ec340_b565_4048_83a2_342485b1a68d.slice/crio-751e4cd00192644b3729e77f52045edb4491f04cfd03d2847db06d80438cd63b WatchSource:0}: Error finding container 751e4cd00192644b3729e77f52045edb4491f04cfd03d2847db06d80438cd63b: Status 404 returned error can't find the container with id 751e4cd00192644b3729e77f52045edb4491f04cfd03d2847db06d80438cd63b Oct 30 05:13:52.462752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:52.462726 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz"] Oct 30 05:13:52.465014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:52.464978 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:52.465121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:52.465067 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:52.465121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:52.465109 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:13:52.934523 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:52.934482 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" event={"ID":"f58ec340-b565-4048-83a2-342485b1a68d","Type":"ContainerStarted","Data":"3e471749cc6fa7236e4de7139ed84a4597e45a8392020a50c3a5c59deb660bdc"} Oct 30 05:13:52.935149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:52.935117 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" event={"ID":"f58ec340-b565-4048-83a2-342485b1a68d","Type":"ContainerStarted","Data":"751e4cd00192644b3729e77f52045edb4491f04cfd03d2847db06d80438cd63b"} Oct 30 05:13:52.945347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:52.945317 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" event={"ID":"a2fb6f3a-1091-4baa-be60-aab470fa8e84","Type":"ContainerStarted","Data":"a3ba2b84b60b972d5e22c7914e643afb7acd4e91f3f8fa2a8e6dc89ac0208995"} Oct 30 05:13:53.002433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:53.001211 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" podStartSLOduration=2.001188268 podStartE2EDuration="2.001188268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:52.989466625 +0000 UTC m=+1757.517233412" watchObservedRunningTime="2026-10-30 05:13:53.001188268 +0000 UTC m=+1757.528955055" Oct 30 05:13:53.013905 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:53.013861 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:13:53.023391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:53.022488 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 05:13:53.023391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:53.022552 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" podUID="f58ec340-b565-4048-83a2-342485b1a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 05:13:53.196893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:53.196798 6587 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 05:13:53.197452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:53.197417 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 05:13:53.509192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:53.509048 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:13:53.510055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:53.510036 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:13:53.510122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:53.510084 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 05:13:53.796403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:53.796311 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:13:54.013273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:54.013220 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 05:13:54.013668 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:54.013289 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" podUID="f58ec340-b565-4048-83a2-342485b1a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 05:13:54.224392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:54.215444 6587 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 05:13:54.224392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:54.215509 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 05:13:54.609382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:54.602627 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:13:54.609382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:54.602682 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 05:13:55.163613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.158929 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 05:13:55.337567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.332952 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" podUID="f58ec340-b565-4048-83a2-342485b1a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 05:13:55.337567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.332220 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:55.337567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.285410 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:55.337567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.202849 6587 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 05:13:55.337567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.337200 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 05:13:55.562445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.514312 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:13:55.562445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.514418 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 05:13:55.562445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.522787 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8"] Oct 30 05:13:55.630270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.623249 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:55.665623 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.664511 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:55.878121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.878003 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:55.939114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.918728 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:13:55.982802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:55.966793 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:56.204011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.203881 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8"] Oct 30 05:13:56.246533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.246478 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 05:13:56.280382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.278788 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" podUID="f58ec340-b565-4048-83a2-342485b1a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 05:13:56.280382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.273615 6587 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 05:13:56.280382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.279252 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 05:13:56.287048 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:13:56.285612 6587 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8\" is forbidden: User \"system:node:ip-10-0-134-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-134-105.ec2.internal' and this object" podUID="f45e9047-555f-47c2-b65f-f920bfc5cf1c" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ac96f3-6ngh8" Oct 30 05:13:56.338589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.338543 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:56.363846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.363800 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:13:56.364462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.364437 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:13:56.364654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.364637 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:13:56.406241 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.401705 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78"] Oct 30 05:13:56.406241 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.401945 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" containerID="cri-o://f380abee44cda86058a7992ea1f698fb29eac236235ea2bb5a6bd997bc30cd67" gracePeriod=30 Oct 30 05:13:56.511852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.511756 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:13:56.512087 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.512057 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 05:13:56.586857 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:13:56.586820 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:13:56.669489 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.669445 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:56.670146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.669765 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:13:56.683387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.670351 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:13:56.720381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.711485 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:13:56.720381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.682565 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:56.898389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.889741 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:56.926617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:56.899032 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:13:57.043801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.043763 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 05:13:57.066944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.066898 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" podUID="f58ec340-b565-4048-83a2-342485b1a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 05:13:57.210972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.208706 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:57.213352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.208985 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:57.214843 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.214821 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:13:57.215034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.215019 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:13:57.218435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.218173 6587 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 05:13:57.218435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.218225 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 05:13:57.242821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.242596 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:57.280686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.280559 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:13:57.389081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.389033 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:57.393556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.392653 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:13:57.544859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.541411 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:57.544859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.544237 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:13:57.544859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.544284 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 05:13:57.563910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.563880 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:13:57.602484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.588449 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:57.610652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.610616 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45e9047-555f-47c2-b65f-f920bfc5cf1c" path="/var/lib/kubelet/pods/f45e9047-555f-47c2-b65f-f920bfc5cf1c/volumes" Oct 30 05:13:57.730392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.727977 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:57.730392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:57.729290 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:58.039397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.038503 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 05:13:58.039397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.038568 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" podUID="f58ec340-b565-4048-83a2-342485b1a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 05:13:58.143384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.139214 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:13:58.237409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.213188 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:13:58.242974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.217541 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:58.288377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.272720 6587 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 05:13:58.288377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.272790 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 05:13:58.397595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.397542 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:13:58.510486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.510451 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:13:58.513381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.511949 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:13:58.513613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.513585 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 05:13:58.520988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.520963 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:13:58.594380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.591125 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:58.624394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.598947 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:13:58.754132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.754018 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:13:58.811899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.811868 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:13:58.890383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:58.877340 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:13:59.013221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:59.013131 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 05:13:59.013221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:59.013185 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" podUID="f58ec340-b565-4048-83a2-342485b1a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 05:13:59.019636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:59.019608 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:13:59.400510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:59.400458 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:13:59.483378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:59.483194 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:13:59.488104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:59.484960 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:13:59.507384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:59.488762 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:13:59.533393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:59.529975 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:13:59.533393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:59.530035 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 05:13:59.729941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:59.729830 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/health\": context deadline exceeded" Oct 30 05:13:59.935041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:13:59.934993 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:14:00.016498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:00.016382 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 05:14:00.016812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:00.016753 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" podUID="f58ec340-b565-4048-83a2-342485b1a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 05:14:00.123393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:00.102595 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:14:00.373198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:00.372675 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:14:00.432693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:00.432634 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:14:00.519508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:00.510299 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:14:00.546624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:00.546581 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:14:00.546936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:00.546890 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 05:14:00.579159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:00.579130 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:14:00.583125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:00.583091 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:14:00.706183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:00.706093 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:14:01.057518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:01.054521 6587 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 05:14:01.057518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:01.054597 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" podUID="f58ec340-b565-4048-83a2-342485b1a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 05:14:01.515039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:01.515001 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:14:01.552048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:01.552018 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:14:01.554367 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:14:01.554320 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff505e8_bee4_4f87_93e7_113497e13649.slice/crio-conmon-f380abee44cda86058a7992ea1f698fb29eac236235ea2bb5a6bd997bc30cd67.scope\": RecentStats: unable to find data in memory cache]" Oct 30 05:14:01.629747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:01.629708 6587 generic.go:356] "Generic (PLEG): container finished" podID="6ff505e8-bee4-4f87-93e7-113497e13649" containerID="f380abee44cda86058a7992ea1f698fb29eac236235ea2bb5a6bd997bc30cd67" exitCode=0 Oct 30 05:14:01.630054 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:01.629788 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" event={"ID":"6ff505e8-bee4-4f87-93e7-113497e13649","Type":"ContainerDied","Data":"f380abee44cda86058a7992ea1f698fb29eac236235ea2bb5a6bd997bc30cd67"} Oct 30 05:14:01.630169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:01.630103 6587 scope.go:122] "RemoveContainer" containerID="93a935b2fdbd1e32c26a7cf0c6d6b6e26ff57dfe72f43e7aba3fcfa9474d928f" Oct 30 05:14:02.034218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.033858 6587 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 05:14:02.034218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.033916 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 05:14:02.095505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.094837 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:14:02.096845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.096775 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:14:02.108407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.099079 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:14:02.234793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.234716 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 05:14:02.374667 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.374632 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-certs\") pod \"6ff505e8-bee4-4f87-93e7-113497e13649\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " Oct 30 05:14:02.374667 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.374670 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-envoy\") pod \"6ff505e8-bee4-4f87-93e7-113497e13649\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " Oct 30 05:14:02.374922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.374722 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-istio-token\" (UniqueName: \"kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-istio-token\") pod \"6ff505e8-bee4-4f87-93e7-113497e13649\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " Oct 30 05:14:02.374922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.374747 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff505e8-bee4-4f87-93e7-113497e13649-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff505e8-bee4-4f87-93e7-113497e13649-istiod-ca-cert\") pod \"6ff505e8-bee4-4f87-93e7-113497e13649\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " Oct 30 05:14:02.374922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.374796 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-credential-socket\") pod \"6ff505e8-bee4-4f87-93e7-113497e13649\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " Oct 30 05:14:02.374922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.374816 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-socket\") pod \"6ff505e8-bee4-4f87-93e7-113497e13649\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " Oct 30 05:14:02.374922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.374845 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ff505e8-bee4-4f87-93e7-113497e13649-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff505e8-bee4-4f87-93e7-113497e13649-istio-podinfo\") pod \"6ff505e8-bee4-4f87-93e7-113497e13649\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " Oct 30 05:14:02.374922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.374915 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-kube-api-access-7zt4r\" (UniqueName: \"kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-kube-api-access-7zt4r\") pod \"6ff505e8-bee4-4f87-93e7-113497e13649\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " Oct 30 05:14:02.375344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.374996 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-data\") pod \"6ff505e8-bee4-4f87-93e7-113497e13649\" (UID: \"6ff505e8-bee4-4f87-93e7-113497e13649\") " Oct 30 05:14:02.375344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.374994 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-certs" pod "6ff505e8-bee4-4f87-93e7-113497e13649" (UID: "6ff505e8-bee4-4f87-93e7-113497e13649"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:02.375344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.375189 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-socket" pod "6ff505e8-bee4-4f87-93e7-113497e13649" (UID: "6ff505e8-bee4-4f87-93e7-113497e13649"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:02.375344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.375209 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-credential-socket" pod "6ff505e8-bee4-4f87-93e7-113497e13649" (UID: "6ff505e8-bee4-4f87-93e7-113497e13649"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:02.375588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.375517 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-data" pod "6ff505e8-bee4-4f87-93e7-113497e13649" (UID: "6ff505e8-bee4-4f87-93e7-113497e13649"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:02.375588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.375445 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff505e8-bee4-4f87-93e7-113497e13649-istiod-ca-cert" pod "6ff505e8-bee4-4f87-93e7-113497e13649" (UID: "6ff505e8-bee4-4f87-93e7-113497e13649"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:02.375893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.375869 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:02.375989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.375899 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff505e8-bee4-4f87-93e7-113497e13649-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:02.375989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.375914 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:02.375989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.375928 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:02.375989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.375943 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:02.377196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.377164 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-envoy" pod "6ff505e8-bee4-4f87-93e7-113497e13649" (UID: "6ff505e8-bee4-4f87-93e7-113497e13649"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:02.377196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.377169 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ff505e8-bee4-4f87-93e7-113497e13649-istio-podinfo" pod "6ff505e8-bee4-4f87-93e7-113497e13649" (UID: "6ff505e8-bee4-4f87-93e7-113497e13649"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:14:02.377376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.377232 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-istio-token" pod "6ff505e8-bee4-4f87-93e7-113497e13649" (UID: "6ff505e8-bee4-4f87-93e7-113497e13649"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:02.377376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.377288 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-kube-api-access-7zt4r" pod "6ff505e8-bee4-4f87-93e7-113497e13649" (UID: "6ff505e8-bee4-4f87-93e7-113497e13649"). InnerVolumeSpecName "kube-api-access-7zt4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:02.477532 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.477499 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff505e8-bee4-4f87-93e7-113497e13649-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:02.477532 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.477529 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:02.477532 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.477538 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff505e8-bee4-4f87-93e7-113497e13649-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:02.477773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.477547 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zt4r\" (UniqueName: \"kubernetes.io/projected/6ff505e8-bee4-4f87-93e7-113497e13649-kube-api-access-7zt4r\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:02.653351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.653315 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" Oct 30 05:14:02.653351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.653326 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78" event={"ID":"6ff505e8-bee4-4f87-93e7-113497e13649","Type":"ContainerDied","Data":"c6412295bf5d5425f6bf2be7f54dd1623be0c1f8b06eda094a4e572ea994e302"} Oct 30 05:14:02.653915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.653484 6587 scope.go:122] "RemoveContainer" containerID="f380abee44cda86058a7992ea1f698fb29eac236235ea2bb5a6bd997bc30cd67" Oct 30 05:14:02.712778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.712749 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78"] Oct 30 05:14:02.725555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:02.725525 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78"] Oct 30 05:14:03.587645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:03.587603 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" path="/var/lib/kubelet/pods/6ff505e8-bee4-4f87-93e7-113497e13649/volumes" Oct 30 05:14:03.813088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:03.813053 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc"] Oct 30 05:14:03.813609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:03.813252 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" podUID="7ede7fd7-21cf-4250-bb20-708d1f2d6240" containerName="istio-proxy" containerID="cri-o://a390aa386c46073ea644e42affaa85db462900a2bbe60f69e52e2c578798d0b4" gracePeriod=30 Oct 30 05:14:04.051801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.051768 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5"] Oct 30 05:14:04.052646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.052625 6587 cpu_manager.go:414] "containerMap: removing container" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" Oct 30 05:14:04.052785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.052648 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" Oct 30 05:14:04.052857 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.052838 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" Oct 30 05:14:04.052913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.052862 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" Oct 30 05:14:04.056800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.056781 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.084658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.084633 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8afb26-openshift-default-dockercfg-bnnnm\"" Oct 30 05:14:04.085069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.085043 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5"] Oct 30 05:14:04.210090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.210057 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-envoy\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.210287 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.210102 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-istio-token\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.210287 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.210184 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98061727-7ccf-40ba-88a2-1442bcb73855-istio-podinfo\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.210468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.210300 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c8q9\" (UniqueName: \"kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-kube-api-access-5c8q9\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.210468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.210416 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-certs\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.210574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.210469 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-data\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.210631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.210583 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98061727-7ccf-40ba-88a2-1442bcb73855-istiod-ca-cert\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.210715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.210699 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-socket\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.210775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.210762 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-credential-socket\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.323431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.320513 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-certs\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.323431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.320582 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-data\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.323431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.320698 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98061727-7ccf-40ba-88a2-1442bcb73855-istiod-ca-cert\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.323431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.320825 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-socket\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.323431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.320904 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-credential-socket\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.323431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.321002 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-envoy\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.323431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.321044 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-istio-token\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.323431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.321083 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98061727-7ccf-40ba-88a2-1442bcb73855-istio-podinfo\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.323431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.321179 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8q9\" (UniqueName: \"kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-kube-api-access-5c8q9\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.346381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.340607 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-credential-socket\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.346381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.340915 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-data\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.346381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.341564 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98061727-7ccf-40ba-88a2-1442bcb73855-istiod-ca-cert\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.346381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.341798 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-socket\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.356659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.352731 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-certs\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.369535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.367131 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-envoy\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.400381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.398140 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c8q9\" (UniqueName: \"kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-kube-api-access-5c8q9\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.422570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.415961 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98061727-7ccf-40ba-88a2-1442bcb73855-istio-podinfo\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.441394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.435514 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-istio-token\") pod \"e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.675678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.675597 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:04.766186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.762407 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7"] Oct 30 05:14:04.766186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.763645 6587 cpu_manager.go:414] "containerMap: removing container" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" Oct 30 05:14:04.766186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.763663 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff505e8-bee4-4f87-93e7-113497e13649" containerName="istio-proxy" Oct 30 05:14:04.787766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.781350 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" Oct 30 05:14:04.793587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.793563 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7"] Oct 30 05:14:04.800589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.800561 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:14:04.800704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.800619 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:14:04.800841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.800822 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:14:04.800905 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.800829 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:14:04.838079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.838018 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7\" (UID: \"734e3867-4a6c-46e1-b43f-220b7427bb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" Oct 30 05:14:04.838893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.838869 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cldtp\" (UniqueName: \"kubernetes.io/projected/734e3867-4a6c-46e1-b43f-220b7427bb64-kube-api-access-cldtp\") pod \"maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7\" (UID: \"734e3867-4a6c-46e1-b43f-220b7427bb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" Oct 30 05:14:04.839132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.839115 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7\" (UID: \"734e3867-4a6c-46e1-b43f-220b7427bb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" Oct 30 05:14:04.941081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.940985 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7\" (UID: \"734e3867-4a6c-46e1-b43f-220b7427bb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" Oct 30 05:14:04.941263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.941171 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7\" (UID: \"734e3867-4a6c-46e1-b43f-220b7427bb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" Oct 30 05:14:04.941543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.941500 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cldtp\" (UniqueName: \"kubernetes.io/projected/734e3867-4a6c-46e1-b43f-220b7427bb64-kube-api-access-cldtp\") pod \"maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7\" (UID: \"734e3867-4a6c-46e1-b43f-220b7427bb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" Oct 30 05:14:04.941860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.941833 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7\" (UID: \"734e3867-4a6c-46e1-b43f-220b7427bb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" Oct 30 05:14:04.943757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.943738 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7\" (UID: \"734e3867-4a6c-46e1-b43f-220b7427bb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" Oct 30 05:14:04.962322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:04.962296 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cldtp\" (UniqueName: \"kubernetes.io/projected/734e3867-4a6c-46e1-b43f-220b7427bb64-kube-api-access-cldtp\") pod \"maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7\" (UID: \"734e3867-4a6c-46e1-b43f-220b7427bb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" Oct 30 05:14:05.009944 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:14:05.009919 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98061727_7ccf_40ba_88a2_1442bcb73855.slice/crio-6148b67b9d700638ba2a34e7f7eee7b74e1964ecfbe88251aff71d79e5818d38 WatchSource:0}: Error finding container 6148b67b9d700638ba2a34e7f7eee7b74e1964ecfbe88251aff71d79e5818d38: Status 404 returned error can't find the container with id 6148b67b9d700638ba2a34e7f7eee7b74e1964ecfbe88251aff71d79e5818d38 Oct 30 05:14:05.012612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:05.012587 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5"] Oct 30 05:14:05.013535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:05.013498 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:14:05.013613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:05.013584 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:14:05.013655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:05.013629 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:14:05.103745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:05.103720 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" Oct 30 05:14:05.416269 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:14:05.415540 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734e3867_4a6c_46e1_b43f_220b7427bb64.slice/crio-86408734e55e323c2652f7fd2f13eafab8a012830b19356a287c3d7b05618f4b WatchSource:0}: Error finding container 86408734e55e323c2652f7fd2f13eafab8a012830b19356a287c3d7b05618f4b: Status 404 returned error can't find the container with id 86408734e55e323c2652f7fd2f13eafab8a012830b19356a287c3d7b05618f4b Oct 30 05:14:05.417906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:05.417862 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7"] Oct 30 05:14:05.687330 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:05.687233 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" event={"ID":"734e3867-4a6c-46e1-b43f-220b7427bb64","Type":"ContainerStarted","Data":"86408734e55e323c2652f7fd2f13eafab8a012830b19356a287c3d7b05618f4b"} Oct 30 05:14:05.689208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:05.689172 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" event={"ID":"98061727-7ccf-40ba-88a2-1442bcb73855","Type":"ContainerStarted","Data":"00ffca31366e48a3b1f02bcd15a884a7c26022813912795ba1f2a9f3d4004859"} Oct 30 05:14:05.689397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:05.689226 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" event={"ID":"98061727-7ccf-40ba-88a2-1442bcb73855","Type":"ContainerStarted","Data":"6148b67b9d700638ba2a34e7f7eee7b74e1964ecfbe88251aff71d79e5818d38"} Oct 30 05:14:05.736032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:05.735970 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" podStartSLOduration=1.735951092 podStartE2EDuration="1.735951092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:05.732665549 +0000 UTC m=+1770.260432338" watchObservedRunningTime="2026-10-30 05:14:05.735951092 +0000 UTC m=+1770.263717878" Oct 30 05:14:06.715536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:06.711142 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:06.962105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:06.958745 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:06.982446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:06.980470 6587 generic.go:356] "Generic (PLEG): container finished" podID="734e3867-4a6c-46e1-b43f-220b7427bb64" containerID="cf29b333ab59bc423e9cb9dacd6b4c10fc25fe8cecc76a11ce75e9f4b467d397" exitCode=0 Oct 30 05:14:06.984007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:06.983979 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" event={"ID":"734e3867-4a6c-46e1-b43f-220b7427bb64","Type":"ContainerDied","Data":"cf29b333ab59bc423e9cb9dacd6b4c10fc25fe8cecc76a11ce75e9f4b467d397"} Oct 30 05:14:06.984182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:06.984167 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:06.985862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:06.985826 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:14:08.188441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:08.188410 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" Oct 30 05:14:08.261260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:08.261171 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-xks-ca\") pod \"734e3867-4a6c-46e1-b43f-220b7427bb64\" (UID: \"734e3867-4a6c-46e1-b43f-220b7427bb64\") " Oct 30 05:14:08.261441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:08.261279 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734e3867-4a6c-46e1-b43f-220b7427bb64-kube-api-access-cldtp\" (UniqueName: \"kubernetes.io/projected/734e3867-4a6c-46e1-b43f-220b7427bb64-kube-api-access-cldtp\") pod \"734e3867-4a6c-46e1-b43f-220b7427bb64\" (UID: \"734e3867-4a6c-46e1-b43f-220b7427bb64\") " Oct 30 05:14:08.261441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:08.261352 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-openshift-ca\") pod \"734e3867-4a6c-46e1-b43f-220b7427bb64\" (UID: \"734e3867-4a6c-46e1-b43f-220b7427bb64\") " Oct 30 05:14:08.261751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:08.261708 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-openshift-ca" pod "734e3867-4a6c-46e1-b43f-220b7427bb64" (UID: "734e3867-4a6c-46e1-b43f-220b7427bb64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:08.262522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:08.262498 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:08.263646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:08.263616 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-xks-ca" pod "734e3867-4a6c-46e1-b43f-220b7427bb64" (UID: "734e3867-4a6c-46e1-b43f-220b7427bb64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:08.264453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:08.264430 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734e3867-4a6c-46e1-b43f-220b7427bb64-kube-api-access-cldtp" pod "734e3867-4a6c-46e1-b43f-220b7427bb64" (UID: "734e3867-4a6c-46e1-b43f-220b7427bb64"). InnerVolumeSpecName "kube-api-access-cldtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:08.364202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:08.364166 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cldtp\" (UniqueName: \"kubernetes.io/projected/734e3867-4a6c-46e1-b43f-220b7427bb64-kube-api-access-cldtp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:08.364202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:08.364197 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734e3867-4a6c-46e1-b43f-220b7427bb64-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:09.017818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.014511 6587 generic.go:356] "Generic (PLEG): container finished" podID="7ede7fd7-21cf-4250-bb20-708d1f2d6240" containerID="a390aa386c46073ea644e42affaa85db462900a2bbe60f69e52e2c578798d0b4" exitCode=0 Oct 30 05:14:09.017818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.014915 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" event={"ID":"7ede7fd7-21cf-4250-bb20-708d1f2d6240","Type":"ContainerDied","Data":"a390aa386c46073ea644e42affaa85db462900a2bbe60f69e52e2c578798d0b4"} Oct 30 05:14:09.019907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.019878 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" Oct 30 05:14:09.020147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.020067 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7" event={"ID":"734e3867-4a6c-46e1-b43f-220b7427bb64","Type":"ContainerDied","Data":"86408734e55e323c2652f7fd2f13eafab8a012830b19356a287c3d7b05618f4b"} Oct 30 05:14:09.020147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.020127 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86408734e55e323c2652f7fd2f13eafab8a012830b19356a287c3d7b05618f4b" Oct 30 05:14:09.138386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.138341 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:14:09.188848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.188814 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-socket\") pod \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " Oct 30 05:14:09.189572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.189549 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-token\" (UniqueName: \"kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-token\") pod \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " Oct 30 05:14:09.189733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.189719 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-data\") pod \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " Oct 30 05:14:09.189903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.189890 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-credential-socket\") pod \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " Oct 30 05:14:09.190094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.190082 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-certs\") pod \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " Oct 30 05:14:09.190208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.190196 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-kube-api-access-p5r6s\" (UniqueName: \"kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-kube-api-access-p5r6s\") pod \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " Oct 30 05:14:09.190407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.190391 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istiod-ca-cert\") pod \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " Oct 30 05:14:09.190537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.190525 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-podinfo\") pod \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " Oct 30 05:14:09.190629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.190617 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-envoy\") pod \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\" (UID: \"7ede7fd7-21cf-4250-bb20-708d1f2d6240\") " Oct 30 05:14:09.194399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.192317 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-credential-socket" pod "7ede7fd7-21cf-4250-bb20-708d1f2d6240" (UID: "7ede7fd7-21cf-4250-bb20-708d1f2d6240"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:09.194399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.192519 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-socket" pod "7ede7fd7-21cf-4250-bb20-708d1f2d6240" (UID: "7ede7fd7-21cf-4250-bb20-708d1f2d6240"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:09.195885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.195856 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-data" pod "7ede7fd7-21cf-4250-bb20-708d1f2d6240" (UID: "7ede7fd7-21cf-4250-bb20-708d1f2d6240"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:09.196651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.196622 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-certs" pod "7ede7fd7-21cf-4250-bb20-708d1f2d6240" (UID: "7ede7fd7-21cf-4250-bb20-708d1f2d6240"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:09.196951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.196926 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istiod-ca-cert" pod "7ede7fd7-21cf-4250-bb20-708d1f2d6240" (UID: "7ede7fd7-21cf-4250-bb20-708d1f2d6240"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:09.198076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.198036 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-token" pod "7ede7fd7-21cf-4250-bb20-708d1f2d6240" (UID: "7ede7fd7-21cf-4250-bb20-708d1f2d6240"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:09.200505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.200474 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-envoy" pod "7ede7fd7-21cf-4250-bb20-708d1f2d6240" (UID: "7ede7fd7-21cf-4250-bb20-708d1f2d6240"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:09.202869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.202841 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-kube-api-access-p5r6s" pod "7ede7fd7-21cf-4250-bb20-708d1f2d6240" (UID: "7ede7fd7-21cf-4250-bb20-708d1f2d6240"). InnerVolumeSpecName "kube-api-access-p5r6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:09.207558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.207522 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-podinfo" pod "7ede7fd7-21cf-4250-bb20-708d1f2d6240" (UID: "7ede7fd7-21cf-4250-bb20-708d1f2d6240"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:14:09.293102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.293065 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:09.293102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.293102 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5r6s\" (UniqueName: \"kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-kube-api-access-p5r6s\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:09.293318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.293118 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:09.293318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.293133 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:09.293318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.293146 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:09.293318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.293160 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:09.293318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.293174 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:09.293318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.293188 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:09.293318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:09.293200 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede7fd7-21cf-4250-bb20-708d1f2d6240-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:10.031694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.031607 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" Oct 30 05:14:10.031694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.031638 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc" event={"ID":"7ede7fd7-21cf-4250-bb20-708d1f2d6240","Type":"ContainerDied","Data":"79bcaca4f61e9706ec297b42d1083c2fddd7149b3f8dc7e5cbc1c50df9c33712"} Oct 30 05:14:10.031915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.031703 6587 scope.go:122] "RemoveContainer" containerID="a390aa386c46073ea644e42affaa85db462900a2bbe60f69e52e2c578798d0b4" Oct 30 05:14:10.117019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.116988 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc"] Oct 30 05:14:10.130268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.130236 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc"] Oct 30 05:14:10.148991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.148952 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn"] Oct 30 05:14:10.150056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.150037 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7ede7fd7-21cf-4250-bb20-708d1f2d6240" containerName="istio-proxy" Oct 30 05:14:10.150118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.150059 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ede7fd7-21cf-4250-bb20-708d1f2d6240" containerName="istio-proxy" Oct 30 05:14:10.150118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.150102 6587 cpu_manager.go:414] "containerMap: removing container" podUID="734e3867-4a6c-46e1-b43f-220b7427bb64" containerName="revoke-keys" Oct 30 05:14:10.150118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.150110 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734e3867-4a6c-46e1-b43f-220b7427bb64" containerName="revoke-keys" Oct 30 05:14:10.150322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.150310 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="734e3867-4a6c-46e1-b43f-220b7427bb64" containerName="revoke-keys" Oct 30 05:14:10.150377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.150335 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ede7fd7-21cf-4250-bb20-708d1f2d6240" containerName="istio-proxy" Oct 30 05:14:10.160776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.160736 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" Oct 30 05:14:10.176684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.176647 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn"] Oct 30 05:14:10.181430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.181346 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8afb26\"" Oct 30 05:14:10.243383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.213223 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mgw4\" (UniqueName: \"kubernetes.io/projected/c4f882bc-ffd5-4288-bf40-388de69479bf-kube-api-access-4mgw4\") pod \"payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn\" (UID: \"c4f882bc-ffd5-4288-bf40-388de69479bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" Oct 30 05:14:10.243383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.213375 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f882bc-ffd5-4288-bf40-388de69479bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn\" (UID: \"c4f882bc-ffd5-4288-bf40-388de69479bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" Oct 30 05:14:10.243383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.239326 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8afb26-dockercfg-l4hpn\"" Oct 30 05:14:10.394478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.390584 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgw4\" (UniqueName: \"kubernetes.io/projected/c4f882bc-ffd5-4288-bf40-388de69479bf-kube-api-access-4mgw4\") pod \"payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn\" (UID: \"c4f882bc-ffd5-4288-bf40-388de69479bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" Oct 30 05:14:10.394478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.390720 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f882bc-ffd5-4288-bf40-388de69479bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn\" (UID: \"c4f882bc-ffd5-4288-bf40-388de69479bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" Oct 30 05:14:10.394478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.392068 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f882bc-ffd5-4288-bf40-388de69479bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn\" (UID: \"c4f882bc-ffd5-4288-bf40-388de69479bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" Oct 30 05:14:10.492203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.479696 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn"] Oct 30 05:14:10.492203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.487009 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mgw4\" (UniqueName: \"kubernetes.io/projected/c4f882bc-ffd5-4288-bf40-388de69479bf-kube-api-access-4mgw4\") pod \"payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn\" (UID: \"c4f882bc-ffd5-4288-bf40-388de69479bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" Oct 30 05:14:10.492203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.487854 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn"] Oct 30 05:14:10.492203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.487943 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" Oct 30 05:14:10.517725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.500441 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" Oct 30 05:14:10.595598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.595534 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31b88a5-6d36-4588-8505-33ce9292c685-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn\" (UID: \"c31b88a5-6d36-4588-8505-33ce9292c685\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" Oct 30 05:14:10.597396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.596804 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4bk5\" (UniqueName: \"kubernetes.io/projected/c31b88a5-6d36-4588-8505-33ce9292c685-kube-api-access-w4bk5\") pod \"payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn\" (UID: \"c31b88a5-6d36-4588-8505-33ce9292c685\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" Oct 30 05:14:10.699306 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.699216 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4bk5\" (UniqueName: \"kubernetes.io/projected/c31b88a5-6d36-4588-8505-33ce9292c685-kube-api-access-w4bk5\") pod \"payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn\" (UID: \"c31b88a5-6d36-4588-8505-33ce9292c685\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" Oct 30 05:14:10.699550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.699391 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31b88a5-6d36-4588-8505-33ce9292c685-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn\" (UID: \"c31b88a5-6d36-4588-8505-33ce9292c685\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" Oct 30 05:14:10.700549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.700503 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31b88a5-6d36-4588-8505-33ce9292c685-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn\" (UID: \"c31b88a5-6d36-4588-8505-33ce9292c685\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" Oct 30 05:14:10.722049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.722012 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4bk5\" (UniqueName: \"kubernetes.io/projected/c31b88a5-6d36-4588-8505-33ce9292c685-kube-api-access-w4bk5\") pod \"payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn\" (UID: \"c31b88a5-6d36-4588-8505-33ce9292c685\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" Oct 30 05:14:10.820022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.819992 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" Oct 30 05:14:10.864012 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:14:10.863980 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f882bc_ffd5_4288_bf40_388de69479bf.slice/crio-32a251c1f9f64e89a6b636380a3614a5409982ec81090cd85a994115b530fce8 WatchSource:0}: Error finding container 32a251c1f9f64e89a6b636380a3614a5409982ec81090cd85a994115b530fce8: Status 404 returned error can't find the container with id 32a251c1f9f64e89a6b636380a3614a5409982ec81090cd85a994115b530fce8 Oct 30 05:14:10.865178 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:10.865087 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn"] Oct 30 05:14:11.197184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:11.186799 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn"] Oct 30 05:14:11.197184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:11.196816 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" event={"ID":"c4f882bc-ffd5-4288-bf40-388de69479bf","Type":"ContainerStarted","Data":"32a251c1f9f64e89a6b636380a3614a5409982ec81090cd85a994115b530fce8"} Oct 30 05:14:11.587437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:11.587404 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ede7fd7-21cf-4250-bb20-708d1f2d6240" path="/var/lib/kubelet/pods/7ede7fd7-21cf-4250-bb20-708d1f2d6240/volumes" Oct 30 05:14:11.677449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:11.677272 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss"] Oct 30 05:14:11.681901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:11.681870 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47af37-g87ss"] Oct 30 05:14:12.207252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:12.207211 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" event={"ID":"c31b88a5-6d36-4588-8505-33ce9292c685","Type":"ContainerStarted","Data":"b9c64a0237c2a1000b4a7aec847dcd8440fba0b99b6093d69f0a6fcc6e1e3de5"} Oct 30 05:14:12.207453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:12.207280 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" Oct 30 05:14:12.207453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:12.207304 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" event={"ID":"c31b88a5-6d36-4588-8505-33ce9292c685","Type":"ContainerStarted","Data":"1ef7a3953d9318cd68af044e1e619c8ab4b843ef127442a0412d0d0b335dd466"} Oct 30 05:14:12.209033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:12.209003 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" event={"ID":"c4f882bc-ffd5-4288-bf40-388de69479bf","Type":"ContainerStarted","Data":"97fe5c925156c3c69681fc6f0ea5f07bfb3cc41749e17b183c6c6692363d3a8e"} Oct 30 05:14:12.209166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:12.209087 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" Oct 30 05:14:12.255105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:12.255052 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" podStartSLOduration=2.255038571 podStartE2EDuration="2.255038571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:12.253785249 +0000 UTC m=+1776.781552037" watchObservedRunningTime="2026-10-30 05:14:12.255038571 +0000 UTC m=+1776.782805356" Oct 30 05:14:12.299035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:12.298983 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" podStartSLOduration=2.298969155 podStartE2EDuration="2.298969155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:12.295232036 +0000 UTC m=+1776.822998823" watchObservedRunningTime="2026-10-30 05:14:12.298969155 +0000 UTC m=+1776.826735942" Oct 30 05:14:13.588497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:13.588462 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3baff95-ae1b-4863-8516-b566fdbdf1b8" path="/var/lib/kubelet/pods/b3baff95-ae1b-4863-8516-b566fdbdf1b8/volumes" Oct 30 05:14:15.248000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:15.247964 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft"] Oct 30 05:14:20.248908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:20.248865 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" podUID="e4567cee-33c8-4e3d-a858-6b7f1082b779" containerName="payload-pre-processing" containerID="cri-o://96c67553b8d37bc9b308bdf5cfaee847415abdafc8ddbe91753c6f9d8a4f3e76" gracePeriod=25 Oct 30 05:14:20.871636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:20.871594 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" podUID="e4567cee-33c8-4e3d-a858-6b7f1082b779" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:14:23.222987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:23.222901 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" Oct 30 05:14:23.223547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:23.223220 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" Oct 30 05:14:25.987369 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:14:25.987318 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:14:30.901378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:30.893452 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" podUID="e4567cee-33c8-4e3d-a858-6b7f1082b779" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:14:35.960176 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:35.960135 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:14:35.961036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:35.961014 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:14:38.164917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:38.164874 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj"] Oct 30 05:14:38.181843 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:38.181805 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0814c-n5xkj"] Oct 30 05:14:39.236088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:39.236052 6587 scope.go:122] "RemoveContainer" containerID="de94da50167894e51e4c1019c2f3fb5dd65172075374fa55bede2a777db994f0" Oct 30 05:14:39.246966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:39.246944 6587 scope.go:122] "RemoveContainer" containerID="a6c8015c03515e355df19b0bba0cf66040d8c1dcfd942e5e13643d7639b888af" Oct 30 05:14:39.585780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:39.585745 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7458a4c6-ac23-492d-9256-bc6f75b52474" path="/var/lib/kubelet/pods/7458a4c6-ac23-492d-9256-bc6f75b52474/volumes" Oct 30 05:14:40.672789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:40.672751 6587 generic.go:356] "Generic (PLEG): container finished" podID="e4567cee-33c8-4e3d-a858-6b7f1082b779" containerID="96c67553b8d37bc9b308bdf5cfaee847415abdafc8ddbe91753c6f9d8a4f3e76" exitCode=0 Oct 30 05:14:40.673235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:40.672973 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" event={"ID":"e4567cee-33c8-4e3d-a858-6b7f1082b779","Type":"ContainerDied","Data":"96c67553b8d37bc9b308bdf5cfaee847415abdafc8ddbe91753c6f9d8a4f3e76"} Oct 30 05:14:40.673235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:40.673065 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" event={"ID":"e4567cee-33c8-4e3d-a858-6b7f1082b779","Type":"ContainerDied","Data":"c2ab7cd2c989b08f12b5d237769b7318990e10133019bf169f31e46f27b3becb"} Oct 30 05:14:40.673235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:40.673092 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2ab7cd2c989b08f12b5d237769b7318990e10133019bf169f31e46f27b3becb" Oct 30 05:14:40.673235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:40.673220 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" Oct 30 05:14:40.771090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:40.771053 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4567cee-33c8-4e3d-a858-6b7f1082b779-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4567cee-33c8-4e3d-a858-6b7f1082b779-plugins-config-volume\") pod \"e4567cee-33c8-4e3d-a858-6b7f1082b779\" (UID: \"e4567cee-33c8-4e3d-a858-6b7f1082b779\") " Oct 30 05:14:40.771295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:40.771189 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4567cee-33c8-4e3d-a858-6b7f1082b779-kube-api-access-8m7jq\" (UniqueName: \"kubernetes.io/projected/e4567cee-33c8-4e3d-a858-6b7f1082b779-kube-api-access-8m7jq\") pod \"e4567cee-33c8-4e3d-a858-6b7f1082b779\" (UID: \"e4567cee-33c8-4e3d-a858-6b7f1082b779\") " Oct 30 05:14:40.771801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:40.771776 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4567cee-33c8-4e3d-a858-6b7f1082b779-plugins-config-volume" pod "e4567cee-33c8-4e3d-a858-6b7f1082b779" (UID: "e4567cee-33c8-4e3d-a858-6b7f1082b779"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:40.774185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:40.774155 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4567cee-33c8-4e3d-a858-6b7f1082b779-kube-api-access-8m7jq" pod "e4567cee-33c8-4e3d-a858-6b7f1082b779" (UID: "e4567cee-33c8-4e3d-a858-6b7f1082b779"). InnerVolumeSpecName "kube-api-access-8m7jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:40.873063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:40.872975 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m7jq\" (UniqueName: \"kubernetes.io/projected/e4567cee-33c8-4e3d-a858-6b7f1082b779-kube-api-access-8m7jq\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:40.873063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:40.873010 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4567cee-33c8-4e3d-a858-6b7f1082b779-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:41.681135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:41.681102 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft" Oct 30 05:14:41.756094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:41.755993 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft"] Oct 30 05:14:41.763179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:41.763148 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft"] Oct 30 05:14:43.592821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:43.592783 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4567cee-33c8-4e3d-a858-6b7f1082b779" path="/var/lib/kubelet/pods/e4567cee-33c8-4e3d-a858-6b7f1082b779/volumes" Oct 30 05:14:46.357411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.357343 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r"] Oct 30 05:14:46.358324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.358300 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e4567cee-33c8-4e3d-a858-6b7f1082b779" containerName="payload-pre-processing" Oct 30 05:14:46.358324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.358324 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4567cee-33c8-4e3d-a858-6b7f1082b779" containerName="payload-pre-processing" Oct 30 05:14:46.358503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.358467 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4567cee-33c8-4e3d-a858-6b7f1082b779" containerName="payload-pre-processing" Oct 30 05:14:46.364323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.364299 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" Oct 30 05:14:46.381020 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.380985 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r"] Oct 30 05:14:46.391964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.391912 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:14:46.392322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.392285 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:14:46.422066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.422030 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r\" (UID: \"e371f129-d805-4277-973e-3717fdb94ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" Oct 30 05:14:46.422397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.422369 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg2v2\" (UniqueName: \"kubernetes.io/projected/e371f129-d805-4277-973e-3717fdb94ca6-kube-api-access-pg2v2\") pod \"maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r\" (UID: \"e371f129-d805-4277-973e-3717fdb94ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" Oct 30 05:14:46.422588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.422547 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r\" (UID: \"e371f129-d805-4277-973e-3717fdb94ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" Oct 30 05:14:46.474519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.474315 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:14:46.474519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.474477 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:14:46.524691 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.524631 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r\" (UID: \"e371f129-d805-4277-973e-3717fdb94ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" Oct 30 05:14:46.524872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.524747 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg2v2\" (UniqueName: \"kubernetes.io/projected/e371f129-d805-4277-973e-3717fdb94ca6-kube-api-access-pg2v2\") pod \"maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r\" (UID: \"e371f129-d805-4277-973e-3717fdb94ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" Oct 30 05:14:46.524872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.524777 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r\" (UID: \"e371f129-d805-4277-973e-3717fdb94ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" Oct 30 05:14:46.530212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.530140 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r\" (UID: \"e371f129-d805-4277-973e-3717fdb94ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" Oct 30 05:14:46.552384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.548812 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r\" (UID: \"e371f129-d805-4277-973e-3717fdb94ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" Oct 30 05:14:46.571108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.570055 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg2v2\" (UniqueName: \"kubernetes.io/projected/e371f129-d805-4277-973e-3717fdb94ca6-kube-api-access-pg2v2\") pod \"maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r\" (UID: \"e371f129-d805-4277-973e-3717fdb94ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" Oct 30 05:14:46.686442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:46.686348 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" Oct 30 05:14:47.048965 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:14:47.048917 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode371f129_d805_4277_973e_3717fdb94ca6.slice/crio-0d064492992d2007d0496526b42b5703f32f692e1010b92619d443786b47156e WatchSource:0}: Error finding container 0d064492992d2007d0496526b42b5703f32f692e1010b92619d443786b47156e: Status 404 returned error can't find the container with id 0d064492992d2007d0496526b42b5703f32f692e1010b92619d443786b47156e Oct 30 05:14:47.049072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:47.049013 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r"] Oct 30 05:14:47.763965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:47.763839 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" event={"ID":"e371f129-d805-4277-973e-3717fdb94ca6","Type":"ContainerStarted","Data":"2d7dd1708db6c9c4203683c521d15a84503428695ad10257051c886d6eb52089"} Oct 30 05:14:47.763965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:47.763953 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" event={"ID":"e371f129-d805-4277-973e-3717fdb94ca6","Type":"ContainerStarted","Data":"0d064492992d2007d0496526b42b5703f32f692e1010b92619d443786b47156e"} Oct 30 05:14:47.809249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:47.809196 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" podStartSLOduration=1.809182971 podStartE2EDuration="1.809182971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:47.807077931 +0000 UTC m=+1812.334844732" watchObservedRunningTime="2026-10-30 05:14:47.809182971 +0000 UTC m=+1812.336949758" Oct 30 05:14:48.775525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:48.775491 6587 generic.go:356] "Generic (PLEG): container finished" podID="e371f129-d805-4277-973e-3717fdb94ca6" containerID="2d7dd1708db6c9c4203683c521d15a84503428695ad10257051c886d6eb52089" exitCode=0 Oct 30 05:14:48.776032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:48.775549 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" event={"ID":"e371f129-d805-4277-973e-3717fdb94ca6","Type":"ContainerDied","Data":"2d7dd1708db6c9c4203683c521d15a84503428695ad10257051c886d6eb52089"} Oct 30 05:14:50.045999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.045928 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" Oct 30 05:14:50.160373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.156888 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e371f129-d805-4277-973e-3717fdb94ca6-kube-api-access-pg2v2\" (UniqueName: \"kubernetes.io/projected/e371f129-d805-4277-973e-3717fdb94ca6-kube-api-access-pg2v2\") pod \"e371f129-d805-4277-973e-3717fdb94ca6\" (UID: \"e371f129-d805-4277-973e-3717fdb94ca6\") " Oct 30 05:14:50.160373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.156958 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-openshift-ca\") pod \"e371f129-d805-4277-973e-3717fdb94ca6\" (UID: \"e371f129-d805-4277-973e-3717fdb94ca6\") " Oct 30 05:14:50.160373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.157010 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-xks-ca\") pod \"e371f129-d805-4277-973e-3717fdb94ca6\" (UID: \"e371f129-d805-4277-973e-3717fdb94ca6\") " Oct 30 05:14:50.161369 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.161307 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-openshift-ca" pod "e371f129-d805-4277-973e-3717fdb94ca6" (UID: "e371f129-d805-4277-973e-3717fdb94ca6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:50.161649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.161620 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-xks-ca" pod "e371f129-d805-4277-973e-3717fdb94ca6" (UID: "e371f129-d805-4277-973e-3717fdb94ca6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:50.167016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.166946 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e371f129-d805-4277-973e-3717fdb94ca6-kube-api-access-pg2v2" pod "e371f129-d805-4277-973e-3717fdb94ca6" (UID: "e371f129-d805-4277-973e-3717fdb94ca6"). InnerVolumeSpecName "kube-api-access-pg2v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:50.260146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.260107 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:50.260307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.260150 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e371f129-d805-4277-973e-3717fdb94ca6-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:50.260307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.260169 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg2v2\" (UniqueName: \"kubernetes.io/projected/e371f129-d805-4277-973e-3717fdb94ca6-kube-api-access-pg2v2\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:14:50.796827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.796725 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" Oct 30 05:14:50.796827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.796739 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r" event={"ID":"e371f129-d805-4277-973e-3717fdb94ca6","Type":"ContainerDied","Data":"0d064492992d2007d0496526b42b5703f32f692e1010b92619d443786b47156e"} Oct 30 05:14:50.796827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.796794 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d064492992d2007d0496526b42b5703f32f692e1010b92619d443786b47156e" Oct 30 05:14:50.798792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.798767 6587 generic.go:356] "Generic (PLEG): container finished" podID="548909cc-6196-4b54-9372-e9f61190ddce" containerID="145cc54dd3354c34a60099025653385ed8c73ed200d9a158cae9c68ebba6580b" exitCode=0 Oct 30 05:14:50.798939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.798822 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" event={"ID":"548909cc-6196-4b54-9372-e9f61190ddce","Type":"ContainerDied","Data":"145cc54dd3354c34a60099025653385ed8c73ed200d9a158cae9c68ebba6580b"} Oct 30 05:14:50.799299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.799281 6587 scope.go:122] "RemoveContainer" containerID="145cc54dd3354c34a60099025653385ed8c73ed200d9a158cae9c68ebba6580b" Oct 30 05:14:50.801241 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.801212 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:14:50.801336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.801291 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:14:50.801336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:50.801323 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:14:51.809369 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:51.809328 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" event={"ID":"548909cc-6196-4b54-9372-e9f61190ddce","Type":"ContainerStarted","Data":"1a460147748390515cee934b5a061105f177ccfccf23f29dd450b14cad1356a2"} Oct 30 05:14:51.915224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:51.915192 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:14:51.915447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:51.915403 6587 patch_prober.go:29] interesting pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 05:14:51.938525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:51.915457 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" podUID="548909cc-6196-4b54-9372-e9f61190ddce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 05:14:52.914637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:52.914601 6587 patch_prober.go:29] interesting pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 05:14:52.915039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:52.914661 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" podUID="548909cc-6196-4b54-9372-e9f61190ddce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 05:14:53.915227 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:53.915194 6587 patch_prober.go:29] interesting pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 05:14:53.915764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:53.915251 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" podUID="548909cc-6196-4b54-9372-e9f61190ddce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 05:14:54.915016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:54.914983 6587 patch_prober.go:29] interesting pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 05:14:54.915220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:54.915036 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" podUID="548909cc-6196-4b54-9372-e9f61190ddce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 05:14:55.924506 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:55.924465 6587 patch_prober.go:29] interesting pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 05:14:55.925125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:55.925090 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" podUID="548909cc-6196-4b54-9372-e9f61190ddce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 05:14:55.986003 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:14:55.985969 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:14:56.915053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:56.915017 6587 patch_prober.go:29] interesting pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 05:14:56.915053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:56.915083 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" podUID="548909cc-6196-4b54-9372-e9f61190ddce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 05:14:57.702382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:57.696630 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7"] Oct 30 05:14:57.820917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:57.820004 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4225df-t4hd7"] Oct 30 05:14:57.935389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:57.934739 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:14:57.936420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:57.935839 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:14:57.939385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:57.938465 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:14:59.781943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:14:59.781904 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734e3867-4a6c-46e1-b43f-220b7427bb64" path="/var/lib/kubelet/pods/734e3867-4a6c-46e1-b43f-220b7427bb64/volumes" Oct 30 05:15:00.159479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.159439 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25"] Oct 30 05:15:00.160917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.160890 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e371f129-d805-4277-973e-3717fdb94ca6" containerName="revoke-keys" Oct 30 05:15:00.160917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.160915 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e371f129-d805-4277-973e-3717fdb94ca6" containerName="revoke-keys" Oct 30 05:15:00.161130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.161119 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e371f129-d805-4277-973e-3717fdb94ca6" containerName="revoke-keys" Oct 30 05:15:00.165714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.165692 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" Oct 30 05:15:00.166437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.166415 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw"] Oct 30 05:15:00.171120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.171102 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" Oct 30 05:15:00.172847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.172830 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958"] Oct 30 05:15:00.178926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.178906 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws"] Oct 30 05:15:00.179616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.179600 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" Oct 30 05:15:00.184958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.184942 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" Oct 30 05:15:00.186818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.186787 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8"] Oct 30 05:15:00.199885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.199857 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25"] Oct 30 05:15:00.200098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.200080 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" Oct 30 05:15:00.202266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.202251 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:15:00.229591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.213025 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh"] Oct 30 05:15:00.245381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.236835 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:15:00.245381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.236948 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:15:00.258963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.258928 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lmsp\" (UniqueName: \"kubernetes.io/projected/01e7d7ed-2e36-4285-a316-c6b296a75cd2-kube-api-access-2lmsp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25\" (UID: \"01e7d7ed-2e36-4285-a316-c6b296a75cd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" Oct 30 05:15:00.259296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.259272 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94412d8-2396-4b33-a05a-5189b4ba72b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws\" (UID: \"b94412d8-2396-4b33-a05a-5189b4ba72b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" Oct 30 05:15:00.262612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.259491 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958\" (UID: \"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" Oct 30 05:15:00.262844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.262821 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l2rm\" (UniqueName: \"kubernetes.io/projected/dc428e2b-af51-47bb-809f-a34110a0e2ef-kube-api-access-9l2rm\") pod \"maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw\" (UID: \"dc428e2b-af51-47bb-809f-a34110a0e2ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" Oct 30 05:15:00.263048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.263028 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpgfn\" (UniqueName: \"kubernetes.io/projected/b94412d8-2396-4b33-a05a-5189b4ba72b5-kube-api-access-vpgfn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws\" (UID: \"b94412d8-2396-4b33-a05a-5189b4ba72b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" Oct 30 05:15:00.263280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.263261 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvgf\" (UniqueName: \"kubernetes.io/projected/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-kube-api-access-bxvgf\") pod \"maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958\" (UID: \"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" Oct 30 05:15:00.263620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.263585 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc428e2b-af51-47bb-809f-a34110a0e2ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw\" (UID: \"dc428e2b-af51-47bb-809f-a34110a0e2ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" Oct 30 05:15:00.263790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.263771 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e7d7ed-2e36-4285-a316-c6b296a75cd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25\" (UID: \"01e7d7ed-2e36-4285-a316-c6b296a75cd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" Oct 30 05:15:00.264390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.264310 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958"] Oct 30 05:15:00.264499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.264487 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" Oct 30 05:15:00.285189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.269554 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw"] Oct 30 05:15:00.285189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.278400 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws"] Oct 30 05:15:00.288090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.288064 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8"] Oct 30 05:15:00.303514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.301774 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m"] Oct 30 05:15:00.313775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.311118 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" Oct 30 05:15:00.313775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.313379 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh"] Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.365616 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvgf\" (UniqueName: \"kubernetes.io/projected/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-kube-api-access-bxvgf\") pod \"maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958\" (UID: \"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.365690 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2tc6\" (UniqueName: \"kubernetes.io/projected/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-kube-api-access-w2tc6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh\" (UID: \"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.365766 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc428e2b-af51-47bb-809f-a34110a0e2ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw\" (UID: \"dc428e2b-af51-47bb-809f-a34110a0e2ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.365794 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e7d7ed-2e36-4285-a316-c6b296a75cd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25\" (UID: \"01e7d7ed-2e36-4285-a316-c6b296a75cd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.365828 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lmsp\" (UniqueName: \"kubernetes.io/projected/01e7d7ed-2e36-4285-a316-c6b296a75cd2-kube-api-access-2lmsp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25\" (UID: \"01e7d7ed-2e36-4285-a316-c6b296a75cd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.365869 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94412d8-2396-4b33-a05a-5189b4ba72b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws\" (UID: \"b94412d8-2396-4b33-a05a-5189b4ba72b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.365892 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958\" (UID: \"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.365937 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1154868f-4d78-401f-8fd0-cdd9a0fda825-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8\" (UID: \"1154868f-4d78-401f-8fd0-cdd9a0fda825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.365959 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2rm\" (UniqueName: \"kubernetes.io/projected/dc428e2b-af51-47bb-809f-a34110a0e2ef-kube-api-access-9l2rm\") pod \"maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw\" (UID: \"dc428e2b-af51-47bb-809f-a34110a0e2ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.365990 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh\" (UID: \"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.366010 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l94qc\" (UniqueName: \"kubernetes.io/projected/1154868f-4d78-401f-8fd0-cdd9a0fda825-kube-api-access-l94qc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8\" (UID: \"1154868f-4d78-401f-8fd0-cdd9a0fda825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.366038 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpgfn\" (UniqueName: \"kubernetes.io/projected/b94412d8-2396-4b33-a05a-5189b4ba72b5-kube-api-access-vpgfn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws\" (UID: \"b94412d8-2396-4b33-a05a-5189b4ba72b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.366961 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc428e2b-af51-47bb-809f-a34110a0e2ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw\" (UID: \"dc428e2b-af51-47bb-809f-a34110a0e2ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" Oct 30 05:15:00.367621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.367396 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e7d7ed-2e36-4285-a316-c6b296a75cd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25\" (UID: \"01e7d7ed-2e36-4285-a316-c6b296a75cd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" Oct 30 05:15:00.369753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.367998 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94412d8-2396-4b33-a05a-5189b4ba72b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws\" (UID: \"b94412d8-2396-4b33-a05a-5189b4ba72b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" Oct 30 05:15:00.369753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.368438 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958\" (UID: \"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" Oct 30 05:15:00.373828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.373805 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2"] Oct 30 05:15:00.384501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.383536 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn"] Oct 30 05:15:00.386066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.385396 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" Oct 30 05:15:00.390503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.390477 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" Oct 30 05:15:00.394460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.393465 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx"] Oct 30 05:15:00.400002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.398738 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvgf\" (UniqueName: \"kubernetes.io/projected/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-kube-api-access-bxvgf\") pod \"maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958\" (UID: \"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" Oct 30 05:15:00.400090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.400000 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpgfn\" (UniqueName: \"kubernetes.io/projected/b94412d8-2396-4b33-a05a-5189b4ba72b5-kube-api-access-vpgfn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws\" (UID: \"b94412d8-2396-4b33-a05a-5189b4ba72b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" Oct 30 05:15:00.400915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.400893 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l2rm\" (UniqueName: \"kubernetes.io/projected/dc428e2b-af51-47bb-809f-a34110a0e2ef-kube-api-access-9l2rm\") pod \"maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw\" (UID: \"dc428e2b-af51-47bb-809f-a34110a0e2ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" Oct 30 05:15:00.401736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.401648 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" Oct 30 05:15:00.402042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.402024 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d"] Oct 30 05:15:00.402605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.402581 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lmsp\" (UniqueName: \"kubernetes.io/projected/01e7d7ed-2e36-4285-a316-c6b296a75cd2-kube-api-access-2lmsp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25\" (UID: \"01e7d7ed-2e36-4285-a316-c6b296a75cd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" Oct 30 05:15:00.409953 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.408396 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m"] Oct 30 05:15:00.409953 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.408490 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" Oct 30 05:15:00.411513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.411492 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn"] Oct 30 05:15:00.414846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.414827 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx"] Oct 30 05:15:00.464557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.464470 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh"] Oct 30 05:15:00.470560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.470532 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2"] Oct 30 05:15:00.470560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.470553 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" Oct 30 05:15:00.476054 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.476026 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1154868f-4d78-401f-8fd0-cdd9a0fda825-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8\" (UID: \"1154868f-4d78-401f-8fd0-cdd9a0fda825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" Oct 30 05:15:00.476183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.476103 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh\" (UID: \"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" Oct 30 05:15:00.476183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.476143 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l94qc\" (UniqueName: \"kubernetes.io/projected/1154868f-4d78-401f-8fd0-cdd9a0fda825-kube-api-access-l94qc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8\" (UID: \"1154868f-4d78-401f-8fd0-cdd9a0fda825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" Oct 30 05:15:00.476307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.476280 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m\" (UID: \"06dd03d6-332a-4912-b8e8-8fc243f2e8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" Oct 30 05:15:00.476387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.476352 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7jz4\" (UniqueName: \"kubernetes.io/projected/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-kube-api-access-c7jz4\") pod \"maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2\" (UID: \"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" Oct 30 05:15:00.476455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.476424 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2tc6\" (UniqueName: \"kubernetes.io/projected/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-kube-api-access-w2tc6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh\" (UID: \"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" Oct 30 05:15:00.476547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.476527 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l47vc\" (UniqueName: \"kubernetes.io/projected/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-kube-api-access-l47vc\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m\" (UID: \"06dd03d6-332a-4912-b8e8-8fc243f2e8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" Oct 30 05:15:00.476617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.476600 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2\" (UID: \"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" Oct 30 05:15:00.478451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.477755 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1154868f-4d78-401f-8fd0-cdd9a0fda825-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8\" (UID: \"1154868f-4d78-401f-8fd0-cdd9a0fda825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" Oct 30 05:15:00.478451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.478429 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh\" (UID: \"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" Oct 30 05:15:00.479297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.479273 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl"] Oct 30 05:15:00.484315 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.484292 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" Oct 30 05:15:00.488425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.488403 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh"] Oct 30 05:15:00.488524 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.488510 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" Oct 30 05:15:00.490891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.490787 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl"] Oct 30 05:15:00.494125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.494077 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d"] Oct 30 05:15:00.501539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.501514 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l94qc\" (UniqueName: \"kubernetes.io/projected/1154868f-4d78-401f-8fd0-cdd9a0fda825-kube-api-access-l94qc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8\" (UID: \"1154868f-4d78-401f-8fd0-cdd9a0fda825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" Oct 30 05:15:00.502130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.502105 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2tc6\" (UniqueName: \"kubernetes.io/projected/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-kube-api-access-w2tc6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh\" (UID: \"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" Oct 30 05:15:00.560384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.559619 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.577843 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l47vc\" (UniqueName: \"kubernetes.io/projected/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-kube-api-access-l47vc\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m\" (UID: \"06dd03d6-332a-4912-b8e8-8fc243f2e8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.577899 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzjrs\" (UniqueName: \"kubernetes.io/projected/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-kube-api-access-gzjrs\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d\" (UID: \"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.577921 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2\" (UID: \"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.577954 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888955-g2mnh\" (UID: \"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.578564 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tm5m\" (UniqueName: \"kubernetes.io/projected/7b67626a-07e8-4b08-bee3-243b269fb5dc-kube-api-access-9tm5m\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx\" (UID: \"7b67626a-07e8-4b08-bee3-243b269fb5dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.578604 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxps\" (UniqueName: \"kubernetes.io/projected/a387704a-20ad-4fa6-abb8-46ffc2b82f94-kube-api-access-vrxps\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn\" (UID: \"a387704a-20ad-4fa6-abb8-46ffc2b82f94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.578652 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387704a-20ad-4fa6-abb8-46ffc2b82f94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn\" (UID: \"a387704a-20ad-4fa6-abb8-46ffc2b82f94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.578685 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvv7v\" (UniqueName: \"kubernetes.io/projected/0dd99f41-71bc-41f0-abed-4be3c6110a1e-kube-api-access-xvv7v\") pod \"maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl\" (UID: \"0dd99f41-71bc-41f0-abed-4be3c6110a1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.578705 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnrfw\" (UniqueName: \"kubernetes.io/projected/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-kube-api-access-dnrfw\") pod \"maas-api-key-cleanup-29888955-g2mnh\" (UID: \"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.578741 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b67626a-07e8-4b08-bee3-243b269fb5dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx\" (UID: \"7b67626a-07e8-4b08-bee3-243b269fb5dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.579549 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2\" (UID: \"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.579779 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m\" (UID: \"06dd03d6-332a-4912-b8e8-8fc243f2e8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.579813 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d\" (UID: \"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.579875 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7jz4\" (UniqueName: \"kubernetes.io/projected/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-kube-api-access-c7jz4\") pod \"maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2\" (UID: \"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" Oct 30 05:15:00.580382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.579923 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd99f41-71bc-41f0-abed-4be3c6110a1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl\" (UID: \"0dd99f41-71bc-41f0-abed-4be3c6110a1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" Oct 30 05:15:00.581261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.580643 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m\" (UID: \"06dd03d6-332a-4912-b8e8-8fc243f2e8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" Oct 30 05:15:00.588604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.588578 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" Oct 30 05:15:00.685535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.685415 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l47vc\" (UniqueName: \"kubernetes.io/projected/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-kube-api-access-l47vc\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m\" (UID: \"06dd03d6-332a-4912-b8e8-8fc243f2e8b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" Oct 30 05:15:00.702970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.702858 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7jz4\" (UniqueName: \"kubernetes.io/projected/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-kube-api-access-c7jz4\") pod \"maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2\" (UID: \"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" Oct 30 05:15:00.762896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.762842 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387704a-20ad-4fa6-abb8-46ffc2b82f94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn\" (UID: \"a387704a-20ad-4fa6-abb8-46ffc2b82f94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" Oct 30 05:15:00.764246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.763978 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" Oct 30 05:15:00.766071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.766004 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvv7v\" (UniqueName: \"kubernetes.io/projected/0dd99f41-71bc-41f0-abed-4be3c6110a1e-kube-api-access-xvv7v\") pod \"maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl\" (UID: \"0dd99f41-71bc-41f0-abed-4be3c6110a1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" Oct 30 05:15:00.767298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.766992 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" Oct 30 05:15:00.782648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.782469 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" Oct 30 05:15:00.783043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.766261 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnrfw\" (UniqueName: \"kubernetes.io/projected/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-kube-api-access-dnrfw\") pod \"maas-api-key-cleanup-29888955-g2mnh\" (UID: \"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" Oct 30 05:15:00.783043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.782766 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387704a-20ad-4fa6-abb8-46ffc2b82f94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn\" (UID: \"a387704a-20ad-4fa6-abb8-46ffc2b82f94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" Oct 30 05:15:00.783043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.782807 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" Oct 30 05:15:00.783043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.782858 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b67626a-07e8-4b08-bee3-243b269fb5dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx\" (UID: \"7b67626a-07e8-4b08-bee3-243b269fb5dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" Oct 30 05:15:00.783280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.783061 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" Oct 30 05:15:00.783280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.783097 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d\" (UID: \"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" Oct 30 05:15:00.784656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.783316 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd99f41-71bc-41f0-abed-4be3c6110a1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl\" (UID: \"0dd99f41-71bc-41f0-abed-4be3c6110a1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" Oct 30 05:15:00.797494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.797468 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzjrs\" (UniqueName: \"kubernetes.io/projected/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-kube-api-access-gzjrs\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d\" (UID: \"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" Oct 30 05:15:00.804085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.803293 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888955-g2mnh\" (UID: \"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" Oct 30 05:15:00.804085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.803489 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tm5m\" (UniqueName: \"kubernetes.io/projected/7b67626a-07e8-4b08-bee3-243b269fb5dc-kube-api-access-9tm5m\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx\" (UID: \"7b67626a-07e8-4b08-bee3-243b269fb5dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" Oct 30 05:15:00.804085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.803544 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxps\" (UniqueName: \"kubernetes.io/projected/a387704a-20ad-4fa6-abb8-46ffc2b82f94-kube-api-access-vrxps\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn\" (UID: \"a387704a-20ad-4fa6-abb8-46ffc2b82f94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" Oct 30 05:15:00.804085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.800964 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b67626a-07e8-4b08-bee3-243b269fb5dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx\" (UID: \"7b67626a-07e8-4b08-bee3-243b269fb5dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" Oct 30 05:15:00.807150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.804484 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888955-g2mnh\" (UID: \"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" Oct 30 05:15:00.807150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.801901 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d\" (UID: \"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" Oct 30 05:15:00.848384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.846495 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd99f41-71bc-41f0-abed-4be3c6110a1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl\" (UID: \"0dd99f41-71bc-41f0-abed-4be3c6110a1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" Oct 30 05:15:00.869440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.869396 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tm5m\" (UniqueName: \"kubernetes.io/projected/7b67626a-07e8-4b08-bee3-243b269fb5dc-kube-api-access-9tm5m\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx\" (UID: \"7b67626a-07e8-4b08-bee3-243b269fb5dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" Oct 30 05:15:00.870296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.870262 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxps\" (UniqueName: \"kubernetes.io/projected/a387704a-20ad-4fa6-abb8-46ffc2b82f94-kube-api-access-vrxps\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn\" (UID: \"a387704a-20ad-4fa6-abb8-46ffc2b82f94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" Oct 30 05:15:00.875228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.875169 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzjrs\" (UniqueName: \"kubernetes.io/projected/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-kube-api-access-gzjrs\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d\" (UID: \"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" Oct 30 05:15:00.882184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.882137 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvv7v\" (UniqueName: \"kubernetes.io/projected/0dd99f41-71bc-41f0-abed-4be3c6110a1e-kube-api-access-xvv7v\") pod \"maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl\" (UID: \"0dd99f41-71bc-41f0-abed-4be3c6110a1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" Oct 30 05:15:00.883089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:00.882864 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnrfw\" (UniqueName: \"kubernetes.io/projected/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-kube-api-access-dnrfw\") pod \"maas-api-key-cleanup-29888955-g2mnh\" (UID: \"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" Oct 30 05:15:01.080546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.073092 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" Oct 30 05:15:01.088761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.088725 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25"] Oct 30 05:15:01.102173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.102005 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" Oct 30 05:15:01.110030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.109969 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" Oct 30 05:15:01.143398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.142861 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" Oct 30 05:15:01.154939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.154724 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" Oct 30 05:15:01.265386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.263043 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw"] Oct 30 05:15:01.364093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.364049 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958"] Oct 30 05:15:01.374959 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:15:01.374922 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b16ea10_8c3f_40e9_8f76_c2840e19d3d0.slice/crio-e08d60a7015e0dc3106770a2447327c2ad038fc27b63151d7b1c1dd888227aa5 WatchSource:0}: Error finding container e08d60a7015e0dc3106770a2447327c2ad038fc27b63151d7b1c1dd888227aa5: Status 404 returned error can't find the container with id e08d60a7015e0dc3106770a2447327c2ad038fc27b63151d7b1c1dd888227aa5 Oct 30 05:15:01.761170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.761141 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2"] Oct 30 05:15:01.771383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.771327 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws"] Oct 30 05:15:01.806999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.806972 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m"] Oct 30 05:15:01.854900 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:15:01.854851 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94412d8_2396_4b33_a05a_5189b4ba72b5.slice/crio-6e249c87780db871d60995f8432fbad3b66cf9a5cf1837af9bae4b594ea9f8cb WatchSource:0}: Error finding container 6e249c87780db871d60995f8432fbad3b66cf9a5cf1837af9bae4b594ea9f8cb: Status 404 returned error can't find the container with id 6e249c87780db871d60995f8432fbad3b66cf9a5cf1837af9bae4b594ea9f8cb Oct 30 05:15:01.855656 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:15:01.855634 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06dd03d6_332a_4912_b8e8_8fc243f2e8b9.slice/crio-d3dd3a2ad163c5f67fd1d0202b5a0ffc67dd74dd6e7ef0dfc223bd0e49231ff0 WatchSource:0}: Error finding container d3dd3a2ad163c5f67fd1d0202b5a0ffc67dd74dd6e7ef0dfc223bd0e49231ff0: Status 404 returned error can't find the container with id d3dd3a2ad163c5f67fd1d0202b5a0ffc67dd74dd6e7ef0dfc223bd0e49231ff0 Oct 30 05:15:01.909627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.909606 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8"] Oct 30 05:15:01.957299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.957196 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh"] Oct 30 05:15:01.970081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:01.969998 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh"] Oct 30 05:15:02.013723 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:15:02.013589 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b01cc1_b83a_4fb9_abf6_276ad20b45ce.slice/crio-26eeb0086b552497be920c641a3ec836b1709620f085b0541bc846a39cf76958 WatchSource:0}: Error finding container 26eeb0086b552497be920c641a3ec836b1709620f085b0541bc846a39cf76958: Status 404 returned error can't find the container with id 26eeb0086b552497be920c641a3ec836b1709620f085b0541bc846a39cf76958 Oct 30 05:15:02.083425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:02.082561 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx"] Oct 30 05:15:02.085062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:02.085020 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" event={"ID":"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e","Type":"ContainerStarted","Data":"678067f1fa4bb5c9117b21edf968ef4dc8f0b812b46697cc62eda71ec3ebe56d"} Oct 30 05:15:02.087427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:02.087394 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" event={"ID":"01e7d7ed-2e36-4285-a316-c6b296a75cd2","Type":"ContainerStarted","Data":"2d64ac0d3fba3a218460387f40cf0bf06de72760dbb14a27b995f34d3fcee119"} Oct 30 05:15:02.089805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:02.089742 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" event={"ID":"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0","Type":"ContainerStarted","Data":"e08d60a7015e0dc3106770a2447327c2ad038fc27b63151d7b1c1dd888227aa5"} Oct 30 05:15:02.091190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:02.091168 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d"] Oct 30 05:15:02.093051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:02.093024 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" event={"ID":"dc428e2b-af51-47bb-809f-a34110a0e2ef","Type":"ContainerStarted","Data":"f4c4fd5b7433e37021b052013be0a5ad0a1b988d163a777cc1072a73e779bf0c"} Oct 30 05:15:02.096340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:02.096228 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" event={"ID":"06dd03d6-332a-4912-b8e8-8fc243f2e8b9","Type":"ContainerStarted","Data":"d3dd3a2ad163c5f67fd1d0202b5a0ffc67dd74dd6e7ef0dfc223bd0e49231ff0"} Oct 30 05:15:02.098532 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:02.098512 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn"] Oct 30 05:15:02.101816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:02.101765 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" event={"ID":"1154868f-4d78-401f-8fd0-cdd9a0fda825","Type":"ContainerStarted","Data":"9669a8759585eeaf88fe94ccd74bf890126de4fd786ff59900fc6b05d3a6db20"} Oct 30 05:15:02.103713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:02.103683 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" event={"ID":"b94412d8-2396-4b33-a05a-5189b4ba72b5","Type":"ContainerStarted","Data":"6e249c87780db871d60995f8432fbad3b66cf9a5cf1837af9bae4b594ea9f8cb"} Oct 30 05:15:02.103906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:02.103886 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl"] Oct 30 05:15:03.052377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.052028 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf"] Oct 30 05:15:03.057046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.057019 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d238c2-8p4vf"] Oct 30 05:15:03.159436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.157893 6587 generic.go:356] "Generic (PLEG): container finished" podID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" containerID="02bf6549ba66c829f41ac928204a576eec6c27e862bb4166da19acbb1233c5be" exitCode=60 Oct 30 05:15:03.159436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.158009 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" event={"ID":"01e7d7ed-2e36-4285-a316-c6b296a75cd2","Type":"ContainerDied","Data":"02bf6549ba66c829f41ac928204a576eec6c27e862bb4166da19acbb1233c5be"} Oct 30 05:15:03.159436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.158552 6587 scope.go:122] "RemoveContainer" containerID="02bf6549ba66c829f41ac928204a576eec6c27e862bb4166da19acbb1233c5be" Oct 30 05:15:03.167935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.167431 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" event={"ID":"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04","Type":"ContainerStarted","Data":"0aea7a0be9e7187f783539d29b71c6642c22aff30c0b88bd124dcc9f3d628da8"} Oct 30 05:15:03.172121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.172078 6587 generic.go:356] "Generic (PLEG): container finished" podID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" containerID="7c6520039ca98388effe07a75f1f78b4c5670087226a74726b87140d801b22e4" exitCode=60 Oct 30 05:15:03.172222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.172206 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" event={"ID":"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0","Type":"ContainerDied","Data":"7c6520039ca98388effe07a75f1f78b4c5670087226a74726b87140d801b22e4"} Oct 30 05:15:03.172653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.172634 6587 scope.go:122] "RemoveContainer" containerID="7c6520039ca98388effe07a75f1f78b4c5670087226a74726b87140d801b22e4" Oct 30 05:15:03.174223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.174200 6587 generic.go:356] "Generic (PLEG): container finished" podID="dc428e2b-af51-47bb-809f-a34110a0e2ef" containerID="7d008b20247d6be9a9a686c8e82eb5f9a24bfcfef5e4d4b5976c832c0ac6d234" exitCode=60 Oct 30 05:15:03.174336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.174294 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" event={"ID":"dc428e2b-af51-47bb-809f-a34110a0e2ef","Type":"ContainerDied","Data":"7d008b20247d6be9a9a686c8e82eb5f9a24bfcfef5e4d4b5976c832c0ac6d234"} Oct 30 05:15:03.174659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.174643 6587 scope.go:122] "RemoveContainer" containerID="7d008b20247d6be9a9a686c8e82eb5f9a24bfcfef5e4d4b5976c832c0ac6d234" Oct 30 05:15:03.176591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.176552 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" event={"ID":"06dd03d6-332a-4912-b8e8-8fc243f2e8b9","Type":"ContainerStarted","Data":"25286f3027664d923d4845bd9326ef6b7ac2ddc2be9d7c6cd64bacc09fae7c16"} Oct 30 05:15:03.180168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.180146 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" event={"ID":"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5","Type":"ContainerStarted","Data":"b76445958dbc3a7765f156a7559ab4f9dcfe82ccb84db8cd61fbe56336481a28"} Oct 30 05:15:03.180303 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.180289 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" event={"ID":"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5","Type":"ContainerStarted","Data":"43ad01c677c7f4302794511c1a3b70bd2c7ba1762032748028338f0ee45b7e73"} Oct 30 05:15:03.193393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.192951 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" event={"ID":"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce","Type":"ContainerStarted","Data":"26eeb0086b552497be920c641a3ec836b1709620f085b0541bc846a39cf76958"} Oct 30 05:15:03.195474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.194760 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" event={"ID":"1154868f-4d78-401f-8fd0-cdd9a0fda825","Type":"ContainerStarted","Data":"cd1bd824e21582832d26f08cd47cdb12bc5f08f65aa7130b086cdc57cd8b05d9"} Oct 30 05:15:03.197338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.197268 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" event={"ID":"a387704a-20ad-4fa6-abb8-46ffc2b82f94","Type":"ContainerStarted","Data":"5ce4bde4cb3b2470472edaab5215fb05ff52256c8ccd383b4f478d65c0dcd2af"} Oct 30 05:15:03.197338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.197306 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" event={"ID":"a387704a-20ad-4fa6-abb8-46ffc2b82f94","Type":"ContainerStarted","Data":"975f32d095baacfaa4e4cc2eef0dc2bca1af1533fdcc29087a34d42e38c56f7d"} Oct 30 05:15:03.199075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.199049 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" event={"ID":"b94412d8-2396-4b33-a05a-5189b4ba72b5","Type":"ContainerStarted","Data":"20d86bd3d9ffe2623fea2ab0c5eb26563fb7f7c235410eb4c703adb00e95855c"} Oct 30 05:15:03.201136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.201064 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" event={"ID":"0dd99f41-71bc-41f0-abed-4be3c6110a1e","Type":"ContainerStarted","Data":"d4bedba704effa469cd3f93987e4a1107b3c806dfda1ef9001e0b0c7a97767e5"} Oct 30 05:15:03.204023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.204001 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" event={"ID":"7b67626a-07e8-4b08-bee3-243b269fb5dc","Type":"ContainerStarted","Data":"7587d5f7e2a1768d65176c32670ff1193aaa4adf2f63ca2129c7646866747e9a"} Oct 30 05:15:03.206433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.206410 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" event={"ID":"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e","Type":"ContainerStarted","Data":"320a234554200238c423370bb2db18b94148a3ec197b71b6342980841b50c27b"} Oct 30 05:15:03.268576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.268512 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" podStartSLOduration=3.268490888 podStartE2EDuration="3.268490888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.263739474 +0000 UTC m=+1827.791506263" watchObservedRunningTime="2026-10-30 05:15:03.268490888 +0000 UTC m=+1827.796257871" Oct 30 05:15:03.322573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.315838 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" podStartSLOduration=3.3158205020000002 podStartE2EDuration="3.315820502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.312313121 +0000 UTC m=+1827.840079908" watchObservedRunningTime="2026-10-30 05:15:03.315820502 +0000 UTC m=+1827.843587289" Oct 30 05:15:03.377403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.377337 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" podStartSLOduration=3.377320167 podStartE2EDuration="3.377320167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.373190058 +0000 UTC m=+1827.900956844" watchObservedRunningTime="2026-10-30 05:15:03.377320167 +0000 UTC m=+1827.905086967" Oct 30 05:15:03.500068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.500016 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" podStartSLOduration=3.500002136 podStartE2EDuration="3.500002136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.451662236 +0000 UTC m=+1827.979429011" watchObservedRunningTime="2026-10-30 05:15:03.500002136 +0000 UTC m=+1828.027768947" Oct 30 05:15:03.564043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.560854 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" podStartSLOduration=3.560833236 podStartE2EDuration="3.560833236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.546834893 +0000 UTC m=+1828.074601682" watchObservedRunningTime="2026-10-30 05:15:03.560833236 +0000 UTC m=+1828.088600029" Oct 30 05:15:03.586549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:03.586519 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dde22e0-86b6-4456-a7eb-26e22afecd69" path="/var/lib/kubelet/pods/1dde22e0-86b6-4456-a7eb-26e22afecd69/volumes" Oct 30 05:15:04.219445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.219404 6587 generic.go:356] "Generic (PLEG): container finished" podID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" containerID="957cac571ed359cca1ebb2b354a08df0f662a58b7a97058a42dcc26087d63474" exitCode=60 Oct 30 05:15:04.219951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.219655 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" event={"ID":"0dd99f41-71bc-41f0-abed-4be3c6110a1e","Type":"ContainerDied","Data":"957cac571ed359cca1ebb2b354a08df0f662a58b7a97058a42dcc26087d63474"} Oct 30 05:15:04.220089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.220068 6587 scope.go:122] "RemoveContainer" containerID="957cac571ed359cca1ebb2b354a08df0f662a58b7a97058a42dcc26087d63474" Oct 30 05:15:04.232791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.231648 6587 generic.go:356] "Generic (PLEG): container finished" podID="7b67626a-07e8-4b08-bee3-243b269fb5dc" containerID="bff5ec6ae0a93aa30501f89bcc57604baeee605d1ded2d81f70accdc87749d0c" exitCode=60 Oct 30 05:15:04.232791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.231812 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" event={"ID":"7b67626a-07e8-4b08-bee3-243b269fb5dc","Type":"ContainerDied","Data":"bff5ec6ae0a93aa30501f89bcc57604baeee605d1ded2d81f70accdc87749d0c"} Oct 30 05:15:04.232791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.232215 6587 scope.go:122] "RemoveContainer" containerID="bff5ec6ae0a93aa30501f89bcc57604baeee605d1ded2d81f70accdc87749d0c" Oct 30 05:15:04.237537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.234885 6587 generic.go:356] "Generic (PLEG): container finished" podID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" containerID="320a234554200238c423370bb2db18b94148a3ec197b71b6342980841b50c27b" exitCode=60 Oct 30 05:15:04.237537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.235067 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" event={"ID":"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e","Type":"ContainerDied","Data":"320a234554200238c423370bb2db18b94148a3ec197b71b6342980841b50c27b"} Oct 30 05:15:04.237537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.235585 6587 scope.go:122] "RemoveContainer" containerID="320a234554200238c423370bb2db18b94148a3ec197b71b6342980841b50c27b" Oct 30 05:15:04.238003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.237978 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" event={"ID":"01e7d7ed-2e36-4285-a316-c6b296a75cd2","Type":"ContainerStarted","Data":"798150eb9ce050bd291c3c025d6a6b733cc988814ff18d91d5ac4fc43eb36ee6"} Oct 30 05:15:04.246708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.246676 6587 generic.go:356] "Generic (PLEG): container finished" podID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" containerID="c5d4d7f0c0154dfea763f1cd15ef052c2fedc102368138961f722f5be43c0cfe" exitCode=60 Oct 30 05:15:04.247189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.247170 6587 scope.go:122] "RemoveContainer" containerID="c5d4d7f0c0154dfea763f1cd15ef052c2fedc102368138961f722f5be43c0cfe" Oct 30 05:15:04.247384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.247348 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" event={"ID":"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04","Type":"ContainerDied","Data":"c5d4d7f0c0154dfea763f1cd15ef052c2fedc102368138961f722f5be43c0cfe"} Oct 30 05:15:04.251322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.251295 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" event={"ID":"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0","Type":"ContainerStarted","Data":"736ef2e0c39956a376449256255fcf35135d547949b12be3144cb7fddb35ae7b"} Oct 30 05:15:04.253858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.253835 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" event={"ID":"dc428e2b-af51-47bb-809f-a34110a0e2ef","Type":"ContainerStarted","Data":"1da754fe18cb67bc98307178f4e9185393c96bb2798bd41ed7b97ae777277e87"} Oct 30 05:15:04.256466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.256437 6587 generic.go:356] "Generic (PLEG): container finished" podID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" containerID="25286f3027664d923d4845bd9326ef6b7ac2ddc2be9d7c6cd64bacc09fae7c16" exitCode=60 Oct 30 05:15:04.256754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.256734 6587 scope.go:122] "RemoveContainer" containerID="25286f3027664d923d4845bd9326ef6b7ac2ddc2be9d7c6cd64bacc09fae7c16" Oct 30 05:15:04.256873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.256521 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" event={"ID":"06dd03d6-332a-4912-b8e8-8fc243f2e8b9","Type":"ContainerDied","Data":"25286f3027664d923d4845bd9326ef6b7ac2ddc2be9d7c6cd64bacc09fae7c16"} Oct 30 05:15:04.263889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.263807 6587 generic.go:356] "Generic (PLEG): container finished" podID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" containerID="b76445958dbc3a7765f156a7559ab4f9dcfe82ccb84db8cd61fbe56336481a28" exitCode=60 Oct 30 05:15:04.263964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.263896 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" event={"ID":"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5","Type":"ContainerDied","Data":"b76445958dbc3a7765f156a7559ab4f9dcfe82ccb84db8cd61fbe56336481a28"} Oct 30 05:15:04.264245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.264224 6587 scope.go:122] "RemoveContainer" containerID="b76445958dbc3a7765f156a7559ab4f9dcfe82ccb84db8cd61fbe56336481a28" Oct 30 05:15:04.266861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.266840 6587 generic.go:356] "Generic (PLEG): container finished" podID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" containerID="1a97ce1bb37790619d1a0744eef36580f727f1522c021dd0df815a66d3d4b744" exitCode=60 Oct 30 05:15:04.266952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.266928 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" event={"ID":"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce","Type":"ContainerDied","Data":"1a97ce1bb37790619d1a0744eef36580f727f1522c021dd0df815a66d3d4b744"} Oct 30 05:15:04.267275 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.267245 6587 scope.go:122] "RemoveContainer" containerID="1a97ce1bb37790619d1a0744eef36580f727f1522c021dd0df815a66d3d4b744" Oct 30 05:15:04.273535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.273440 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" podStartSLOduration=4.273428261 podStartE2EDuration="4.273428261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.651687175 +0000 UTC m=+1828.179453963" watchObservedRunningTime="2026-10-30 05:15:04.273428261 +0000 UTC m=+1828.801195081" Oct 30 05:15:04.277183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.277163 6587 generic.go:356] "Generic (PLEG): container finished" podID="1154868f-4d78-401f-8fd0-cdd9a0fda825" containerID="cd1bd824e21582832d26f08cd47cdb12bc5f08f65aa7130b086cdc57cd8b05d9" exitCode=60 Oct 30 05:15:04.277277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.277262 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" event={"ID":"1154868f-4d78-401f-8fd0-cdd9a0fda825","Type":"ContainerDied","Data":"cd1bd824e21582832d26f08cd47cdb12bc5f08f65aa7130b086cdc57cd8b05d9"} Oct 30 05:15:04.278771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.278733 6587 scope.go:122] "RemoveContainer" containerID="cd1bd824e21582832d26f08cd47cdb12bc5f08f65aa7130b086cdc57cd8b05d9" Oct 30 05:15:04.279840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.279817 6587 generic.go:356] "Generic (PLEG): container finished" podID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" containerID="5ce4bde4cb3b2470472edaab5215fb05ff52256c8ccd383b4f478d65c0dcd2af" exitCode=60 Oct 30 05:15:04.279934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.279896 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" event={"ID":"a387704a-20ad-4fa6-abb8-46ffc2b82f94","Type":"ContainerDied","Data":"5ce4bde4cb3b2470472edaab5215fb05ff52256c8ccd383b4f478d65c0dcd2af"} Oct 30 05:15:04.280114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.280097 6587 scope.go:122] "RemoveContainer" containerID="5ce4bde4cb3b2470472edaab5215fb05ff52256c8ccd383b4f478d65c0dcd2af" Oct 30 05:15:04.283208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.283183 6587 generic.go:356] "Generic (PLEG): container finished" podID="b94412d8-2396-4b33-a05a-5189b4ba72b5" containerID="20d86bd3d9ffe2623fea2ab0c5eb26563fb7f7c235410eb4c703adb00e95855c" exitCode=60 Oct 30 05:15:04.283292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.283248 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" event={"ID":"b94412d8-2396-4b33-a05a-5189b4ba72b5","Type":"ContainerDied","Data":"20d86bd3d9ffe2623fea2ab0c5eb26563fb7f7c235410eb4c703adb00e95855c"} Oct 30 05:15:04.283647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.283627 6587 scope.go:122] "RemoveContainer" containerID="20d86bd3d9ffe2623fea2ab0c5eb26563fb7f7c235410eb4c703adb00e95855c" Oct 30 05:15:04.467005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.466934 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" podStartSLOduration=4.466918631 podStartE2EDuration="4.466918631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:04.462315158 +0000 UTC m=+1828.990081945" watchObservedRunningTime="2026-10-30 05:15:04.466918631 +0000 UTC m=+1828.994685438" Oct 30 05:15:04.602656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.600421 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" podStartSLOduration=4.60040421 podStartE2EDuration="4.60040421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:04.597669405 +0000 UTC m=+1829.125436188" watchObservedRunningTime="2026-10-30 05:15:04.60040421 +0000 UTC m=+1829.128170995" Oct 30 05:15:04.695343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:04.695285 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" podStartSLOduration=4.695272167 podStartE2EDuration="4.695272167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:04.693714255 +0000 UTC m=+1829.221481045" watchObservedRunningTime="2026-10-30 05:15:04.695272167 +0000 UTC m=+1829.223038952" Oct 30 05:15:05.295214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.295174 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" event={"ID":"7b67626a-07e8-4b08-bee3-243b269fb5dc","Type":"ContainerStarted","Data":"06c03b7191a0ae6f26b0fe4f5f12c82094399b1d7a301b1a410eac5cdc9fe95c"} Oct 30 05:15:05.297449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.297418 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" event={"ID":"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e","Type":"ContainerStarted","Data":"22b28211d97bef55586a9471d7b97f20dc8af6b062b886ede61e37d289ddf4c6"} Oct 30 05:15:05.299425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.299397 6587 generic.go:356] "Generic (PLEG): container finished" podID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" containerID="798150eb9ce050bd291c3c025d6a6b733cc988814ff18d91d5ac4fc43eb36ee6" exitCode=60 Oct 30 05:15:05.299555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.299461 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" event={"ID":"01e7d7ed-2e36-4285-a316-c6b296a75cd2","Type":"ContainerDied","Data":"798150eb9ce050bd291c3c025d6a6b733cc988814ff18d91d5ac4fc43eb36ee6"} Oct 30 05:15:05.299555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.299505 6587 scope.go:122] "RemoveContainer" containerID="02bf6549ba66c829f41ac928204a576eec6c27e862bb4166da19acbb1233c5be" Oct 30 05:15:05.299717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.299703 6587 scope.go:122] "RemoveContainer" containerID="798150eb9ce050bd291c3c025d6a6b733cc988814ff18d91d5ac4fc43eb36ee6" Oct 30 05:15:05.300116 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:05.300090 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25_odh-ai-gateway-infra(01e7d7ed-2e36-4285-a316-c6b296a75cd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" podUID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" Oct 30 05:15:05.301814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.301789 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" event={"ID":"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04","Type":"ContainerStarted","Data":"405a625c7ee7452b10d176b876a5fcc4e02eb9c426fc580a4aa866f9ee33bdc3"} Oct 30 05:15:05.304096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.304071 6587 generic.go:356] "Generic (PLEG): container finished" podID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" containerID="736ef2e0c39956a376449256255fcf35135d547949b12be3144cb7fddb35ae7b" exitCode=60 Oct 30 05:15:05.304189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.304150 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" event={"ID":"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0","Type":"ContainerDied","Data":"736ef2e0c39956a376449256255fcf35135d547949b12be3144cb7fddb35ae7b"} Oct 30 05:15:05.304411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.304392 6587 scope.go:122] "RemoveContainer" containerID="736ef2e0c39956a376449256255fcf35135d547949b12be3144cb7fddb35ae7b" Oct 30 05:15:05.304807 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:05.304772 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958_odh-ai-gateway-infra(9b16ea10-8c3f-40e9-8f76-c2840e19d3d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" podUID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" Oct 30 05:15:05.306279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.306248 6587 generic.go:356] "Generic (PLEG): container finished" podID="dc428e2b-af51-47bb-809f-a34110a0e2ef" containerID="1da754fe18cb67bc98307178f4e9185393c96bb2798bd41ed7b97ae777277e87" exitCode=60 Oct 30 05:15:05.306435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.306274 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" event={"ID":"dc428e2b-af51-47bb-809f-a34110a0e2ef","Type":"ContainerDied","Data":"1da754fe18cb67bc98307178f4e9185393c96bb2798bd41ed7b97ae777277e87"} Oct 30 05:15:05.306602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.306584 6587 scope.go:122] "RemoveContainer" containerID="1da754fe18cb67bc98307178f4e9185393c96bb2798bd41ed7b97ae777277e87" Oct 30 05:15:05.306998 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:05.306924 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw_odh-ai-gateway-infra(dc428e2b-af51-47bb-809f-a34110a0e2ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" podUID="dc428e2b-af51-47bb-809f-a34110a0e2ef" Oct 30 05:15:05.308636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.308607 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" event={"ID":"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5","Type":"ContainerStarted","Data":"8c960fbd04a858adaf23e4bb1be15e1363ff0da211a9116e859e14c58caa4768"} Oct 30 05:15:05.311422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.311135 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" event={"ID":"1154868f-4d78-401f-8fd0-cdd9a0fda825","Type":"ContainerStarted","Data":"b5810720f4b4f6419eda6fff9ac7782967a08146d19987a8f52890863c74a881"} Oct 30 05:15:05.313438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.313410 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" event={"ID":"0dd99f41-71bc-41f0-abed-4be3c6110a1e","Type":"ContainerStarted","Data":"3609c2b5ccf090c9f30aaa0a04a21dbb0d7fb74d0d3687e0b5d05c5e0d087b8b"} Oct 30 05:15:05.345313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.345260 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" podStartSLOduration=5.345234355 podStartE2EDuration="5.345234355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:05.341829517 +0000 UTC m=+1829.869596304" watchObservedRunningTime="2026-10-30 05:15:05.345234355 +0000 UTC m=+1829.873001141" Oct 30 05:15:05.346047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.345562 6587 scope.go:122] "RemoveContainer" containerID="7c6520039ca98388effe07a75f1f78b4c5670087226a74726b87140d801b22e4" Oct 30 05:15:05.794966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:05.794938 6587 scope.go:122] "RemoveContainer" containerID="7d008b20247d6be9a9a686c8e82eb5f9a24bfcfef5e4d4b5976c832c0ac6d234" Oct 30 05:15:06.330966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.330924 6587 generic.go:356] "Generic (PLEG): container finished" podID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" containerID="22b28211d97bef55586a9471d7b97f20dc8af6b062b886ede61e37d289ddf4c6" exitCode=60 Oct 30 05:15:06.331529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.331007 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" event={"ID":"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e","Type":"ContainerDied","Data":"22b28211d97bef55586a9471d7b97f20dc8af6b062b886ede61e37d289ddf4c6"} Oct 30 05:15:06.331529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.331126 6587 scope.go:122] "RemoveContainer" containerID="320a234554200238c423370bb2db18b94148a3ec197b71b6342980841b50c27b" Oct 30 05:15:06.332032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.332011 6587 scope.go:122] "RemoveContainer" containerID="22b28211d97bef55586a9471d7b97f20dc8af6b062b886ede61e37d289ddf4c6" Oct 30 05:15:06.332498 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:06.332476 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2_odh-ai-gateway-infra(fcac5a1a-ed6a-4fc7-881c-6247bc444b0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" podUID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" Oct 30 05:15:06.334671 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.334652 6587 scope.go:122] "RemoveContainer" containerID="798150eb9ce050bd291c3c025d6a6b733cc988814ff18d91d5ac4fc43eb36ee6" Oct 30 05:15:06.335135 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:06.335105 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25_odh-ai-gateway-infra(01e7d7ed-2e36-4285-a316-c6b296a75cd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" podUID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" Oct 30 05:15:06.338091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.337014 6587 generic.go:356] "Generic (PLEG): container finished" podID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" containerID="405a625c7ee7452b10d176b876a5fcc4e02eb9c426fc580a4aa866f9ee33bdc3" exitCode=60 Oct 30 05:15:06.338091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.337108 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" event={"ID":"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04","Type":"ContainerDied","Data":"405a625c7ee7452b10d176b876a5fcc4e02eb9c426fc580a4aa866f9ee33bdc3"} Oct 30 05:15:06.338091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.337567 6587 scope.go:122] "RemoveContainer" containerID="405a625c7ee7452b10d176b876a5fcc4e02eb9c426fc580a4aa866f9ee33bdc3" Oct 30 05:15:06.338091 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:06.338015 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d_odh-ai-gateway-infra(7b15ecbd-f2d7-430f-a7d9-6e6d53328b04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" podUID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" Oct 30 05:15:06.340922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.340900 6587 scope.go:122] "RemoveContainer" containerID="736ef2e0c39956a376449256255fcf35135d547949b12be3144cb7fddb35ae7b" Oct 30 05:15:06.341275 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:06.341240 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958_odh-ai-gateway-infra(9b16ea10-8c3f-40e9-8f76-c2840e19d3d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" podUID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" Oct 30 05:15:06.343732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.343714 6587 scope.go:122] "RemoveContainer" containerID="1da754fe18cb67bc98307178f4e9185393c96bb2798bd41ed7b97ae777277e87" Oct 30 05:15:06.344065 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:06.344044 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw_odh-ai-gateway-infra(dc428e2b-af51-47bb-809f-a34110a0e2ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" podUID="dc428e2b-af51-47bb-809f-a34110a0e2ef" Oct 30 05:15:06.346615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.346589 6587 generic.go:356] "Generic (PLEG): container finished" podID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" containerID="223ddf3a27d24df61d2f937d7680f124ed51d553b4e1dac76949038018efb5bd" exitCode=60 Oct 30 05:15:06.346855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.346799 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" event={"ID":"06dd03d6-332a-4912-b8e8-8fc243f2e8b9","Type":"ContainerDied","Data":"223ddf3a27d24df61d2f937d7680f124ed51d553b4e1dac76949038018efb5bd"} Oct 30 05:15:06.347413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.347063 6587 scope.go:122] "RemoveContainer" containerID="223ddf3a27d24df61d2f937d7680f124ed51d553b4e1dac76949038018efb5bd" Oct 30 05:15:06.347562 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:06.347488 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m_odh-ai-gateway-infra(06dd03d6-332a-4912-b8e8-8fc243f2e8b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" podUID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" Oct 30 05:15:06.349444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.349421 6587 generic.go:356] "Generic (PLEG): container finished" podID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" containerID="8c960fbd04a858adaf23e4bb1be15e1363ff0da211a9116e859e14c58caa4768" exitCode=60 Oct 30 05:15:06.349535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.349509 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" event={"ID":"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5","Type":"ContainerDied","Data":"8c960fbd04a858adaf23e4bb1be15e1363ff0da211a9116e859e14c58caa4768"} Oct 30 05:15:06.349812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.349792 6587 scope.go:122] "RemoveContainer" containerID="8c960fbd04a858adaf23e4bb1be15e1363ff0da211a9116e859e14c58caa4768" Oct 30 05:15:06.350086 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:06.350064 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888955-g2mnh_odh-ai-gateway-infra(dc3f6c58-ebd5-46d0-b024-7644ec3f06d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" podUID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" Oct 30 05:15:06.351185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.351166 6587 scope.go:122] "RemoveContainer" containerID="c5d4d7f0c0154dfea763f1cd15ef052c2fedc102368138961f722f5be43c0cfe" Oct 30 05:15:06.356219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.356198 6587 generic.go:356] "Generic (PLEG): container finished" podID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" containerID="8a96ded5ea72defc01bb254fff2fe790d99d778ee2501ec8359b4871eb6de39e" exitCode=60 Oct 30 05:15:06.356320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.356299 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" event={"ID":"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce","Type":"ContainerDied","Data":"8a96ded5ea72defc01bb254fff2fe790d99d778ee2501ec8359b4871eb6de39e"} Oct 30 05:15:06.356991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.356972 6587 scope.go:122] "RemoveContainer" containerID="8a96ded5ea72defc01bb254fff2fe790d99d778ee2501ec8359b4871eb6de39e" Oct 30 05:15:06.357328 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:06.357300 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh_odh-ai-gateway-infra(d6b01cc1-b83a-4fb9-abf6-276ad20b45ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" podUID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" Oct 30 05:15:06.359570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.359436 6587 generic.go:356] "Generic (PLEG): container finished" podID="1154868f-4d78-401f-8fd0-cdd9a0fda825" containerID="b5810720f4b4f6419eda6fff9ac7782967a08146d19987a8f52890863c74a881" exitCode=60 Oct 30 05:15:06.359570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.359497 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" event={"ID":"1154868f-4d78-401f-8fd0-cdd9a0fda825","Type":"ContainerDied","Data":"b5810720f4b4f6419eda6fff9ac7782967a08146d19987a8f52890863c74a881"} Oct 30 05:15:06.360570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.360506 6587 scope.go:122] "RemoveContainer" containerID="b5810720f4b4f6419eda6fff9ac7782967a08146d19987a8f52890863c74a881" Oct 30 05:15:06.361071 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:06.361015 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8_odh-ai-gateway-infra(1154868f-4d78-401f-8fd0-cdd9a0fda825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" podUID="1154868f-4d78-401f-8fd0-cdd9a0fda825" Oct 30 05:15:06.363925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.362644 6587 generic.go:356] "Generic (PLEG): container finished" podID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" containerID="ad4ea5496aecf07624576329fb866436e719f8082cc736d91ca2a105671bdd21" exitCode=60 Oct 30 05:15:06.363925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.362737 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" event={"ID":"a387704a-20ad-4fa6-abb8-46ffc2b82f94","Type":"ContainerDied","Data":"ad4ea5496aecf07624576329fb866436e719f8082cc736d91ca2a105671bdd21"} Oct 30 05:15:06.363925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.362947 6587 scope.go:122] "RemoveContainer" containerID="ad4ea5496aecf07624576329fb866436e719f8082cc736d91ca2a105671bdd21" Oct 30 05:15:06.363925 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:06.363199 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn_odh-ai-gateway-infra(a387704a-20ad-4fa6-abb8-46ffc2b82f94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" podUID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" Oct 30 05:15:06.366320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.366292 6587 generic.go:356] "Generic (PLEG): container finished" podID="b94412d8-2396-4b33-a05a-5189b4ba72b5" containerID="6824c07d794ae58959cb19071c960d7d371d31583e692f203eb98897a1a4081b" exitCode=60 Oct 30 05:15:06.366443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.366419 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" event={"ID":"b94412d8-2396-4b33-a05a-5189b4ba72b5","Type":"ContainerDied","Data":"6824c07d794ae58959cb19071c960d7d371d31583e692f203eb98897a1a4081b"} Oct 30 05:15:06.366646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.366626 6587 scope.go:122] "RemoveContainer" containerID="6824c07d794ae58959cb19071c960d7d371d31583e692f203eb98897a1a4081b" Oct 30 05:15:06.366927 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:06.366900 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws_odh-ai-gateway-infra(b94412d8-2396-4b33-a05a-5189b4ba72b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" podUID="b94412d8-2396-4b33-a05a-5189b4ba72b5" Oct 30 05:15:06.368784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.368759 6587 generic.go:356] "Generic (PLEG): container finished" podID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" containerID="3609c2b5ccf090c9f30aaa0a04a21dbb0d7fb74d0d3687e0b5d05c5e0d087b8b" exitCode=60 Oct 30 05:15:06.368887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.368832 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" event={"ID":"0dd99f41-71bc-41f0-abed-4be3c6110a1e","Type":"ContainerDied","Data":"3609c2b5ccf090c9f30aaa0a04a21dbb0d7fb74d0d3687e0b5d05c5e0d087b8b"} Oct 30 05:15:06.369046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.369029 6587 scope.go:122] "RemoveContainer" containerID="3609c2b5ccf090c9f30aaa0a04a21dbb0d7fb74d0d3687e0b5d05c5e0d087b8b" Oct 30 05:15:06.369299 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:06.369277 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl_odh-ai-gateway-infra(0dd99f41-71bc-41f0-abed-4be3c6110a1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" podUID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" Oct 30 05:15:06.371229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.371210 6587 generic.go:356] "Generic (PLEG): container finished" podID="7b67626a-07e8-4b08-bee3-243b269fb5dc" containerID="06c03b7191a0ae6f26b0fe4f5f12c82094399b1d7a301b1a410eac5cdc9fe95c" exitCode=60 Oct 30 05:15:06.371342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.371249 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" event={"ID":"7b67626a-07e8-4b08-bee3-243b269fb5dc","Type":"ContainerDied","Data":"06c03b7191a0ae6f26b0fe4f5f12c82094399b1d7a301b1a410eac5cdc9fe95c"} Oct 30 05:15:06.371571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.371554 6587 scope.go:122] "RemoveContainer" containerID="06c03b7191a0ae6f26b0fe4f5f12c82094399b1d7a301b1a410eac5cdc9fe95c" Oct 30 05:15:06.371912 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:06.371826 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx_odh-ai-gateway-infra(7b67626a-07e8-4b08-bee3-243b269fb5dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" podUID="7b67626a-07e8-4b08-bee3-243b269fb5dc" Oct 30 05:15:06.371963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.371951 6587 scope.go:122] "RemoveContainer" containerID="25286f3027664d923d4845bd9326ef6b7ac2ddc2be9d7c6cd64bacc09fae7c16" Oct 30 05:15:06.388233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.388209 6587 scope.go:122] "RemoveContainer" containerID="b76445958dbc3a7765f156a7559ab4f9dcfe82ccb84db8cd61fbe56336481a28" Oct 30 05:15:06.402895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.402868 6587 scope.go:122] "RemoveContainer" containerID="1a97ce1bb37790619d1a0744eef36580f727f1522c021dd0df815a66d3d4b744" Oct 30 05:15:06.421585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.421528 6587 scope.go:122] "RemoveContainer" containerID="cd1bd824e21582832d26f08cd47cdb12bc5f08f65aa7130b086cdc57cd8b05d9" Oct 30 05:15:06.441068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.441043 6587 scope.go:122] "RemoveContainer" containerID="5ce4bde4cb3b2470472edaab5215fb05ff52256c8ccd383b4f478d65c0dcd2af" Oct 30 05:15:06.474631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.474599 6587 scope.go:122] "RemoveContainer" containerID="20d86bd3d9ffe2623fea2ab0c5eb26563fb7f7c235410eb4c703adb00e95855c" Oct 30 05:15:06.492017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.491989 6587 scope.go:122] "RemoveContainer" containerID="957cac571ed359cca1ebb2b354a08df0f662a58b7a97058a42dcc26087d63474" Oct 30 05:15:06.509566 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.509536 6587 scope.go:122] "RemoveContainer" containerID="bff5ec6ae0a93aa30501f89bcc57604baeee605d1ded2d81f70accdc87749d0c" Oct 30 05:15:06.604838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.604728 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" podStartSLOduration=6.604710499 podStartE2EDuration="6.604710499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:06.603411256 +0000 UTC m=+1831.131178062" watchObservedRunningTime="2026-10-30 05:15:06.604710499 +0000 UTC m=+1831.132477284" Oct 30 05:15:06.748559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:06.748505 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" podStartSLOduration=6.748490124 podStartE2EDuration="6.748490124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:06.737335593 +0000 UTC m=+1831.265102378" watchObservedRunningTime="2026-10-30 05:15:06.748490124 +0000 UTC m=+1831.276256956" Oct 30 05:15:07.382251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:07.382218 6587 scope.go:122] "RemoveContainer" containerID="6824c07d794ae58959cb19071c960d7d371d31583e692f203eb98897a1a4081b" Oct 30 05:15:07.382739 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:07.382703 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws_odh-ai-gateway-infra(b94412d8-2396-4b33-a05a-5189b4ba72b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" podUID="b94412d8-2396-4b33-a05a-5189b4ba72b5" Oct 30 05:15:07.384033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:07.384009 6587 scope.go:122] "RemoveContainer" containerID="3609c2b5ccf090c9f30aaa0a04a21dbb0d7fb74d0d3687e0b5d05c5e0d087b8b" Oct 30 05:15:07.384433 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:07.384400 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl_odh-ai-gateway-infra(0dd99f41-71bc-41f0-abed-4be3c6110a1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" podUID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" Oct 30 05:15:07.385909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:07.385885 6587 scope.go:122] "RemoveContainer" containerID="06c03b7191a0ae6f26b0fe4f5f12c82094399b1d7a301b1a410eac5cdc9fe95c" Oct 30 05:15:07.386275 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:07.386244 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx_odh-ai-gateway-infra(7b67626a-07e8-4b08-bee3-243b269fb5dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" podUID="7b67626a-07e8-4b08-bee3-243b269fb5dc" Oct 30 05:15:07.387459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:07.387435 6587 scope.go:122] "RemoveContainer" containerID="22b28211d97bef55586a9471d7b97f20dc8af6b062b886ede61e37d289ddf4c6" Oct 30 05:15:07.387755 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:07.387730 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2_odh-ai-gateway-infra(fcac5a1a-ed6a-4fc7-881c-6247bc444b0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" podUID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" Oct 30 05:15:07.389055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:07.389037 6587 scope.go:122] "RemoveContainer" containerID="405a625c7ee7452b10d176b876a5fcc4e02eb9c426fc580a4aa866f9ee33bdc3" Oct 30 05:15:07.389289 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:07.389269 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d_odh-ai-gateway-infra(7b15ecbd-f2d7-430f-a7d9-6e6d53328b04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" podUID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" Oct 30 05:15:07.390849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:07.390832 6587 scope.go:122] "RemoveContainer" containerID="223ddf3a27d24df61d2f937d7680f124ed51d553b4e1dac76949038018efb5bd" Oct 30 05:15:07.391071 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:07.391044 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m_odh-ai-gateway-infra(06dd03d6-332a-4912-b8e8-8fc243f2e8b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" podUID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" Oct 30 05:15:07.392548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:07.392529 6587 scope.go:122] "RemoveContainer" containerID="8c960fbd04a858adaf23e4bb1be15e1363ff0da211a9116e859e14c58caa4768" Oct 30 05:15:07.392804 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:07.392784 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888955-g2mnh_odh-ai-gateway-infra(dc3f6c58-ebd5-46d0-b024-7644ec3f06d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" podUID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" Oct 30 05:15:07.394742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:07.394723 6587 scope.go:122] "RemoveContainer" containerID="8a96ded5ea72defc01bb254fff2fe790d99d778ee2501ec8359b4871eb6de39e" Oct 30 05:15:07.394974 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:07.394955 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh_odh-ai-gateway-infra(d6b01cc1-b83a-4fb9-abf6-276ad20b45ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" podUID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" Oct 30 05:15:07.397051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:07.397024 6587 scope.go:122] "RemoveContainer" containerID="b5810720f4b4f6419eda6fff9ac7782967a08146d19987a8f52890863c74a881" Oct 30 05:15:07.397399 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:07.397324 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8_odh-ai-gateway-infra(1154868f-4d78-401f-8fd0-cdd9a0fda825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" podUID="1154868f-4d78-401f-8fd0-cdd9a0fda825" Oct 30 05:15:07.398996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:07.398978 6587 scope.go:122] "RemoveContainer" containerID="ad4ea5496aecf07624576329fb866436e719f8082cc736d91ca2a105671bdd21" Oct 30 05:15:07.401033 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:07.399747 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn_odh-ai-gateway-infra(a387704a-20ad-4fa6-abb8-46ffc2b82f94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" podUID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" Oct 30 05:15:07.912711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:07.912667 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478"] Oct 30 05:15:07.913209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:07.913168 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" podUID="7d74c3e2-27d2-4650-97ea-7c99efe8f924" containerName="istio-proxy" containerID="cri-o://e4692988d03f1fec873183e3610f3db84c28bb99c45bb826458d22306d3c87fb" gracePeriod=30 Oct 30 05:15:09.477382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:09.477147 6587 generic.go:356] "Generic (PLEG): container finished" podID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerID="8c851debf1dd9da389a6ab50742afa32f1556ac8fabe2b89a39b5d92266a3d52" exitCode=0 Oct 30 05:15:09.477382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:09.477289 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" event={"ID":"e5e4e00a-ee98-4cf5-a016-cbf70647fea9","Type":"ContainerDied","Data":"8c851debf1dd9da389a6ab50742afa32f1556ac8fabe2b89a39b5d92266a3d52"} Oct 30 05:15:09.478049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:09.477875 6587 scope.go:122] "RemoveContainer" containerID="8c851debf1dd9da389a6ab50742afa32f1556ac8fabe2b89a39b5d92266a3d52" Oct 30 05:15:09.514392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:09.512080 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:09.514392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:09.512207 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:09.514392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:09.512262 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:10.470589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:10.470551 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8"] Oct 30 05:15:10.470806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:10.470785 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" podUID="14e81da2-cc1a-4b99-9a85-ee73d843b8ad" containerName="istio-proxy" containerID="cri-o://c3d8715d05837259ecd239198bac5a25f074df2968188eb811e82ad8d2fc37ef" gracePeriod=30 Oct 30 05:15:10.490765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:10.490720 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" event={"ID":"e5e4e00a-ee98-4cf5-a016-cbf70647fea9","Type":"ContainerStarted","Data":"9e53bd822b72b2677720d4076af28b0df85381bd517fe96b5103efc2f11dfaa4"} Oct 30 05:15:10.833277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:10.833235 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:15:10.834913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:10.834883 6587 patch_prober.go:29] interesting pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:15:10.835076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:10.834939 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 05:15:11.841396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:11.838969 6587 patch_prober.go:29] interesting pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:15:11.841396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:11.839034 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 05:15:12.838386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:12.836885 6587 patch_prober.go:29] interesting pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:15:12.838386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:12.836969 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 05:15:13.366274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.365929 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:15:13.480434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.480391 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-credential-socket\") pod \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " Oct 30 05:15:13.480625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.480474 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-data\") pod \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " Oct 30 05:15:13.480625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.480509 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istiod-ca-cert\") pod \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " Oct 30 05:15:13.480625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.480539 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-envoy\") pod \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " Oct 30 05:15:13.480813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.480671 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-credential-socket" pod "7d74c3e2-27d2-4650-97ea-7c99efe8f924" (UID: "7d74c3e2-27d2-4650-97ea-7c99efe8f924"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:13.480813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.480697 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-socket\") pod \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " Oct 30 05:15:13.480813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.480745 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-podinfo\") pod \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " Oct 30 05:15:13.480813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.480814 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-certs\") pod \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " Oct 30 05:15:13.481087 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.480853 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-token\" (UniqueName: \"kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-token\") pod \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " Oct 30 05:15:13.481087 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.480873 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-kube-api-access-flfjj\" (UniqueName: \"kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-kube-api-access-flfjj\") pod \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\" (UID: \"7d74c3e2-27d2-4650-97ea-7c99efe8f924\") " Oct 30 05:15:13.481087 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.481025 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istiod-ca-cert" pod "7d74c3e2-27d2-4650-97ea-7c99efe8f924" (UID: "7d74c3e2-27d2-4650-97ea-7c99efe8f924"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:13.481304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.481282 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-data" pod "7d74c3e2-27d2-4650-97ea-7c99efe8f924" (UID: "7d74c3e2-27d2-4650-97ea-7c99efe8f924"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:13.481885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.481857 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:13.482006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.481888 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:13.482006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.481903 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:13.482512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.482459 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-certs" pod "7d74c3e2-27d2-4650-97ea-7c99efe8f924" (UID: "7d74c3e2-27d2-4650-97ea-7c99efe8f924"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:13.482702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.482670 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-socket" pod "7d74c3e2-27d2-4650-97ea-7c99efe8f924" (UID: "7d74c3e2-27d2-4650-97ea-7c99efe8f924"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:13.484100 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.484069 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-kube-api-access-flfjj" pod "7d74c3e2-27d2-4650-97ea-7c99efe8f924" (UID: "7d74c3e2-27d2-4650-97ea-7c99efe8f924"). InnerVolumeSpecName "kube-api-access-flfjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:13.484491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.484313 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-envoy" pod "7d74c3e2-27d2-4650-97ea-7c99efe8f924" (UID: "7d74c3e2-27d2-4650-97ea-7c99efe8f924"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:13.484700 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.484680 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-podinfo" pod "7d74c3e2-27d2-4650-97ea-7c99efe8f924" (UID: "7d74c3e2-27d2-4650-97ea-7c99efe8f924"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:15:13.484905 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.484876 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-token" pod "7d74c3e2-27d2-4650-97ea-7c99efe8f924" (UID: "7d74c3e2-27d2-4650-97ea-7c99efe8f924"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:13.569383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.568672 6587 generic.go:356] "Generic (PLEG): container finished" podID="7d74c3e2-27d2-4650-97ea-7c99efe8f924" containerID="e4692988d03f1fec873183e3610f3db84c28bb99c45bb826458d22306d3c87fb" exitCode=0 Oct 30 05:15:13.569383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.568934 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" event={"ID":"7d74c3e2-27d2-4650-97ea-7c99efe8f924","Type":"ContainerDied","Data":"e4692988d03f1fec873183e3610f3db84c28bb99c45bb826458d22306d3c87fb"} Oct 30 05:15:13.569383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.569024 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" event={"ID":"7d74c3e2-27d2-4650-97ea-7c99efe8f924","Type":"ContainerDied","Data":"677aee15e62845d3b4135c9727083ef0e199c36a6fab09f8ceee95c2146a7778"} Oct 30 05:15:13.569383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.569056 6587 scope.go:122] "RemoveContainer" containerID="e4692988d03f1fec873183e3610f3db84c28bb99c45bb826458d22306d3c87fb" Oct 30 05:15:13.569383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.569230 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478" Oct 30 05:15:13.586618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.584971 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:13.586618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.584990 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:13.586618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.585001 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:13.586618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.585011 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:13.586618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.585019 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flfjj\" (UniqueName: \"kubernetes.io/projected/7d74c3e2-27d2-4650-97ea-7c99efe8f924-kube-api-access-flfjj\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:13.586618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.585027 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d74c3e2-27d2-4650-97ea-7c99efe8f924-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:13.598209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.598192 6587 scope.go:122] "RemoveContainer" containerID="e4692988d03f1fec873183e3610f3db84c28bb99c45bb826458d22306d3c87fb" Oct 30 05:15:13.598555 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:13.598528 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4692988d03f1fec873183e3610f3db84c28bb99c45bb826458d22306d3c87fb\": container with ID starting with e4692988d03f1fec873183e3610f3db84c28bb99c45bb826458d22306d3c87fb not found: ID does not exist" containerID="e4692988d03f1fec873183e3610f3db84c28bb99c45bb826458d22306d3c87fb" Oct 30 05:15:13.598620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.598569 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4692988d03f1fec873183e3610f3db84c28bb99c45bb826458d22306d3c87fb"} err="failed to get container status \"e4692988d03f1fec873183e3610f3db84c28bb99c45bb826458d22306d3c87fb\": rpc error: code = NotFound desc = could not find container \"e4692988d03f1fec873183e3610f3db84c28bb99c45bb826458d22306d3c87fb\": container with ID starting with e4692988d03f1fec873183e3610f3db84c28bb99c45bb826458d22306d3c87fb not found: ID does not exist" Oct 30 05:15:13.665519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.664630 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478"] Oct 30 05:15:13.669191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.669162 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478"] Oct 30 05:15:13.841425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.840712 6587 patch_prober.go:29] interesting pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:15:13.841425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.840786 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 05:15:13.856853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:13.856813 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:15:14.848382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:14.834698 6587 patch_prober.go:29] interesting pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:15:14.848382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:14.834771 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 05:15:15.601028 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:15.600597 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d74c3e2-27d2-4650-97ea-7c99efe8f924" path="/var/lib/kubelet/pods/7d74c3e2-27d2-4650-97ea-7c99efe8f924/volumes" Oct 30 05:15:15.604882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:15.604858 6587 generic.go:356] "Generic (PLEG): container finished" podID="14e81da2-cc1a-4b99-9a85-ee73d843b8ad" containerID="c3d8715d05837259ecd239198bac5a25f074df2968188eb811e82ad8d2fc37ef" exitCode=0 Oct 30 05:15:15.604992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:15.604925 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" event={"ID":"14e81da2-cc1a-4b99-9a85-ee73d843b8ad","Type":"ContainerDied","Data":"c3d8715d05837259ecd239198bac5a25f074df2968188eb811e82ad8d2fc37ef"} Oct 30 05:15:15.879380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:15.879285 6587 patch_prober.go:29] interesting pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:15:15.879380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:15.879346 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 05:15:16.257297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.257270 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 05:15:16.349379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.349045 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-token\" (UniqueName: \"kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-token\") pod \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " Oct 30 05:15:16.349379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.349134 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-podinfo\") pod \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " Oct 30 05:15:16.349379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.349215 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-certs\") pod \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " Oct 30 05:15:16.349379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.349285 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istiod-ca-cert\") pod \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " Oct 30 05:15:16.349379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.349380 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-socket\") pod \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " Oct 30 05:15:16.349796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.349413 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-credential-socket\") pod \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " Oct 30 05:15:16.349796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.349512 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-envoy\") pod \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " Oct 30 05:15:16.349796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.349605 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-kube-api-access-6h6jz\" (UniqueName: \"kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-kube-api-access-6h6jz\") pod \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " Oct 30 05:15:16.349796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.349691 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-data\") pod \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\" (UID: \"14e81da2-cc1a-4b99-9a85-ee73d843b8ad\") " Oct 30 05:15:16.359381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.359255 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istiod-ca-cert" pod "14e81da2-cc1a-4b99-9a85-ee73d843b8ad" (UID: "14e81da2-cc1a-4b99-9a85-ee73d843b8ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:16.359678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.359646 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-socket" pod "14e81da2-cc1a-4b99-9a85-ee73d843b8ad" (UID: "14e81da2-cc1a-4b99-9a85-ee73d843b8ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:16.360086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.360048 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-credential-socket" pod "14e81da2-cc1a-4b99-9a85-ee73d843b8ad" (UID: "14e81da2-cc1a-4b99-9a85-ee73d843b8ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:16.360086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.360070 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-certs" pod "14e81da2-cc1a-4b99-9a85-ee73d843b8ad" (UID: "14e81da2-cc1a-4b99-9a85-ee73d843b8ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:16.374645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.366512 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-kube-api-access-6h6jz" pod "14e81da2-cc1a-4b99-9a85-ee73d843b8ad" (UID: "14e81da2-cc1a-4b99-9a85-ee73d843b8ad"). InnerVolumeSpecName "kube-api-access-6h6jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:16.374645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.366923 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-token" pod "14e81da2-cc1a-4b99-9a85-ee73d843b8ad" (UID: "14e81da2-cc1a-4b99-9a85-ee73d843b8ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:16.374645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.366991 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-data" pod "14e81da2-cc1a-4b99-9a85-ee73d843b8ad" (UID: "14e81da2-cc1a-4b99-9a85-ee73d843b8ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:16.379237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.379195 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-envoy" pod "14e81da2-cc1a-4b99-9a85-ee73d843b8ad" (UID: "14e81da2-cc1a-4b99-9a85-ee73d843b8ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:16.388351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.388324 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-podinfo" pod "14e81da2-cc1a-4b99-9a85-ee73d843b8ad" (UID: "14e81da2-cc1a-4b99-9a85-ee73d843b8ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:15:16.460956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.460821 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:16.461200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.461176 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:16.461392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.461372 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h6jz\" (UniqueName: \"kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-kube-api-access-6h6jz\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:16.461509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.461487 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:16.461509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.461508 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:16.461621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.461521 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:16.461621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.461536 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:16.461621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.461550 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:16.461621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.461563 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e81da2-cc1a-4b99-9a85-ee73d843b8ad-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:16.573798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.573761 6587 scope.go:122] "RemoveContainer" containerID="8c960fbd04a858adaf23e4bb1be15e1363ff0da211a9116e859e14c58caa4768" Oct 30 05:15:16.622784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.622588 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" Oct 30 05:15:16.622784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.622723 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" event={"ID":"14e81da2-cc1a-4b99-9a85-ee73d843b8ad","Type":"ContainerDied","Data":"b801ecf3309d16f21201e9d1bea251dcef14cb3211ef85b2004e3de842cc4066"} Oct 30 05:15:16.623005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.622791 6587 scope.go:122] "RemoveContainer" containerID="c3d8715d05837259ecd239198bac5a25f074df2968188eb811e82ad8d2fc37ef" Oct 30 05:15:16.686890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.686857 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8"] Oct 30 05:15:16.689723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.689691 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8"] Oct 30 05:15:16.928403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.925729 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:15:16.928403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.927789 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:15:16.992803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.992772 6587 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:15:16.992960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:16.992820 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8" podUID="14e81da2-cc1a-4b99-9a85-ee73d843b8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:15:17.575633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:17.575069 6587 scope.go:122] "RemoveContainer" containerID="1da754fe18cb67bc98307178f4e9185393c96bb2798bd41ed7b97ae777277e87" Oct 30 05:15:17.575633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:17.575198 6587 scope.go:122] "RemoveContainer" containerID="798150eb9ce050bd291c3c025d6a6b733cc988814ff18d91d5ac4fc43eb36ee6" Oct 30 05:15:17.575633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:17.575302 6587 scope.go:122] "RemoveContainer" containerID="ad4ea5496aecf07624576329fb866436e719f8082cc736d91ca2a105671bdd21" Oct 30 05:15:17.575633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:17.575409 6587 scope.go:122] "RemoveContainer" containerID="736ef2e0c39956a376449256255fcf35135d547949b12be3144cb7fddb35ae7b" Oct 30 05:15:17.575978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:17.575936 6587 scope.go:122] "RemoveContainer" containerID="6824c07d794ae58959cb19071c960d7d371d31583e692f203eb98897a1a4081b" Oct 30 05:15:17.576407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:17.576176 6587 scope.go:122] "RemoveContainer" containerID="06c03b7191a0ae6f26b0fe4f5f12c82094399b1d7a301b1a410eac5cdc9fe95c" Oct 30 05:15:17.593049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:17.593026 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e81da2-cc1a-4b99-9a85-ee73d843b8ad" path="/var/lib/kubelet/pods/14e81da2-cc1a-4b99-9a85-ee73d843b8ad/volumes" Oct 30 05:15:17.663895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:17.663691 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" event={"ID":"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5","Type":"ContainerStarted","Data":"ede0ae55f6022d8b359f2a4a7c5b2be471027d3b6f653464ddc40c406ca9eedb"} Oct 30 05:15:18.574748 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:18.574723 6587 scope.go:122] "RemoveContainer" containerID="223ddf3a27d24df61d2f937d7680f124ed51d553b4e1dac76949038018efb5bd" Oct 30 05:15:18.674859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:18.674823 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" event={"ID":"a387704a-20ad-4fa6-abb8-46ffc2b82f94","Type":"ContainerStarted","Data":"54b1836a3c1784392da6b10c4bebc345643581cbff4ef8303f9b3ce394061e8f"} Oct 30 05:15:18.676999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:18.676962 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" event={"ID":"b94412d8-2396-4b33-a05a-5189b4ba72b5","Type":"ContainerStarted","Data":"8b4d6128e69361c1eef7512e664eb5000ab38b6184ac45b4784b2d3a5ac0e3b7"} Oct 30 05:15:18.679156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:18.679131 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" event={"ID":"7b67626a-07e8-4b08-bee3-243b269fb5dc","Type":"ContainerStarted","Data":"3c6fef9c132c8d8c6305406f6c42e4cc01532f34c3194d52587ed1e8a47a9746"} Oct 30 05:15:18.681307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:18.681281 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" event={"ID":"01e7d7ed-2e36-4285-a316-c6b296a75cd2","Type":"ContainerStarted","Data":"c9018121b705385ff6932f925bf61844ded989a944b8dce6dc4f9ee9f2a6678b"} Oct 30 05:15:18.683322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:18.683303 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" event={"ID":"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0","Type":"ContainerStarted","Data":"6d8e256cb49b202868c2d0a30aac271b952651d4afd84bfb5dc2b24a188ea296"} Oct 30 05:15:18.685341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:18.685320 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" event={"ID":"dc428e2b-af51-47bb-809f-a34110a0e2ef","Type":"ContainerStarted","Data":"82b08e3748358e85d42c0089d9dc157238aaa16b338932e9410e5858de4c7519"} Oct 30 05:15:18.687328 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:18.687302 6587 generic.go:356] "Generic (PLEG): container finished" podID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" containerID="ede0ae55f6022d8b359f2a4a7c5b2be471027d3b6f653464ddc40c406ca9eedb" exitCode=60 Oct 30 05:15:18.687465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:18.687343 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" event={"ID":"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5","Type":"ContainerDied","Data":"ede0ae55f6022d8b359f2a4a7c5b2be471027d3b6f653464ddc40c406ca9eedb"} Oct 30 05:15:18.687465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:18.687417 6587 scope.go:122] "RemoveContainer" containerID="8c960fbd04a858adaf23e4bb1be15e1363ff0da211a9116e859e14c58caa4768" Oct 30 05:15:18.687715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:18.687696 6587 scope.go:122] "RemoveContainer" containerID="ede0ae55f6022d8b359f2a4a7c5b2be471027d3b6f653464ddc40c406ca9eedb" Oct 30 05:15:18.688012 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:18.687992 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888955-g2mnh_odh-ai-gateway-infra(dc3f6c58-ebd5-46d0-b024-7644ec3f06d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" podUID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" Oct 30 05:15:18.720681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:18.715893 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh"] Oct 30 05:15:19.575012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.574972 6587 scope.go:122] "RemoveContainer" containerID="3609c2b5ccf090c9f30aaa0a04a21dbb0d7fb74d0d3687e0b5d05c5e0d087b8b" Oct 30 05:15:19.576035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.576005 6587 scope.go:122] "RemoveContainer" containerID="b5810720f4b4f6419eda6fff9ac7782967a08146d19987a8f52890863c74a881" Oct 30 05:15:19.576829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.576804 6587 scope.go:122] "RemoveContainer" containerID="22b28211d97bef55586a9471d7b97f20dc8af6b062b886ede61e37d289ddf4c6" Oct 30 05:15:19.583775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.583341 6587 scope.go:122] "RemoveContainer" containerID="405a625c7ee7452b10d176b876a5fcc4e02eb9c426fc580a4aa866f9ee33bdc3" Oct 30 05:15:19.711252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.711205 6587 generic.go:356] "Generic (PLEG): container finished" podID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" containerID="c9018121b705385ff6932f925bf61844ded989a944b8dce6dc4f9ee9f2a6678b" exitCode=60 Oct 30 05:15:19.711532 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.711428 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" event={"ID":"01e7d7ed-2e36-4285-a316-c6b296a75cd2","Type":"ContainerDied","Data":"c9018121b705385ff6932f925bf61844ded989a944b8dce6dc4f9ee9f2a6678b"} Oct 30 05:15:19.711532 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.711520 6587 scope.go:122] "RemoveContainer" containerID="798150eb9ce050bd291c3c025d6a6b733cc988814ff18d91d5ac4fc43eb36ee6" Oct 30 05:15:19.712737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.712686 6587 scope.go:122] "RemoveContainer" containerID="c9018121b705385ff6932f925bf61844ded989a944b8dce6dc4f9ee9f2a6678b" Oct 30 05:15:19.713424 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:19.713280 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25_odh-ai-gateway-infra(01e7d7ed-2e36-4285-a316-c6b296a75cd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" podUID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" Oct 30 05:15:19.718899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.718867 6587 generic.go:356] "Generic (PLEG): container finished" podID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" containerID="6d8e256cb49b202868c2d0a30aac271b952651d4afd84bfb5dc2b24a188ea296" exitCode=60 Oct 30 05:15:19.719011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.718972 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" event={"ID":"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0","Type":"ContainerDied","Data":"6d8e256cb49b202868c2d0a30aac271b952651d4afd84bfb5dc2b24a188ea296"} Oct 30 05:15:19.719601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.719562 6587 scope.go:122] "RemoveContainer" containerID="6d8e256cb49b202868c2d0a30aac271b952651d4afd84bfb5dc2b24a188ea296" Oct 30 05:15:19.720100 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:19.720069 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958_odh-ai-gateway-infra(9b16ea10-8c3f-40e9-8f76-c2840e19d3d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" podUID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" Oct 30 05:15:19.728077 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.726296 6587 generic.go:356] "Generic (PLEG): container finished" podID="dc428e2b-af51-47bb-809f-a34110a0e2ef" containerID="82b08e3748358e85d42c0089d9dc157238aaa16b338932e9410e5858de4c7519" exitCode=60 Oct 30 05:15:19.728077 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.726443 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" event={"ID":"dc428e2b-af51-47bb-809f-a34110a0e2ef","Type":"ContainerDied","Data":"82b08e3748358e85d42c0089d9dc157238aaa16b338932e9410e5858de4c7519"} Oct 30 05:15:19.728077 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.726940 6587 scope.go:122] "RemoveContainer" containerID="82b08e3748358e85d42c0089d9dc157238aaa16b338932e9410e5858de4c7519" Oct 30 05:15:19.728077 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:19.727510 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw_odh-ai-gateway-infra(dc428e2b-af51-47bb-809f-a34110a0e2ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" podUID="dc428e2b-af51-47bb-809f-a34110a0e2ef" Oct 30 05:15:19.759852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.759535 6587 scope.go:122] "RemoveContainer" containerID="736ef2e0c39956a376449256255fcf35135d547949b12be3144cb7fddb35ae7b" Oct 30 05:15:19.766820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.766790 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn"] Oct 30 05:15:19.770260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.770234 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" event={"ID":"06dd03d6-332a-4912-b8e8-8fc243f2e8b9","Type":"ContainerStarted","Data":"9edec324fa0a9bf3c26d89db53060ec9fed87c6b2569b29c1158da899d89a7a7"} Oct 30 05:15:19.780721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.780430 6587 generic.go:356] "Generic (PLEG): container finished" podID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" containerID="54b1836a3c1784392da6b10c4bebc345643581cbff4ef8303f9b3ce394061e8f" exitCode=60 Oct 30 05:15:19.780721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.780603 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" event={"ID":"a387704a-20ad-4fa6-abb8-46ffc2b82f94","Type":"ContainerDied","Data":"54b1836a3c1784392da6b10c4bebc345643581cbff4ef8303f9b3ce394061e8f"} Oct 30 05:15:19.780945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.780781 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx"] Oct 30 05:15:19.783730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.783705 6587 generic.go:356] "Generic (PLEG): container finished" podID="b94412d8-2396-4b33-a05a-5189b4ba72b5" containerID="8b4d6128e69361c1eef7512e664eb5000ab38b6184ac45b4784b2d3a5ac0e3b7" exitCode=60 Oct 30 05:15:19.783873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.783779 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" event={"ID":"b94412d8-2396-4b33-a05a-5189b4ba72b5","Type":"ContainerDied","Data":"8b4d6128e69361c1eef7512e664eb5000ab38b6184ac45b4784b2d3a5ac0e3b7"} Oct 30 05:15:19.784311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.784293 6587 scope.go:122] "RemoveContainer" containerID="8b4d6128e69361c1eef7512e664eb5000ab38b6184ac45b4784b2d3a5ac0e3b7" Oct 30 05:15:19.784737 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:19.784714 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws_odh-ai-gateway-infra(b94412d8-2396-4b33-a05a-5189b4ba72b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" podUID="b94412d8-2396-4b33-a05a-5189b4ba72b5" Oct 30 05:15:19.786639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.786617 6587 generic.go:356] "Generic (PLEG): container finished" podID="7b67626a-07e8-4b08-bee3-243b269fb5dc" containerID="3c6fef9c132c8d8c6305406f6c42e4cc01532f34c3194d52587ed1e8a47a9746" exitCode=60 Oct 30 05:15:19.786804 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.786787 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" event={"ID":"7b67626a-07e8-4b08-bee3-243b269fb5dc","Type":"ContainerDied","Data":"3c6fef9c132c8d8c6305406f6c42e4cc01532f34c3194d52587ed1e8a47a9746"} Oct 30 05:15:19.789292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.789067 6587 scope.go:122] "RemoveContainer" containerID="1da754fe18cb67bc98307178f4e9185393c96bb2798bd41ed7b97ae777277e87" Oct 30 05:15:19.813183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.813144 6587 scope.go:122] "RemoveContainer" containerID="ad4ea5496aecf07624576329fb866436e719f8082cc736d91ca2a105671bdd21" Oct 30 05:15:19.887168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:19.887086 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958"] Oct 30 05:15:20.355402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.351915 6587 scope.go:122] "RemoveContainer" containerID="6824c07d794ae58959cb19071c960d7d371d31583e692f203eb98897a1a4081b" Oct 30 05:15:20.446052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.445818 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25"] Oct 30 05:15:20.465464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.465416 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" Oct 30 05:15:20.488281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.483465 6587 scope.go:122] "RemoveContainer" containerID="06c03b7191a0ae6f26b0fe4f5f12c82094399b1d7a301b1a410eac5cdc9fe95c" Oct 30 05:15:20.533517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.533482 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws"] Oct 30 05:15:20.550449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.543193 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a387704a-20ad-4fa6-abb8-46ffc2b82f94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387704a-20ad-4fa6-abb8-46ffc2b82f94-maas-api-service-ca\") pod \"a387704a-20ad-4fa6-abb8-46ffc2b82f94\" (UID: \"a387704a-20ad-4fa6-abb8-46ffc2b82f94\") " Oct 30 05:15:20.550449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.543304 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a387704a-20ad-4fa6-abb8-46ffc2b82f94-kube-api-access-vrxps\" (UniqueName: \"kubernetes.io/projected/a387704a-20ad-4fa6-abb8-46ffc2b82f94-kube-api-access-vrxps\") pod \"a387704a-20ad-4fa6-abb8-46ffc2b82f94\" (UID: \"a387704a-20ad-4fa6-abb8-46ffc2b82f94\") " Oct 30 05:15:20.550449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.543748 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a387704a-20ad-4fa6-abb8-46ffc2b82f94-maas-api-service-ca" pod "a387704a-20ad-4fa6-abb8-46ffc2b82f94" (UID: "a387704a-20ad-4fa6-abb8-46ffc2b82f94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:20.550449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.550246 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387704a-20ad-4fa6-abb8-46ffc2b82f94-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:20.564215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.564152 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a387704a-20ad-4fa6-abb8-46ffc2b82f94-kube-api-access-vrxps" pod "a387704a-20ad-4fa6-abb8-46ffc2b82f94" (UID: "a387704a-20ad-4fa6-abb8-46ffc2b82f94"). InnerVolumeSpecName "kube-api-access-vrxps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:20.572800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.572768 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw"] Oct 30 05:15:20.588094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.587870 6587 scope.go:122] "RemoveContainer" containerID="8a96ded5ea72defc01bb254fff2fe790d99d778ee2501ec8359b4871eb6de39e" Oct 30 05:15:20.614668 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.614242 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" Oct 30 05:15:20.707393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.696420 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrxps\" (UniqueName: \"kubernetes.io/projected/a387704a-20ad-4fa6-abb8-46ffc2b82f94-kube-api-access-vrxps\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:20.855485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.855431 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-maas-api-service-ca\") pod \"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5\" (UID: \"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5\") " Oct 30 05:15:20.872733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.872645 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-maas-api-service-ca" pod "dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" (UID: "dc3f6c58-ebd5-46d0-b024-7644ec3f06d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:20.922008 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.918802 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-kube-api-access-dnrfw\" (UniqueName: \"kubernetes.io/projected/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-kube-api-access-dnrfw\") pod \"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5\" (UID: \"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5\") " Oct 30 05:15:20.951225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.951035 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:20.984080 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:20.984045 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-kube-api-access-dnrfw" pod "dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" (UID: "dc3f6c58-ebd5-46d0-b024-7644ec3f06d5"). InnerVolumeSpecName "kube-api-access-dnrfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:21.024241 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.024060 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m"] Oct 30 05:15:21.073859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.073818 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnrfw\" (UniqueName: \"kubernetes.io/projected/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5-kube-api-access-dnrfw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:21.081944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.081278 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" event={"ID":"0dd99f41-71bc-41f0-abed-4be3c6110a1e","Type":"ContainerStarted","Data":"1c6f9811e1eb65a3b74c0ae3a967e42a69aa825be8ef2d9c83453ccd873d716b"} Oct 30 05:15:21.090733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.090699 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" event={"ID":"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e","Type":"ContainerStarted","Data":"2e2dc01718a6f898cedee2aef601d7c3279e35aadd836681d3bd1e238f64baeb"} Oct 30 05:15:21.096951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.096923 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" event={"ID":"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04","Type":"ContainerStarted","Data":"0ffb645adde75f398f0e2ae44a3422ab72c2a4767e07294a32be1b19fd7a5bd0"} Oct 30 05:15:21.147973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.120532 6587 generic.go:356] "Generic (PLEG): container finished" podID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" containerID="9edec324fa0a9bf3c26d89db53060ec9fed87c6b2569b29c1158da899d89a7a7" exitCode=60 Oct 30 05:15:21.147973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.120638 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" event={"ID":"06dd03d6-332a-4912-b8e8-8fc243f2e8b9","Type":"ContainerDied","Data":"9edec324fa0a9bf3c26d89db53060ec9fed87c6b2569b29c1158da899d89a7a7"} Oct 30 05:15:21.147973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.120693 6587 scope.go:122] "RemoveContainer" containerID="223ddf3a27d24df61d2f937d7680f124ed51d553b4e1dac76949038018efb5bd" Oct 30 05:15:21.171278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.169567 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" event={"ID":"dc3f6c58-ebd5-46d0-b024-7644ec3f06d5","Type":"ContainerDied","Data":"43ad01c677c7f4302794511c1a3b70bd2c7ba1762032748028338f0ee45b7e73"} Oct 30 05:15:21.171278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.170735 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh" Oct 30 05:15:21.180718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.178683 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" event={"ID":"1154868f-4d78-401f-8fd0-cdd9a0fda825","Type":"ContainerStarted","Data":"e5267111d7d08ebb713d59b0b9bfd6c964b700caa3949ef4314a346f70faafb1"} Oct 30 05:15:21.189236 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.188616 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" event={"ID":"a387704a-20ad-4fa6-abb8-46ffc2b82f94","Type":"ContainerDied","Data":"975f32d095baacfaa4e4cc2eef0dc2bca1af1533fdcc29087a34d42e38c56f7d"} Oct 30 05:15:21.189236 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.188822 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn" Oct 30 05:15:21.498518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.476633 6587 scope.go:122] "RemoveContainer" containerID="ede0ae55f6022d8b359f2a4a7c5b2be471027d3b6f653464ddc40c406ca9eedb" Oct 30 05:15:21.778400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.751727 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" Oct 30 05:15:21.998403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:21.995397 6587 scope.go:122] "RemoveContainer" containerID="54b1836a3c1784392da6b10c4bebc345643581cbff4ef8303f9b3ce394061e8f" Oct 30 05:15:22.118379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.104247 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" Oct 30 05:15:22.118379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.115063 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b67626a-07e8-4b08-bee3-243b269fb5dc-kube-api-access-9tm5m\" (UniqueName: \"kubernetes.io/projected/7b67626a-07e8-4b08-bee3-243b269fb5dc-kube-api-access-9tm5m\") pod \"7b67626a-07e8-4b08-bee3-243b269fb5dc\" (UID: \"7b67626a-07e8-4b08-bee3-243b269fb5dc\") " Oct 30 05:15:22.118379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.115110 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b67626a-07e8-4b08-bee3-243b269fb5dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b67626a-07e8-4b08-bee3-243b269fb5dc-maas-api-service-ca\") pod \"7b67626a-07e8-4b08-bee3-243b269fb5dc\" (UID: \"7b67626a-07e8-4b08-bee3-243b269fb5dc\") " Oct 30 05:15:22.143048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.137042 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b67626a-07e8-4b08-bee3-243b269fb5dc-maas-api-service-ca" pod "7b67626a-07e8-4b08-bee3-243b269fb5dc" (UID: "7b67626a-07e8-4b08-bee3-243b269fb5dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:22.143048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.137872 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b67626a-07e8-4b08-bee3-243b269fb5dc-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:22.153055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.145532 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" Oct 30 05:15:22.169013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.168954 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b67626a-07e8-4b08-bee3-243b269fb5dc-kube-api-access-9tm5m" pod "7b67626a-07e8-4b08-bee3-243b269fb5dc" (UID: "7b67626a-07e8-4b08-bee3-243b269fb5dc"). InnerVolumeSpecName "kube-api-access-9tm5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:22.175913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.175839 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh"] Oct 30 05:15:22.204471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.204433 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888955-g2mnh"] Oct 30 05:15:22.231382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.216462 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl"] Oct 30 05:15:22.249997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.245453 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e7d7ed-2e36-4285-a316-c6b296a75cd2-kube-api-access-2lmsp\" (UniqueName: \"kubernetes.io/projected/01e7d7ed-2e36-4285-a316-c6b296a75cd2-kube-api-access-2lmsp\") pod \"01e7d7ed-2e36-4285-a316-c6b296a75cd2\" (UID: \"01e7d7ed-2e36-4285-a316-c6b296a75cd2\") " Oct 30 05:15:22.249997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.245666 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-kube-api-access-bxvgf\" (UniqueName: \"kubernetes.io/projected/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-kube-api-access-bxvgf\") pod \"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0\" (UID: \"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0\") " Oct 30 05:15:22.249997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.245812 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-maas-api-service-ca\") pod \"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0\" (UID: \"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0\") " Oct 30 05:15:22.249997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.245956 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e7d7ed-2e36-4285-a316-c6b296a75cd2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e7d7ed-2e36-4285-a316-c6b296a75cd2-maas-api-service-ca\") pod \"01e7d7ed-2e36-4285-a316-c6b296a75cd2\" (UID: \"01e7d7ed-2e36-4285-a316-c6b296a75cd2\") " Oct 30 05:15:22.249997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.246852 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tm5m\" (UniqueName: \"kubernetes.io/projected/7b67626a-07e8-4b08-bee3-243b269fb5dc-kube-api-access-9tm5m\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:22.260428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.260343 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e7d7ed-2e36-4285-a316-c6b296a75cd2-maas-api-service-ca" pod "01e7d7ed-2e36-4285-a316-c6b296a75cd2" (UID: "01e7d7ed-2e36-4285-a316-c6b296a75cd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:22.261577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.261549 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-maas-api-service-ca" pod "9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" (UID: "9b16ea10-8c3f-40e9-8f76-c2840e19d3d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:22.271922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.271884 6587 generic.go:356] "Generic (PLEG): container finished" podID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" containerID="2e2dc01718a6f898cedee2aef601d7c3279e35aadd836681d3bd1e238f64baeb" exitCode=60 Oct 30 05:15:22.273546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.272066 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" event={"ID":"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e","Type":"ContainerDied","Data":"2e2dc01718a6f898cedee2aef601d7c3279e35aadd836681d3bd1e238f64baeb"} Oct 30 05:15:22.273546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.272699 6587 scope.go:122] "RemoveContainer" containerID="2e2dc01718a6f898cedee2aef601d7c3279e35aadd836681d3bd1e238f64baeb" Oct 30 05:15:22.273546 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:22.273160 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2_odh-ai-gateway-infra(fcac5a1a-ed6a-4fc7-881c-6247bc444b0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" podUID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" Oct 30 05:15:22.279821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.279741 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e7d7ed-2e36-4285-a316-c6b296a75cd2-kube-api-access-2lmsp" pod "01e7d7ed-2e36-4285-a316-c6b296a75cd2" (UID: "01e7d7ed-2e36-4285-a316-c6b296a75cd2"). InnerVolumeSpecName "kube-api-access-2lmsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:22.289458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.283750 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" event={"ID":"01e7d7ed-2e36-4285-a316-c6b296a75cd2","Type":"ContainerDied","Data":"2d64ac0d3fba3a218460387f40cf0bf06de72760dbb14a27b995f34d3fcee119"} Oct 30 05:15:22.289458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.283870 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25" Oct 30 05:15:22.289458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.287434 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-kube-api-access-bxvgf" pod "9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" (UID: "9b16ea10-8c3f-40e9-8f76-c2840e19d3d0"). InnerVolumeSpecName "kube-api-access-bxvgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:22.293378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.291113 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d"] Oct 30 05:15:22.301976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.301926 6587 scope.go:122] "RemoveContainer" containerID="22b28211d97bef55586a9471d7b97f20dc8af6b062b886ede61e37d289ddf4c6" Oct 30 05:15:22.302785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.302757 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn"] Oct 30 05:15:22.306124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.306103 6587 generic.go:356] "Generic (PLEG): container finished" podID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" containerID="0ffb645adde75f398f0e2ae44a3422ab72c2a4767e07294a32be1b19fd7a5bd0" exitCode=60 Oct 30 05:15:22.307710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.306829 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" event={"ID":"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04","Type":"ContainerDied","Data":"0ffb645adde75f398f0e2ae44a3422ab72c2a4767e07294a32be1b19fd7a5bd0"} Oct 30 05:15:22.308063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.307816 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888955-zs8dn"] Oct 30 05:15:22.312366 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.312336 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" event={"ID":"9b16ea10-8c3f-40e9-8f76-c2840e19d3d0","Type":"ContainerDied","Data":"e08d60a7015e0dc3106770a2447327c2ad038fc27b63151d7b1c1dd888227aa5"} Oct 30 05:15:22.312864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.312465 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958" Oct 30 05:15:22.341433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.340691 6587 scope.go:122] "RemoveContainer" containerID="c9018121b705385ff6932f925bf61844ded989a944b8dce6dc4f9ee9f2a6678b" Oct 30 05:15:22.354302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.354198 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:22.354302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.354229 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e7d7ed-2e36-4285-a316-c6b296a75cd2-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:22.354302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.354244 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lmsp\" (UniqueName: \"kubernetes.io/projected/01e7d7ed-2e36-4285-a316-c6b296a75cd2-kube-api-access-2lmsp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:22.354302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.354259 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxvgf\" (UniqueName: \"kubernetes.io/projected/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0-kube-api-access-bxvgf\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:22.355475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.355219 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" event={"ID":"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce","Type":"ContainerStarted","Data":"63ce4375a70f11c528f2cd0205ae5207d3d6f1d56f88aef5887a0e7d4bb3f15f"} Oct 30 05:15:22.357806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.357581 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" Oct 30 05:15:22.360969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.359795 6587 generic.go:356] "Generic (PLEG): container finished" podID="1154868f-4d78-401f-8fd0-cdd9a0fda825" containerID="e5267111d7d08ebb713d59b0b9bfd6c964b700caa3949ef4314a346f70faafb1" exitCode=60 Oct 30 05:15:22.360969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.359864 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" event={"ID":"1154868f-4d78-401f-8fd0-cdd9a0fda825","Type":"ContainerDied","Data":"e5267111d7d08ebb713d59b0b9bfd6c964b700caa3949ef4314a346f70faafb1"} Oct 30 05:15:22.360969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.360302 6587 scope.go:122] "RemoveContainer" containerID="e5267111d7d08ebb713d59b0b9bfd6c964b700caa3949ef4314a346f70faafb1" Oct 30 05:15:22.360969 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:22.360834 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8_odh-ai-gateway-infra(1154868f-4d78-401f-8fd0-cdd9a0fda825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" podUID="1154868f-4d78-401f-8fd0-cdd9a0fda825" Oct 30 05:15:22.366068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.366045 6587 generic.go:356] "Generic (PLEG): container finished" podID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" containerID="1c6f9811e1eb65a3b74c0ae3a967e42a69aa825be8ef2d9c83453ccd873d716b" exitCode=60 Oct 30 05:15:22.366173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.366131 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" event={"ID":"0dd99f41-71bc-41f0-abed-4be3c6110a1e","Type":"ContainerDied","Data":"1c6f9811e1eb65a3b74c0ae3a967e42a69aa825be8ef2d9c83453ccd873d716b"} Oct 30 05:15:22.367482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.366682 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" Oct 30 05:15:22.373547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.369508 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" event={"ID":"7b67626a-07e8-4b08-bee3-243b269fb5dc","Type":"ContainerDied","Data":"7587d5f7e2a1768d65176c32670ff1193aaa4adf2f63ca2129c7646866747e9a"} Oct 30 05:15:22.373547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.369593 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx" Oct 30 05:15:22.379465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.379427 6587 scope.go:122] "RemoveContainer" containerID="405a625c7ee7452b10d176b876a5fcc4e02eb9c426fc580a4aa866f9ee33bdc3" Oct 30 05:15:22.379572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.379476 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" Oct 30 05:15:22.466263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.466230 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94412d8-2396-4b33-a05a-5189b4ba72b5-kube-api-access-vpgfn\" (UniqueName: \"kubernetes.io/projected/b94412d8-2396-4b33-a05a-5189b4ba72b5-kube-api-access-vpgfn\") pod \"b94412d8-2396-4b33-a05a-5189b4ba72b5\" (UID: \"b94412d8-2396-4b33-a05a-5189b4ba72b5\") " Oct 30 05:15:22.466413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.466290 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-kube-api-access-l47vc\" (UniqueName: \"kubernetes.io/projected/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-kube-api-access-l47vc\") pod \"06dd03d6-332a-4912-b8e8-8fc243f2e8b9\" (UID: \"06dd03d6-332a-4912-b8e8-8fc243f2e8b9\") " Oct 30 05:15:22.466506 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.466487 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b94412d8-2396-4b33-a05a-5189b4ba72b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94412d8-2396-4b33-a05a-5189b4ba72b5-maas-api-service-ca\") pod \"b94412d8-2396-4b33-a05a-5189b4ba72b5\" (UID: \"b94412d8-2396-4b33-a05a-5189b4ba72b5\") " Oct 30 05:15:22.466608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.466593 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc428e2b-af51-47bb-809f-a34110a0e2ef-kube-api-access-9l2rm\" (UniqueName: \"kubernetes.io/projected/dc428e2b-af51-47bb-809f-a34110a0e2ef-kube-api-access-9l2rm\") pod \"dc428e2b-af51-47bb-809f-a34110a0e2ef\" (UID: \"dc428e2b-af51-47bb-809f-a34110a0e2ef\") " Oct 30 05:15:22.466674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.466638 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-maas-api-service-ca\") pod \"06dd03d6-332a-4912-b8e8-8fc243f2e8b9\" (UID: \"06dd03d6-332a-4912-b8e8-8fc243f2e8b9\") " Oct 30 05:15:22.466802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.466788 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc428e2b-af51-47bb-809f-a34110a0e2ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc428e2b-af51-47bb-809f-a34110a0e2ef-maas-api-service-ca\") pod \"dc428e2b-af51-47bb-809f-a34110a0e2ef\" (UID: \"dc428e2b-af51-47bb-809f-a34110a0e2ef\") " Oct 30 05:15:22.467839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.467815 6587 scope.go:122] "RemoveContainer" containerID="6d8e256cb49b202868c2d0a30aac271b952651d4afd84bfb5dc2b24a188ea296" Oct 30 05:15:22.469302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.469271 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94412d8-2396-4b33-a05a-5189b4ba72b5-maas-api-service-ca" pod "b94412d8-2396-4b33-a05a-5189b4ba72b5" (UID: "b94412d8-2396-4b33-a05a-5189b4ba72b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:22.469706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.469679 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc428e2b-af51-47bb-809f-a34110a0e2ef-maas-api-service-ca" pod "dc428e2b-af51-47bb-809f-a34110a0e2ef" (UID: "dc428e2b-af51-47bb-809f-a34110a0e2ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:22.470021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.469998 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-maas-api-service-ca" pod "06dd03d6-332a-4912-b8e8-8fc243f2e8b9" (UID: "06dd03d6-332a-4912-b8e8-8fc243f2e8b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:22.471683 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.471663 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94412d8-2396-4b33-a05a-5189b4ba72b5-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:22.472192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.472175 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:22.472344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.472330 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc428e2b-af51-47bb-809f-a34110a0e2ef-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:22.485657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.479453 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc428e2b-af51-47bb-809f-a34110a0e2ef-kube-api-access-9l2rm" pod "dc428e2b-af51-47bb-809f-a34110a0e2ef" (UID: "dc428e2b-af51-47bb-809f-a34110a0e2ef"). InnerVolumeSpecName "kube-api-access-9l2rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:22.486593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.486419 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94412d8-2396-4b33-a05a-5189b4ba72b5-kube-api-access-vpgfn" pod "b94412d8-2396-4b33-a05a-5189b4ba72b5" (UID: "b94412d8-2396-4b33-a05a-5189b4ba72b5"). InnerVolumeSpecName "kube-api-access-vpgfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:22.486593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.486528 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25"] Oct 30 05:15:22.486968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.486932 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-kube-api-access-l47vc" pod "06dd03d6-332a-4912-b8e8-8fc243f2e8b9" (UID: "06dd03d6-332a-4912-b8e8-8fc243f2e8b9"). InnerVolumeSpecName "kube-api-access-l47vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:22.496016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.492675 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888955-6zs25"] Oct 30 05:15:22.515295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.515078 6587 scope.go:122] "RemoveContainer" containerID="b5810720f4b4f6419eda6fff9ac7782967a08146d19987a8f52890863c74a881" Oct 30 05:15:22.589410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.587535 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l2rm\" (UniqueName: \"kubernetes.io/projected/dc428e2b-af51-47bb-809f-a34110a0e2ef-kube-api-access-9l2rm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:22.589410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.587569 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpgfn\" (UniqueName: \"kubernetes.io/projected/b94412d8-2396-4b33-a05a-5189b4ba72b5-kube-api-access-vpgfn\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:22.589410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.587588 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l47vc\" (UniqueName: \"kubernetes.io/projected/06dd03d6-332a-4912-b8e8-8fc243f2e8b9-kube-api-access-l47vc\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:22.608291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.608269 6587 scope.go:122] "RemoveContainer" containerID="3609c2b5ccf090c9f30aaa0a04a21dbb0d7fb74d0d3687e0b5d05c5e0d087b8b" Oct 30 05:15:22.867706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.857601 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2"] Oct 30 05:15:22.867706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.858923 6587 scope.go:122] "RemoveContainer" containerID="3c6fef9c132c8d8c6305406f6c42e4cc01532f34c3194d52587ed1e8a47a9746" Oct 30 05:15:22.951602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.951552 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" Oct 30 05:15:22.956547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:22.952850 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" Oct 30 05:15:23.000957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.000871 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx"] Oct 30 05:15:23.012251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.012207 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888955-kh4cx"] Oct 30 05:15:23.013075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.013047 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd99f41-71bc-41f0-abed-4be3c6110a1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd99f41-71bc-41f0-abed-4be3c6110a1e-maas-api-service-ca\") pod \"0dd99f41-71bc-41f0-abed-4be3c6110a1e\" (UID: \"0dd99f41-71bc-41f0-abed-4be3c6110a1e\") " Oct 30 05:15:23.013161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.013112 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-kube-api-access-gzjrs\" (UniqueName: \"kubernetes.io/projected/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-kube-api-access-gzjrs\") pod \"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04\" (UID: \"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04\") " Oct 30 05:15:23.013161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.013149 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-maas-api-service-ca\") pod \"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04\" (UID: \"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04\") " Oct 30 05:15:23.013284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.013267 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd99f41-71bc-41f0-abed-4be3c6110a1e-kube-api-access-xvv7v\" (UniqueName: \"kubernetes.io/projected/0dd99f41-71bc-41f0-abed-4be3c6110a1e-kube-api-access-xvv7v\") pod \"0dd99f41-71bc-41f0-abed-4be3c6110a1e\" (UID: \"0dd99f41-71bc-41f0-abed-4be3c6110a1e\") " Oct 30 05:15:23.013718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.013677 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-maas-api-service-ca" pod "7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" (UID: "7b15ecbd-f2d7-430f-a7d9-6e6d53328b04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:23.014134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.014115 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd99f41-71bc-41f0-abed-4be3c6110a1e-maas-api-service-ca" pod "0dd99f41-71bc-41f0-abed-4be3c6110a1e" (UID: "0dd99f41-71bc-41f0-abed-4be3c6110a1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:23.014224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.014187 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:23.031593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.031544 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd99f41-71bc-41f0-abed-4be3c6110a1e-kube-api-access-xvv7v" pod "0dd99f41-71bc-41f0-abed-4be3c6110a1e" (UID: "0dd99f41-71bc-41f0-abed-4be3c6110a1e"). InnerVolumeSpecName "kube-api-access-xvv7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:23.052464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.052433 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958"] Oct 30 05:15:23.054122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.054088 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-kube-api-access-gzjrs" pod "7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" (UID: "7b15ecbd-f2d7-430f-a7d9-6e6d53328b04"). InnerVolumeSpecName "kube-api-access-gzjrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:23.056936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.056909 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f8eb4-29888955-xs958"] Oct 30 05:15:23.081566 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.081507 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" podStartSLOduration=23.081488108 podStartE2EDuration="23.081488108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:23.079703777 +0000 UTC m=+1847.607470564" watchObservedRunningTime="2026-10-30 05:15:23.081488108 +0000 UTC m=+1847.609254897" Oct 30 05:15:23.125390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.125292 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd99f41-71bc-41f0-abed-4be3c6110a1e-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:23.125390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.125332 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzjrs\" (UniqueName: \"kubernetes.io/projected/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04-kube-api-access-gzjrs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:23.125390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.125370 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvv7v\" (UniqueName: \"kubernetes.io/projected/0dd99f41-71bc-41f0-abed-4be3c6110a1e-kube-api-access-xvv7v\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:23.278680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.278514 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8"] Oct 30 05:15:23.384429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.384353 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" event={"ID":"0dd99f41-71bc-41f0-abed-4be3c6110a1e","Type":"ContainerDied","Data":"d4bedba704effa469cd3f93987e4a1107b3c806dfda1ef9001e0b0c7a97767e5"} Oct 30 05:15:23.384659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.384647 6587 scope.go:122] "RemoveContainer" containerID="1c6f9811e1eb65a3b74c0ae3a967e42a69aa825be8ef2d9c83453ccd873d716b" Oct 30 05:15:23.384800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.384475 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl" Oct 30 05:15:23.391091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.391066 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" event={"ID":"7b15ecbd-f2d7-430f-a7d9-6e6d53328b04","Type":"ContainerDied","Data":"0aea7a0be9e7187f783539d29b71c6642c22aff30c0b88bd124dcc9f3d628da8"} Oct 30 05:15:23.391202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.391164 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d" Oct 30 05:15:23.404056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.404028 6587 scope.go:122] "RemoveContainer" containerID="0ffb645adde75f398f0e2ae44a3422ab72c2a4767e07294a32be1b19fd7a5bd0" Oct 30 05:15:23.405160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.405137 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" event={"ID":"dc428e2b-af51-47bb-809f-a34110a0e2ef","Type":"ContainerDied","Data":"f4c4fd5b7433e37021b052013be0a5ad0a1b988d163a777cc1072a73e779bf0c"} Oct 30 05:15:23.405300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.405242 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw" Oct 30 05:15:23.410794 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.410770 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" event={"ID":"06dd03d6-332a-4912-b8e8-8fc243f2e8b9","Type":"ContainerDied","Data":"d3dd3a2ad163c5f67fd1d0202b5a0ffc67dd74dd6e7ef0dfc223bd0e49231ff0"} Oct 30 05:15:23.410913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.410889 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m" Oct 30 05:15:23.413847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.413819 6587 generic.go:356] "Generic (PLEG): container finished" podID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" containerID="63ce4375a70f11c528f2cd0205ae5207d3d6f1d56f88aef5887a0e7d4bb3f15f" exitCode=60 Oct 30 05:15:23.413952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.413910 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" event={"ID":"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce","Type":"ContainerDied","Data":"63ce4375a70f11c528f2cd0205ae5207d3d6f1d56f88aef5887a0e7d4bb3f15f"} Oct 30 05:15:23.414332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.414313 6587 scope.go:122] "RemoveContainer" containerID="63ce4375a70f11c528f2cd0205ae5207d3d6f1d56f88aef5887a0e7d4bb3f15f" Oct 30 05:15:23.414673 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:23.414648 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh_odh-ai-gateway-infra(d6b01cc1-b83a-4fb9-abf6-276ad20b45ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" podUID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" Oct 30 05:15:23.457732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.457376 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" event={"ID":"b94412d8-2396-4b33-a05a-5189b4ba72b5","Type":"ContainerDied","Data":"6e249c87780db871d60995f8432fbad3b66cf9a5cf1837af9bae4b594ea9f8cb"} Oct 30 05:15:23.457732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.457506 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws" Oct 30 05:15:23.462516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.462490 6587 scope.go:122] "RemoveContainer" containerID="82b08e3748358e85d42c0089d9dc157238aaa16b338932e9410e5858de4c7519" Oct 30 05:15:23.492642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.492614 6587 scope.go:122] "RemoveContainer" containerID="9edec324fa0a9bf3c26d89db53060ec9fed87c6b2569b29c1158da899d89a7a7" Oct 30 05:15:23.550529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.520001 6587 scope.go:122] "RemoveContainer" containerID="8a96ded5ea72defc01bb254fff2fe790d99d778ee2501ec8359b4871eb6de39e" Oct 30 05:15:23.630112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.630078 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" path="/var/lib/kubelet/pods/01e7d7ed-2e36-4285-a316-c6b296a75cd2/volumes" Oct 30 05:15:23.680510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.677177 6587 scope.go:122] "RemoveContainer" containerID="8b4d6128e69361c1eef7512e664eb5000ab38b6184ac45b4784b2d3a5ac0e3b7" Oct 30 05:15:23.681215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.681136 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b67626a-07e8-4b08-bee3-243b269fb5dc" path="/var/lib/kubelet/pods/7b67626a-07e8-4b08-bee3-243b269fb5dc/volumes" Oct 30 05:15:23.681930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.681789 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" path="/var/lib/kubelet/pods/9b16ea10-8c3f-40e9-8f76-c2840e19d3d0/volumes" Oct 30 05:15:23.682915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.682759 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" path="/var/lib/kubelet/pods/a387704a-20ad-4fa6-abb8-46ffc2b82f94/volumes" Oct 30 05:15:23.683687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.683329 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" path="/var/lib/kubelet/pods/dc3f6c58-ebd5-46d0-b024-7644ec3f06d5/volumes" Oct 30 05:15:23.683879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.683828 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d"] Oct 30 05:15:23.683879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.683865 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888955-j5n8d"] Oct 30 05:15:23.683982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.683929 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw"] Oct 30 05:15:23.683982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.683947 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-453511-29888955-z8gqw"] Oct 30 05:15:23.688962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.688941 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl"] Oct 30 05:15:23.695830 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.695806 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8afb26-29888955-x7dxl"] Oct 30 05:15:23.797960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.797931 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" Oct 30 05:15:23.802825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.802758 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m"] Oct 30 05:15:23.806920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.806889 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888955-m6g4m"] Oct 30 05:15:23.868068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.868006 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws"] Oct 30 05:15:23.873876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.873848 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888955-xkcws"] Oct 30 05:15:23.883632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.883423 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-maas-api-service-ca\") pod \"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e\" (UID: \"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e\") " Oct 30 05:15:23.890404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.883829 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-kube-api-access-c7jz4\" (UniqueName: \"kubernetes.io/projected/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-kube-api-access-c7jz4\") pod \"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e\" (UID: \"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e\") " Oct 30 05:15:23.890404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.885955 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-maas-api-service-ca" pod "fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" (UID: "fcac5a1a-ed6a-4fc7-881c-6247bc444b0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:23.891388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.891155 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-kube-api-access-c7jz4" pod "fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" (UID: "fcac5a1a-ed6a-4fc7-881c-6247bc444b0e"). InnerVolumeSpecName "kube-api-access-c7jz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:23.901645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.901615 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" Oct 30 05:15:23.986649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.986614 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1154868f-4d78-401f-8fd0-cdd9a0fda825-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1154868f-4d78-401f-8fd0-cdd9a0fda825-maas-api-service-ca\") pod \"1154868f-4d78-401f-8fd0-cdd9a0fda825\" (UID: \"1154868f-4d78-401f-8fd0-cdd9a0fda825\") " Oct 30 05:15:23.986845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.986779 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1154868f-4d78-401f-8fd0-cdd9a0fda825-kube-api-access-l94qc\" (UniqueName: \"kubernetes.io/projected/1154868f-4d78-401f-8fd0-cdd9a0fda825-kube-api-access-l94qc\") pod \"1154868f-4d78-401f-8fd0-cdd9a0fda825\" (UID: \"1154868f-4d78-401f-8fd0-cdd9a0fda825\") " Oct 30 05:15:23.987030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.987002 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1154868f-4d78-401f-8fd0-cdd9a0fda825-maas-api-service-ca" pod "1154868f-4d78-401f-8fd0-cdd9a0fda825" (UID: "1154868f-4d78-401f-8fd0-cdd9a0fda825"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:23.987444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.987407 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:23.987444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.987431 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7jz4\" (UniqueName: \"kubernetes.io/projected/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e-kube-api-access-c7jz4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:23.987444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.987441 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1154868f-4d78-401f-8fd0-cdd9a0fda825-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:23.989012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:23.988987 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1154868f-4d78-401f-8fd0-cdd9a0fda825-kube-api-access-l94qc" pod "1154868f-4d78-401f-8fd0-cdd9a0fda825" (UID: "1154868f-4d78-401f-8fd0-cdd9a0fda825"). InnerVolumeSpecName "kube-api-access-l94qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:24.088813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.088725 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l94qc\" (UniqueName: \"kubernetes.io/projected/1154868f-4d78-401f-8fd0-cdd9a0fda825-kube-api-access-l94qc\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:24.091131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.091105 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh"] Oct 30 05:15:24.470409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.470290 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" Oct 30 05:15:24.470409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.470320 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2" event={"ID":"fcac5a1a-ed6a-4fc7-881c-6247bc444b0e","Type":"ContainerDied","Data":"678067f1fa4bb5c9117b21edf968ef4dc8f0b812b46697cc62eda71ec3ebe56d"} Oct 30 05:15:24.470643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.470421 6587 scope.go:122] "RemoveContainer" containerID="2e2dc01718a6f898cedee2aef601d7c3279e35aadd836681d3bd1e238f64baeb" Oct 30 05:15:24.477545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.477523 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" Oct 30 05:15:24.477664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.477540 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8" event={"ID":"1154868f-4d78-401f-8fd0-cdd9a0fda825","Type":"ContainerDied","Data":"9669a8759585eeaf88fe94ccd74bf890126de4fd786ff59900fc6b05d3a6db20"} Oct 30 05:15:24.482485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.482467 6587 scope.go:122] "RemoveContainer" containerID="e5267111d7d08ebb713d59b0b9bfd6c964b700caa3949ef4314a346f70faafb1" Oct 30 05:15:24.563434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.563403 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2"] Oct 30 05:15:24.568627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.568600 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7f962-29888955-hltl2"] Oct 30 05:15:24.666167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.666132 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8"] Oct 30 05:15:24.673449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.673421 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888955-m48s8"] Oct 30 05:15:24.779823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.779762 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" Oct 30 05:15:24.969172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.969143 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-kube-api-access-w2tc6\" (UniqueName: \"kubernetes.io/projected/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-kube-api-access-w2tc6\") pod \"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce\" (UID: \"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce\") " Oct 30 05:15:24.969659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.969288 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-maas-api-service-ca\") pod \"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce\" (UID: \"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce\") " Oct 30 05:15:24.969659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.969629 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-maas-api-service-ca" pod "d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" (UID: "d6b01cc1-b83a-4fb9-abf6-276ad20b45ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:24.969882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.969865 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:24.971459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:24.971427 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-kube-api-access-w2tc6" pod "d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" (UID: "d6b01cc1-b83a-4fb9-abf6-276ad20b45ce"). InnerVolumeSpecName "kube-api-access-w2tc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:25.071121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:25.071078 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2tc6\" (UniqueName: \"kubernetes.io/projected/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce-kube-api-access-w2tc6\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:15:25.492247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:25.492161 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" Oct 30 05:15:25.492247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:25.492166 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh" event={"ID":"d6b01cc1-b83a-4fb9-abf6-276ad20b45ce","Type":"ContainerDied","Data":"26eeb0086b552497be920c641a3ec836b1709620f085b0541bc846a39cf76958"} Oct 30 05:15:25.492511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:25.492287 6587 scope.go:122] "RemoveContainer" containerID="63ce4375a70f11c528f2cd0205ae5207d3d6f1d56f88aef5887a0e7d4bb3f15f" Oct 30 05:15:25.566568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:25.566534 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh"] Oct 30 05:15:25.585274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:25.585238 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" path="/var/lib/kubelet/pods/06dd03d6-332a-4912-b8e8-8fc243f2e8b9/volumes" Oct 30 05:15:25.585840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:25.585818 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" path="/var/lib/kubelet/pods/0dd99f41-71bc-41f0-abed-4be3c6110a1e/volumes" Oct 30 05:15:25.586379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:25.586343 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1154868f-4d78-401f-8fd0-cdd9a0fda825" path="/var/lib/kubelet/pods/1154868f-4d78-401f-8fd0-cdd9a0fda825/volumes" Oct 30 05:15:25.587148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:25.587091 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" path="/var/lib/kubelet/pods/7b15ecbd-f2d7-430f-a7d9-6e6d53328b04/volumes" Oct 30 05:15:25.587534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:25.587518 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b94412d8-2396-4b33-a05a-5189b4ba72b5" path="/var/lib/kubelet/pods/b94412d8-2396-4b33-a05a-5189b4ba72b5/volumes" Oct 30 05:15:25.587937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:25.587911 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc428e2b-af51-47bb-809f-a34110a0e2ef" path="/var/lib/kubelet/pods/dc428e2b-af51-47bb-809f-a34110a0e2ef/volumes" Oct 30 05:15:25.588597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:25.588576 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" path="/var/lib/kubelet/pods/fcac5a1a-ed6a-4fc7-881c-6247bc444b0e/volumes" Oct 30 05:15:25.588997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:25.588978 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888955-ssgnh"] Oct 30 05:15:25.986900 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:25.986867 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:15:27.584614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:27.584582 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" path="/var/lib/kubelet/pods/d6b01cc1-b83a-4fb9-abf6-276ad20b45ce/volumes" Oct 30 05:15:30.554677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:30.554583 6587 generic.go:356] "Generic (PLEG): container finished" podID="98061727-7ccf-40ba-88a2-1442bcb73855" containerID="00ffca31366e48a3b1f02bcd15a884a7c26022813912795ba1f2a9f3d4004859" exitCode=0 Oct 30 05:15:30.555124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:30.554690 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" event={"ID":"98061727-7ccf-40ba-88a2-1442bcb73855","Type":"ContainerDied","Data":"00ffca31366e48a3b1f02bcd15a884a7c26022813912795ba1f2a9f3d4004859"} Oct 30 05:15:30.556067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:30.556042 6587 scope.go:122] "RemoveContainer" containerID="00ffca31366e48a3b1f02bcd15a884a7c26022813912795ba1f2a9f3d4004859" Oct 30 05:15:30.558659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:30.558626 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:30.558791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:30.558715 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:30.558791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:30.558781 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:31.569063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:31.569022 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" event={"ID":"98061727-7ccf-40ba-88a2-1442bcb73855","Type":"ContainerStarted","Data":"7006402bca16b9b843d06477ae2b4d75be437a1679a527d4e97862dcdc1fb39f"} Oct 30 05:15:31.676601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:31.676555 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:15:31.676781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:31.676722 6587 patch_prober.go:29] interesting pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 05:15:31.676781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:31.676768 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 05:15:32.676310 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:32.676273 6587 patch_prober.go:29] interesting pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 05:15:32.676764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:32.676344 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 05:15:33.710382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:33.684065 6587 patch_prober.go:29] interesting pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 05:15:33.710382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:33.684140 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 05:15:34.676288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:34.676249 6587 patch_prober.go:29] interesting pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 05:15:34.676499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:34.676308 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 05:15:35.676665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:35.676621 6587 patch_prober.go:29] interesting pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 05:15:35.677214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:35.676703 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 05:15:36.676594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:36.676555 6587 patch_prober.go:29] interesting pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 05:15:36.676803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:36.676614 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 05:15:36.982583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:36.982493 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:15:37.680877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:37.680840 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:15:37.813384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:37.804613 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:15:39.311873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:39.311842 6587 scope.go:122] "RemoveContainer" containerID="057bc15e73b4656b1a8b000dcac298a9d17a1a83cef10f51335e43bab96ea56b" Oct 30 05:15:39.323773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:39.323745 6587 scope.go:122] "RemoveContainer" containerID="96c67553b8d37bc9b308bdf5cfaee847415abdafc8ddbe91753c6f9d8a4f3e76" Oct 30 05:15:43.604121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604085 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj"] Oct 30 05:15:43.604890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604872 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" containerName="cleanup" Oct 30 05:15:43.604937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604891 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" containerName="cleanup" Oct 30 05:15:43.604937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604904 6587 cpu_manager.go:414] "containerMap: removing container" podUID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" containerName="cleanup" Oct 30 05:15:43.604937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604909 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" containerName="cleanup" Oct 30 05:15:43.604937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604920 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" containerName="cleanup" Oct 30 05:15:43.604937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604925 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" containerName="cleanup" Oct 30 05:15:43.604937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604932 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1154868f-4d78-401f-8fd0-cdd9a0fda825" containerName="cleanup" Oct 30 05:15:43.604937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604937 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1154868f-4d78-401f-8fd0-cdd9a0fda825" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604949 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604954 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604961 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604965 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604973 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b94412d8-2396-4b33-a05a-5189b4ba72b5" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604977 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b94412d8-2396-4b33-a05a-5189b4ba72b5" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604983 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7b67626a-07e8-4b08-bee3-243b269fb5dc" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604988 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b67626a-07e8-4b08-bee3-243b269fb5dc" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.604995 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605001 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605010 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b94412d8-2396-4b33-a05a-5189b4ba72b5" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605015 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b94412d8-2396-4b33-a05a-5189b4ba72b5" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605024 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7b67626a-07e8-4b08-bee3-243b269fb5dc" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605038 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b67626a-07e8-4b08-bee3-243b269fb5dc" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605048 6587 cpu_manager.go:414] "containerMap: removing container" podUID="dc428e2b-af51-47bb-809f-a34110a0e2ef" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605053 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc428e2b-af51-47bb-809f-a34110a0e2ef" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605059 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605063 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605073 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605077 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605088 6587 cpu_manager.go:414] "containerMap: removing container" podUID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605093 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605098 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7b67626a-07e8-4b08-bee3-243b269fb5dc" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605103 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b67626a-07e8-4b08-bee3-243b269fb5dc" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605113 6587 cpu_manager.go:414] "containerMap: removing container" podUID="14e81da2-cc1a-4b99-9a85-ee73d843b8ad" containerName="istio-proxy" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605117 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e81da2-cc1a-4b99-9a85-ee73d843b8ad" containerName="istio-proxy" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605125 6587 cpu_manager.go:414] "containerMap: removing container" podUID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605130 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605135 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605140 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605149 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7d74c3e2-27d2-4650-97ea-7c99efe8f924" containerName="istio-proxy" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605153 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d74c3e2-27d2-4650-97ea-7c99efe8f924" containerName="istio-proxy" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605161 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605165 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605172 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" containerName="cleanup" Oct 30 05:15:43.605164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605177 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605184 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1154868f-4d78-401f-8fd0-cdd9a0fda825" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605189 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1154868f-4d78-401f-8fd0-cdd9a0fda825" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605197 6587 cpu_manager.go:414] "containerMap: removing container" podUID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605201 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605208 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605212 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605225 6587 cpu_manager.go:414] "containerMap: removing container" podUID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605229 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605238 6587 cpu_manager.go:414] "containerMap: removing container" podUID="dc428e2b-af51-47bb-809f-a34110a0e2ef" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605242 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc428e2b-af51-47bb-809f-a34110a0e2ef" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605248 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605261 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605268 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605272 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605283 6587 cpu_manager.go:414] "containerMap: removing container" podUID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605287 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605301 6587 cpu_manager.go:414] "containerMap: removing container" podUID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605305 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605314 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605318 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605461 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605471 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d74c3e2-27d2-4650-97ea-7c99efe8f924" containerName="istio-proxy" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605479 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605486 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605493 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1154868f-4d78-401f-8fd0-cdd9a0fda825" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605499 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc428e2b-af51-47bb-809f-a34110a0e2ef" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605504 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605509 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605516 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605521 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605526 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc428e2b-af51-47bb-809f-a34110a0e2ef" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605532 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605537 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605545 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605550 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605556 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1154868f-4d78-401f-8fd0-cdd9a0fda825" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605562 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e7d7ed-2e36-4285-a316-c6b296a75cd2" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605567 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b94412d8-2396-4b33-a05a-5189b4ba72b5" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605574 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605581 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605587 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b67626a-07e8-4b08-bee3-243b269fb5dc" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605593 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605599 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605604 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b94412d8-2396-4b33-a05a-5189b4ba72b5" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605610 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b67626a-07e8-4b08-bee3-243b269fb5dc" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605617 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605624 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605630 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b67626a-07e8-4b08-bee3-243b269fb5dc" containerName="cleanup" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605635 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e81da2-cc1a-4b99-9a85-ee73d843b8ad" containerName="istio-proxy" Oct 30 05:15:43.606302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.605642 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" containerName="cleanup" Oct 30 05:15:43.609735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.609717 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" Oct 30 05:15:43.628013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.627986 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj"] Oct 30 05:15:43.635477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.635434 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2a8f18\"" Oct 30 05:15:43.675107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.675074 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2a8f18-dockercfg-jgft2\"" Oct 30 05:15:43.675759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.675729 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk"] Oct 30 05:15:43.676914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.676893 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" containerName="cleanup" Oct 30 05:15:43.676914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.676914 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" containerName="cleanup" Oct 30 05:15:43.677049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.676937 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1154868f-4d78-401f-8fd0-cdd9a0fda825" containerName="cleanup" Oct 30 05:15:43.677049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.676945 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1154868f-4d78-401f-8fd0-cdd9a0fda825" containerName="cleanup" Oct 30 05:15:43.677049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.676962 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" containerName="cleanup" Oct 30 05:15:43.677049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.676969 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" containerName="cleanup" Oct 30 05:15:43.677049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.676982 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" containerName="cleanup" Oct 30 05:15:43.677049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.676989 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" containerName="cleanup" Oct 30 05:15:43.677049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677003 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b94412d8-2396-4b33-a05a-5189b4ba72b5" containerName="cleanup" Oct 30 05:15:43.677049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677011 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b94412d8-2396-4b33-a05a-5189b4ba72b5" containerName="cleanup" Oct 30 05:15:43.677049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677040 6587 cpu_manager.go:414] "containerMap: removing container" podUID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" containerName="cleanup" Oct 30 05:15:43.677049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677047 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677060 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677067 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677084 6587 cpu_manager.go:414] "containerMap: removing container" podUID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677090 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677118 6587 cpu_manager.go:414] "containerMap: removing container" podUID="dc428e2b-af51-47bb-809f-a34110a0e2ef" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677124 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc428e2b-af51-47bb-809f-a34110a0e2ef" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677288 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd99f41-71bc-41f0-abed-4be3c6110a1e" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677301 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b15ecbd-f2d7-430f-a7d9-6e6d53328b04" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677310 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b16ea10-8c3f-40e9-8f76-c2840e19d3d0" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677320 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1154868f-4d78-401f-8fd0-cdd9a0fda825" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677331 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3f6c58-ebd5-46d0-b024-7644ec3f06d5" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677343 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcac5a1a-ed6a-4fc7-881c-6247bc444b0e" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677370 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc428e2b-af51-47bb-809f-a34110a0e2ef" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677380 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b01cc1-b83a-4fb9-abf6-276ad20b45ce" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677397 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="06dd03d6-332a-4912-b8e8-8fc243f2e8b9" containerName="cleanup" Oct 30 05:15:43.677411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677409 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a387704a-20ad-4fa6-abb8-46ffc2b82f94" containerName="cleanup" Oct 30 05:15:43.677900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.677419 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b94412d8-2396-4b33-a05a-5189b4ba72b5" containerName="cleanup" Oct 30 05:15:43.680655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.680637 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" Oct 30 05:15:43.695756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.695727 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk"] Oct 30 05:15:43.905236 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.905134 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdcwc\" (UniqueName: \"kubernetes.io/projected/fbe6597b-6a31-4372-a6aa-d6b041b06b56-kube-api-access-qdcwc\") pod \"payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj\" (UID: \"fbe6597b-6a31-4372-a6aa-d6b041b06b56\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" Oct 30 05:15:43.905527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.905260 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d779j\" (UniqueName: \"kubernetes.io/projected/dfe25060-9266-482c-8a47-f2f9e317ecb0-kube-api-access-d779j\") pod \"payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk\" (UID: \"dfe25060-9266-482c-8a47-f2f9e317ecb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" Oct 30 05:15:43.905527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.905334 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe6597b-6a31-4372-a6aa-d6b041b06b56-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj\" (UID: \"fbe6597b-6a31-4372-a6aa-d6b041b06b56\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" Oct 30 05:15:43.905527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:43.905413 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe25060-9266-482c-8a47-f2f9e317ecb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk\" (UID: \"dfe25060-9266-482c-8a47-f2f9e317ecb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" Oct 30 05:15:44.014014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.007507 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe25060-9266-482c-8a47-f2f9e317ecb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk\" (UID: \"dfe25060-9266-482c-8a47-f2f9e317ecb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" Oct 30 05:15:44.014014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.007826 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdcwc\" (UniqueName: \"kubernetes.io/projected/fbe6597b-6a31-4372-a6aa-d6b041b06b56-kube-api-access-qdcwc\") pod \"payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj\" (UID: \"fbe6597b-6a31-4372-a6aa-d6b041b06b56\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" Oct 30 05:15:44.014014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.007944 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d779j\" (UniqueName: \"kubernetes.io/projected/dfe25060-9266-482c-8a47-f2f9e317ecb0-kube-api-access-d779j\") pod \"payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk\" (UID: \"dfe25060-9266-482c-8a47-f2f9e317ecb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" Oct 30 05:15:44.014014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.008023 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe6597b-6a31-4372-a6aa-d6b041b06b56-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj\" (UID: \"fbe6597b-6a31-4372-a6aa-d6b041b06b56\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" Oct 30 05:15:44.014014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.009145 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe6597b-6a31-4372-a6aa-d6b041b06b56-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj\" (UID: \"fbe6597b-6a31-4372-a6aa-d6b041b06b56\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" Oct 30 05:15:44.014014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.010051 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe25060-9266-482c-8a47-f2f9e317ecb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk\" (UID: \"dfe25060-9266-482c-8a47-f2f9e317ecb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" Oct 30 05:15:44.048527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.048495 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d779j\" (UniqueName: \"kubernetes.io/projected/dfe25060-9266-482c-8a47-f2f9e317ecb0-kube-api-access-d779j\") pod \"payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk\" (UID: \"dfe25060-9266-482c-8a47-f2f9e317ecb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" Oct 30 05:15:44.058459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.058318 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdcwc\" (UniqueName: \"kubernetes.io/projected/fbe6597b-6a31-4372-a6aa-d6b041b06b56-kube-api-access-qdcwc\") pod \"payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj\" (UID: \"fbe6597b-6a31-4372-a6aa-d6b041b06b56\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" Oct 30 05:15:44.230033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.229944 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" Oct 30 05:15:44.309061 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.300026 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" Oct 30 05:15:44.534597 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:15:44.534553 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe6597b_6a31_4372_a6aa_d6b041b06b56.slice/crio-c2ff2dbfe567eba0e0cb63214b9d16f42db7af23fe82c8fa2b2cc1e51603d43e WatchSource:0}: Error finding container c2ff2dbfe567eba0e0cb63214b9d16f42db7af23fe82c8fa2b2cc1e51603d43e: Status 404 returned error can't find the container with id c2ff2dbfe567eba0e0cb63214b9d16f42db7af23fe82c8fa2b2cc1e51603d43e Oct 30 05:15:44.536981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.536957 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj"] Oct 30 05:15:44.632869 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:15:44.632838 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe25060_9266_482c_8a47_f2f9e317ecb0.slice/crio-50377e4f7d59813e0b9659af34553c3a0ca8e0be953a721e7a61cb735d616923 WatchSource:0}: Error finding container 50377e4f7d59813e0b9659af34553c3a0ca8e0be953a721e7a61cb735d616923: Status 404 returned error can't find the container with id 50377e4f7d59813e0b9659af34553c3a0ca8e0be953a721e7a61cb735d616923 Oct 30 05:15:44.633281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.633261 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk"] Oct 30 05:15:44.917906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.917785 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" event={"ID":"dfe25060-9266-482c-8a47-f2f9e317ecb0","Type":"ContainerStarted","Data":"755ac7d6aa832662106632d946eb9eeeac60aaa6b5455aeca2b53f6ca9032980"} Oct 30 05:15:44.917906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.917852 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" Oct 30 05:15:44.917906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.917878 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" event={"ID":"dfe25060-9266-482c-8a47-f2f9e317ecb0","Type":"ContainerStarted","Data":"50377e4f7d59813e0b9659af34553c3a0ca8e0be953a721e7a61cb735d616923"} Oct 30 05:15:44.920994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.920301 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" event={"ID":"fbe6597b-6a31-4372-a6aa-d6b041b06b56","Type":"ContainerStarted","Data":"5aeb9ba2a48924face9a4612f54190f27f195565cb74f25c5bc7afee4eb7d41d"} Oct 30 05:15:44.920994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.920667 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" event={"ID":"fbe6597b-6a31-4372-a6aa-d6b041b06b56","Type":"ContainerStarted","Data":"c2ff2dbfe567eba0e0cb63214b9d16f42db7af23fe82c8fa2b2cc1e51603d43e"} Oct 30 05:15:44.920994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.920709 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" Oct 30 05:15:44.968831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:44.968774 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" podStartSLOduration=1.968759893 podStartE2EDuration="1.968759893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:44.964009001 +0000 UTC m=+1869.491775786" watchObservedRunningTime="2026-10-30 05:15:44.968759893 +0000 UTC m=+1869.496526679" Oct 30 05:15:45.008316 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:45.008255 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" podStartSLOduration=2.008242062 podStartE2EDuration="2.008242062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:45.006983549 +0000 UTC m=+1869.534750339" watchObservedRunningTime="2026-10-30 05:15:45.008242062 +0000 UTC m=+1869.536008878" Oct 30 05:15:55.936403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:55.934611 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" Oct 30 05:15:55.936403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:55.934791 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" Oct 30 05:15:55.994388 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:15:55.992822 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:15:57.744113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.744073 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5"] Oct 30 05:15:57.748558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.748529 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.777077 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.777047 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5"] Oct 30 05:15:57.781431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.781404 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e16297-openshift-default-dockercfg-gzhs5\"" Oct 30 05:15:57.868915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.868878 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abdd1519-522f-4302-844d-947822be0960-istiod-ca-cert\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.868915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.868917 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-envoy\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.869131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.869030 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-certs\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.869131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.869074 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-socket\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.869131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.869101 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abdd1519-522f-4302-844d-947822be0960-istio-podinfo\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.869230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.869204 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-data\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.869284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.869227 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc8zp\" (UniqueName: \"kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-kube-api-access-sc8zp\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.869284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.869255 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-istio-token\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.869455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.869440 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-credential-socket\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.977393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.976221 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abdd1519-522f-4302-844d-947822be0960-istiod-ca-cert\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.977393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.976285 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-envoy\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.977393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.976398 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-certs\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.977393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.976447 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-socket\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.977393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.976506 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abdd1519-522f-4302-844d-947822be0960-istio-podinfo\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.977393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.976629 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-data\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.977393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.976661 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc8zp\" (UniqueName: \"kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-kube-api-access-sc8zp\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.977393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.976690 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-istio-token\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.977393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.976843 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-credential-socket\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.990849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.990054 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abdd1519-522f-4302-844d-947822be0960-istiod-ca-cert\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.990849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.990705 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-socket\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:57.990849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:57.990786 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-certs\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:58.008611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:58.008440 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-credential-socket\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:58.009442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:58.009407 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-data\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:58.012609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:58.012552 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-envoy\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:58.038332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:58.038275 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-istio-token\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:58.043467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:58.039111 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc8zp\" (UniqueName: \"kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-kube-api-access-sc8zp\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:58.063377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:58.059984 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abdd1519-522f-4302-844d-947822be0960-istio-podinfo\") pod \"e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:58.092539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:58.091744 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:15:58.611688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:58.610321 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5"] Oct 30 05:15:58.624476 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:15:58.624436 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabdd1519_522f_4302_844d_947822be0960.slice/crio-24986f0358dbeab631d7909392f918624117816e29cdc682eb48a9cdb27bd19d WatchSource:0}: Error finding container 24986f0358dbeab631d7909392f918624117816e29cdc682eb48a9cdb27bd19d: Status 404 returned error can't find the container with id 24986f0358dbeab631d7909392f918624117816e29cdc682eb48a9cdb27bd19d Oct 30 05:15:58.630466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:58.630424 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:58.630567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:58.630515 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:58.630567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:58.630556 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:15:59.185946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:59.185904 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" event={"ID":"abdd1519-522f-4302-844d-947822be0960","Type":"ContainerStarted","Data":"35f1f9dff0904c02d65598be5658e7005270f90d83036415c62557860378e5cc"} Oct 30 05:15:59.186435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:59.185974 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" event={"ID":"abdd1519-522f-4302-844d-947822be0960","Type":"ContainerStarted","Data":"24986f0358dbeab631d7909392f918624117816e29cdc682eb48a9cdb27bd19d"} Oct 30 05:15:59.259122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:15:59.259061 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" podStartSLOduration=2.259044355 podStartE2EDuration="2.259044355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:59.244477795 +0000 UTC m=+1883.772244593" watchObservedRunningTime="2026-10-30 05:15:59.259044355 +0000 UTC m=+1883.786811142" Oct 30 05:16:00.093627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:00.093591 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:16:00.099777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:00.099748 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:16:00.195198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:00.195170 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:16:00.196087 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:00.196062 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:16:03.635979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:03.635941 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv"] Oct 30 05:16:03.640700 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:03.640674 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" Oct 30 05:16:03.659646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:03.659605 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv"] Oct 30 05:16:03.667290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:03.667256 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e16297\"" Oct 30 05:16:03.747384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:03.738438 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2blz\" (UniqueName: \"kubernetes.io/projected/8fc270df-6323-4559-9598-3a0fa672d13d-kube-api-access-x2blz\") pod \"payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv\" (UID: \"8fc270df-6323-4559-9598-3a0fa672d13d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" Oct 30 05:16:03.790380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:03.748048 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc270df-6323-4559-9598-3a0fa672d13d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv\" (UID: \"8fc270df-6323-4559-9598-3a0fa672d13d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" Oct 30 05:16:03.790380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:03.753993 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e16297-dockercfg-tpbv7\"" Oct 30 05:16:03.909734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:03.908554 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2blz\" (UniqueName: \"kubernetes.io/projected/8fc270df-6323-4559-9598-3a0fa672d13d-kube-api-access-x2blz\") pod \"payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv\" (UID: \"8fc270df-6323-4559-9598-3a0fa672d13d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" Oct 30 05:16:03.909734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:03.908836 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc270df-6323-4559-9598-3a0fa672d13d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv\" (UID: \"8fc270df-6323-4559-9598-3a0fa672d13d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" Oct 30 05:16:03.910823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:03.910257 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc270df-6323-4559-9598-3a0fa672d13d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv\" (UID: \"8fc270df-6323-4559-9598-3a0fa672d13d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" Oct 30 05:16:03.968725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:03.964547 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2blz\" (UniqueName: \"kubernetes.io/projected/8fc270df-6323-4559-9598-3a0fa672d13d-kube-api-access-x2blz\") pod \"payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv\" (UID: \"8fc270df-6323-4559-9598-3a0fa672d13d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" Oct 30 05:16:03.968725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:03.966542 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" Oct 30 05:16:03.992665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:03.992611 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q"] Oct 30 05:16:04.007576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:04.004171 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" Oct 30 05:16:04.061396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:04.051249 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q"] Oct 30 05:16:04.150697 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:04.149843 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcndb\" (UniqueName: \"kubernetes.io/projected/02a6a95b-ab1a-441a-8052-ca050858b70a-kube-api-access-gcndb\") pod \"payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q\" (UID: \"02a6a95b-ab1a-441a-8052-ca050858b70a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" Oct 30 05:16:04.150697 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:04.150592 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a6a95b-ab1a-441a-8052-ca050858b70a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q\" (UID: \"02a6a95b-ab1a-441a-8052-ca050858b70a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" Oct 30 05:16:04.251956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:04.251861 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcndb\" (UniqueName: \"kubernetes.io/projected/02a6a95b-ab1a-441a-8052-ca050858b70a-kube-api-access-gcndb\") pod \"payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q\" (UID: \"02a6a95b-ab1a-441a-8052-ca050858b70a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" Oct 30 05:16:04.252115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:04.252093 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a6a95b-ab1a-441a-8052-ca050858b70a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q\" (UID: \"02a6a95b-ab1a-441a-8052-ca050858b70a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" Oct 30 05:16:04.252816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:04.252796 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a6a95b-ab1a-441a-8052-ca050858b70a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q\" (UID: \"02a6a95b-ab1a-441a-8052-ca050858b70a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" Oct 30 05:16:04.272407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:04.272346 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcndb\" (UniqueName: \"kubernetes.io/projected/02a6a95b-ab1a-441a-8052-ca050858b70a-kube-api-access-gcndb\") pod \"payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q\" (UID: \"02a6a95b-ab1a-441a-8052-ca050858b70a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" Oct 30 05:16:04.338078 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:16:04.338035 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc270df_6323_4559_9598_3a0fa672d13d.slice/crio-a88f78e77a02190feeba48281a3a48073fa8ca7fd295b2e936761526743de858 WatchSource:0}: Error finding container a88f78e77a02190feeba48281a3a48073fa8ca7fd295b2e936761526743de858: Status 404 returned error can't find the container with id a88f78e77a02190feeba48281a3a48073fa8ca7fd295b2e936761526743de858 Oct 30 05:16:04.338780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:04.338754 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv"] Oct 30 05:16:04.390114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:04.390085 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" Oct 30 05:16:04.701290 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:16:04.701251 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a6a95b_ab1a_441a_8052_ca050858b70a.slice/crio-88802efed95ef2b475d99e822a34392d26d128e5b3418d2cf6cff59efd02b2c8 WatchSource:0}: Error finding container 88802efed95ef2b475d99e822a34392d26d128e5b3418d2cf6cff59efd02b2c8: Status 404 returned error can't find the container with id 88802efed95ef2b475d99e822a34392d26d128e5b3418d2cf6cff59efd02b2c8 Oct 30 05:16:04.703131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:04.703103 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q"] Oct 30 05:16:05.240932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:05.240896 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" event={"ID":"02a6a95b-ab1a-441a-8052-ca050858b70a","Type":"ContainerStarted","Data":"6d100dd31096fae724a21d4bfd034164535476d58941a8560e04efc8732c60c9"} Oct 30 05:16:05.241162 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:05.240954 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" event={"ID":"02a6a95b-ab1a-441a-8052-ca050858b70a","Type":"ContainerStarted","Data":"88802efed95ef2b475d99e822a34392d26d128e5b3418d2cf6cff59efd02b2c8"} Oct 30 05:16:05.241162 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:05.241129 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" Oct 30 05:16:05.243152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:05.243118 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" event={"ID":"8fc270df-6323-4559-9598-3a0fa672d13d","Type":"ContainerStarted","Data":"fe6808189a05eb7f455ed4d21ab61a36c87188d178637c28421e20a02c194385"} Oct 30 05:16:05.243315 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:05.243169 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" event={"ID":"8fc270df-6323-4559-9598-3a0fa672d13d","Type":"ContainerStarted","Data":"a88f78e77a02190feeba48281a3a48073fa8ca7fd295b2e936761526743de858"} Oct 30 05:16:05.243315 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:05.243200 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" Oct 30 05:16:05.282913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:05.282864 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" podStartSLOduration=2.282848164 podStartE2EDuration="2.282848164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:05.280961062 +0000 UTC m=+1889.808727849" watchObservedRunningTime="2026-10-30 05:16:05.282848164 +0000 UTC m=+1889.810614949" Oct 30 05:16:05.325347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:05.325293 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" podStartSLOduration=2.325279782 podStartE2EDuration="2.325279782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:05.323261757 +0000 UTC m=+1889.851028543" watchObservedRunningTime="2026-10-30 05:16:05.325279782 +0000 UTC m=+1889.853046610" Oct 30 05:16:11.556382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:11.555518 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:16:11.583382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:11.582485 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:16:16.263403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:16.263327 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" Oct 30 05:16:16.264252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:16.264224 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" Oct 30 05:16:21.497286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:21.497227 6587 generic.go:356] "Generic (PLEG): container finished" podID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerID="790730cb8d823c19a885dcfe6b420dfb145711d6756de4efe9584410072d187e" exitCode=0 Oct 30 05:16:21.497994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:21.497306 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" event={"ID":"cb57c417-36f4-4a6b-9521-9bcfd52abf8c","Type":"ContainerDied","Data":"790730cb8d823c19a885dcfe6b420dfb145711d6756de4efe9584410072d187e"} Oct 30 05:16:21.498782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:21.498632 6587 scope.go:122] "RemoveContainer" containerID="790730cb8d823c19a885dcfe6b420dfb145711d6756de4efe9584410072d187e" Oct 30 05:16:21.502746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:21.502197 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:16:21.502746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:21.502302 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:16:21.502746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:21.502396 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:16:22.420778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:22.420741 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r"] Oct 30 05:16:22.435618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:22.435584 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c3486b-9fk6r"] Oct 30 05:16:22.517722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:22.517679 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" event={"ID":"cb57c417-36f4-4a6b-9521-9bcfd52abf8c","Type":"ContainerStarted","Data":"84d4ecc6b6ecf5a2dcfa8e1d9aedf9254e966eac463b790fae718324982688dd"} Oct 30 05:16:23.168876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:23.168762 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:16:23.169019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:23.168962 6587 patch_prober.go:29] interesting pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 05:16:23.169072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:23.169023 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 05:16:23.585039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:23.585003 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e371f129-d805-4277-973e-3717fdb94ca6" path="/var/lib/kubelet/pods/e371f129-d805-4277-973e-3717fdb94ca6/volumes" Oct 30 05:16:24.169164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:24.169126 6587 patch_prober.go:29] interesting pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 05:16:24.169395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:24.169187 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 05:16:25.169228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:25.169187 6587 patch_prober.go:29] interesting pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 05:16:25.169664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:25.169262 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 05:16:26.004894 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:26.004856 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54"] Oct 30 05:16:26.005135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:26.005104 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" podUID="8358c921-c2fe-4b5f-8374-c466080e8631" containerName="istio-proxy" containerID="cri-o://3f3b9ec08203c24a468e95b3925ba0cdee74abc6b5642890d29c76afe90ea365" gracePeriod=30 Oct 30 05:16:26.010809 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:16:26.010783 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:16:26.180228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:26.179787 6587 patch_prober.go:29] interesting pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 05:16:26.180228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:26.179849 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 05:16:27.168784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:27.168751 6587 patch_prober.go:29] interesting pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 05:16:27.169091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:27.168828 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 05:16:28.169156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:28.169114 6587 patch_prober.go:29] interesting pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 05:16:28.169620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:28.169194 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 05:16:29.175035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:29.175004 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:16:29.175516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:29.175476 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:16:29.176164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:29.176145 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:16:30.307189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:30.293978 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk"] Oct 30 05:16:30.307189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:30.294221 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" containerID="cri-o://9e53bd822b72b2677720d4076af28b0df85381bd517fe96b5103efc2f11dfaa4" gracePeriod=30 Oct 30 05:16:30.522438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:30.522407 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt"] Oct 30 05:16:30.522904 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:30.522875 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" containerID="cri-o://3c78bf483706a8240b1864e3a3405aa0cd547f44b1954e4842d5f597d7baff59" gracePeriod=30 Oct 30 05:16:30.534284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:30.532944 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5"] Oct 30 05:16:30.534284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:30.533210 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" podUID="19f5e053-b8c8-4250-b07d-233a445dc717" containerName="istio-proxy" containerID="cri-o://b43fbc9187a2b82bf1da4e57fe3852a5d1a49de62e728a898dde9152c14376ae" gracePeriod=30 Oct 30 05:16:30.543739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:30.536227 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx"] Oct 30 05:16:30.543739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:30.536449 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerName="istio-proxy" containerID="cri-o://4fee2c5ea10ba025fec580594655be2a8a3c76b349e98db7edb7c7df27c06d2f" gracePeriod=30 Oct 30 05:16:31.376138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.376109 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 05:16:31.570381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.569435 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-certs\") pod \"8358c921-c2fe-4b5f-8374-c466080e8631\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " Oct 30 05:16:31.594786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.594693 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-certs" pod "8358c921-c2fe-4b5f-8374-c466080e8631" (UID: "8358c921-c2fe-4b5f-8374-c466080e8631"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:31.660270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.625772 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-data\") pod \"8358c921-c2fe-4b5f-8374-c466080e8631\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " Oct 30 05:16:31.660270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.625831 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8358c921-c2fe-4b5f-8374-c466080e8631-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8358c921-c2fe-4b5f-8374-c466080e8631-istiod-ca-cert\") pod \"8358c921-c2fe-4b5f-8374-c466080e8631\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " Oct 30 05:16:31.660270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.625882 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-envoy\") pod \"8358c921-c2fe-4b5f-8374-c466080e8631\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " Oct 30 05:16:31.660270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.625945 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-socket\") pod \"8358c921-c2fe-4b5f-8374-c466080e8631\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " Oct 30 05:16:31.660270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.626155 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-kube-api-access-qbfcs\" (UniqueName: \"kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-kube-api-access-qbfcs\") pod \"8358c921-c2fe-4b5f-8374-c466080e8631\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " Oct 30 05:16:31.660270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.626282 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-credential-socket\") pod \"8358c921-c2fe-4b5f-8374-c466080e8631\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " Oct 30 05:16:31.660270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.626322 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-istio-token\" (UniqueName: \"kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-istio-token\") pod \"8358c921-c2fe-4b5f-8374-c466080e8631\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " Oct 30 05:16:31.660270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.626376 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8358c921-c2fe-4b5f-8374-c466080e8631-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8358c921-c2fe-4b5f-8374-c466080e8631-istio-podinfo\") pod \"8358c921-c2fe-4b5f-8374-c466080e8631\" (UID: \"8358c921-c2fe-4b5f-8374-c466080e8631\") " Oct 30 05:16:31.660270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.627214 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:31.660270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.648491 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-credential-socket" pod "8358c921-c2fe-4b5f-8374-c466080e8631" (UID: "8358c921-c2fe-4b5f-8374-c466080e8631"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:31.693780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.666593 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8358c921-c2fe-4b5f-8374-c466080e8631-istiod-ca-cert" pod "8358c921-c2fe-4b5f-8374-c466080e8631" (UID: "8358c921-c2fe-4b5f-8374-c466080e8631"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:16:31.693780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.669279 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-data" pod "8358c921-c2fe-4b5f-8374-c466080e8631" (UID: "8358c921-c2fe-4b5f-8374-c466080e8631"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:31.693780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.692935 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-socket" pod "8358c921-c2fe-4b5f-8374-c466080e8631" (UID: "8358c921-c2fe-4b5f-8374-c466080e8631"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:31.748413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.745746 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:31.748413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.745782 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8358c921-c2fe-4b5f-8374-c466080e8631-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:31.748413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.745798 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:31.748413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.745813 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:31.762597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.751838 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-istio-token" pod "8358c921-c2fe-4b5f-8374-c466080e8631" (UID: "8358c921-c2fe-4b5f-8374-c466080e8631"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:31.762597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.754651 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8358c921-c2fe-4b5f-8374-c466080e8631-istio-podinfo" pod "8358c921-c2fe-4b5f-8374-c466080e8631" (UID: "8358c921-c2fe-4b5f-8374-c466080e8631"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:16:31.762597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.759856 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-kube-api-access-qbfcs" pod "8358c921-c2fe-4b5f-8374-c466080e8631" (UID: "8358c921-c2fe-4b5f-8374-c466080e8631"). InnerVolumeSpecName "kube-api-access-qbfcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:31.762597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.760547 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-envoy" pod "8358c921-c2fe-4b5f-8374-c466080e8631" (UID: "8358c921-c2fe-4b5f-8374-c466080e8631"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:31.763256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.763209 6587 generic.go:356] "Generic (PLEG): container finished" podID="8358c921-c2fe-4b5f-8374-c466080e8631" containerID="3f3b9ec08203c24a468e95b3925ba0cdee74abc6b5642890d29c76afe90ea365" exitCode=0 Oct 30 05:16:31.763421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.763338 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" Oct 30 05:16:31.764164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.764140 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" event={"ID":"8358c921-c2fe-4b5f-8374-c466080e8631","Type":"ContainerDied","Data":"3f3b9ec08203c24a468e95b3925ba0cdee74abc6b5642890d29c76afe90ea365"} Oct 30 05:16:31.764277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.764177 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54" event={"ID":"8358c921-c2fe-4b5f-8374-c466080e8631","Type":"ContainerDied","Data":"40da0bac71973a17f06e6414e55c9ff5d0095c3ad48784e87190727d41925690"} Oct 30 05:16:31.764277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.764209 6587 scope.go:122] "RemoveContainer" containerID="3f3b9ec08203c24a468e95b3925ba0cdee74abc6b5642890d29c76afe90ea365" Oct 30 05:16:31.781806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.781786 6587 scope.go:122] "RemoveContainer" containerID="3f3b9ec08203c24a468e95b3925ba0cdee74abc6b5642890d29c76afe90ea365" Oct 30 05:16:31.782106 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:16:31.782083 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3b9ec08203c24a468e95b3925ba0cdee74abc6b5642890d29c76afe90ea365\": container with ID starting with 3f3b9ec08203c24a468e95b3925ba0cdee74abc6b5642890d29c76afe90ea365 not found: ID does not exist" containerID="3f3b9ec08203c24a468e95b3925ba0cdee74abc6b5642890d29c76afe90ea365" Oct 30 05:16:31.782219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.782116 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3b9ec08203c24a468e95b3925ba0cdee74abc6b5642890d29c76afe90ea365"} err="failed to get container status \"3f3b9ec08203c24a468e95b3925ba0cdee74abc6b5642890d29c76afe90ea365\": rpc error: code = NotFound desc = could not find container \"3f3b9ec08203c24a468e95b3925ba0cdee74abc6b5642890d29c76afe90ea365\": container with ID starting with 3f3b9ec08203c24a468e95b3925ba0cdee74abc6b5642890d29c76afe90ea365 not found: ID does not exist" Oct 30 05:16:31.824277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.824246 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54"] Oct 30 05:16:31.838351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.838309 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54"] Oct 30 05:16:31.847053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.846989 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbfcs\" (UniqueName: \"kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-kube-api-access-qbfcs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:31.847053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.847019 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8358c921-c2fe-4b5f-8374-c466080e8631-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:31.847053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.847030 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8358c921-c2fe-4b5f-8374-c466080e8631-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:31.847053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.847041 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8358c921-c2fe-4b5f-8374-c466080e8631-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:31.857616 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:16:31.857586 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8358c921_c2fe_4b5f_8374_c466080e8631.slice/crio-40da0bac71973a17f06e6414e55c9ff5d0095c3ad48784e87190727d41925690\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8358c921_c2fe_4b5f_8374_c466080e8631.slice\": RecentStats: unable to find data in memory cache]" Oct 30 05:16:31.929756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.929723 6587 patch_prober.go:29] interesting pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:16:31.929958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:31.929774 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:16:33.589597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:33.589555 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8358c921-c2fe-4b5f-8374-c466080e8631" path="/var/lib/kubelet/pods/8358c921-c2fe-4b5f-8374-c466080e8631/volumes" Oct 30 05:16:35.809514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.807841 6587 generic.go:356] "Generic (PLEG): container finished" podID="19f5e053-b8c8-4250-b07d-233a445dc717" containerID="b43fbc9187a2b82bf1da4e57fe3852a5d1a49de62e728a898dde9152c14376ae" exitCode=0 Oct 30 05:16:35.809514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.807925 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" event={"ID":"19f5e053-b8c8-4250-b07d-233a445dc717","Type":"ContainerDied","Data":"b43fbc9187a2b82bf1da4e57fe3852a5d1a49de62e728a898dde9152c14376ae"} Oct 30 05:16:35.820828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.810470 6587 generic.go:356] "Generic (PLEG): container finished" podID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerID="3c78bf483706a8240b1864e3a3405aa0cd547f44b1954e4842d5f597d7baff59" exitCode=0 Oct 30 05:16:35.820828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.810587 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" event={"ID":"e53c02d6-a2cb-4d27-b746-10f4a67978b7","Type":"ContainerDied","Data":"3c78bf483706a8240b1864e3a3405aa0cd547f44b1954e4842d5f597d7baff59"} Oct 30 05:16:35.820828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.812790 6587 generic.go:356] "Generic (PLEG): container finished" podID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerID="4fee2c5ea10ba025fec580594655be2a8a3c76b349e98db7edb7c7df27c06d2f" exitCode=0 Oct 30 05:16:35.820828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.812991 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" event={"ID":"7fc74df1-27eb-433d-9e9f-0ad481d00df1","Type":"ContainerDied","Data":"4fee2c5ea10ba025fec580594655be2a8a3c76b349e98db7edb7c7df27c06d2f"} Oct 30 05:16:35.820828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.815428 6587 generic.go:356] "Generic (PLEG): container finished" podID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerID="9e53bd822b72b2677720d4076af28b0df85381bd517fe96b5103efc2f11dfaa4" exitCode=0 Oct 30 05:16:35.820828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.815538 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" event={"ID":"e5e4e00a-ee98-4cf5-a016-cbf70647fea9","Type":"ContainerDied","Data":"9e53bd822b72b2677720d4076af28b0df85381bd517fe96b5103efc2f11dfaa4"} Oct 30 05:16:35.820828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.815580 6587 scope.go:122] "RemoveContainer" containerID="8c851debf1dd9da389a6ab50742afa32f1556ac8fabe2b89a39b5d92266a3d52" Oct 30 05:16:35.896396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.893875 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 05:16:35.922731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.922696 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-socket\") pod \"19f5e053-b8c8-4250-b07d-233a445dc717\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " Oct 30 05:16:35.922899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.922820 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f5e053-b8c8-4250-b07d-233a445dc717-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f5e053-b8c8-4250-b07d-233a445dc717-istiod-ca-cert\") pod \"19f5e053-b8c8-4250-b07d-233a445dc717\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " Oct 30 05:16:35.922899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.922858 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-certs\") pod \"19f5e053-b8c8-4250-b07d-233a445dc717\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " Oct 30 05:16:35.923025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.922922 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-istio-token\" (UniqueName: \"kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-istio-token\") pod \"19f5e053-b8c8-4250-b07d-233a445dc717\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " Oct 30 05:16:35.923113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.923096 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-credential-socket\") pod \"19f5e053-b8c8-4250-b07d-233a445dc717\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " Oct 30 05:16:35.923186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.923172 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-envoy\") pod \"19f5e053-b8c8-4250-b07d-233a445dc717\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " Oct 30 05:16:35.923292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.923206 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-kube-api-access-bdmkk\" (UniqueName: \"kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-kube-api-access-bdmkk\") pod \"19f5e053-b8c8-4250-b07d-233a445dc717\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " Oct 30 05:16:35.923292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.923254 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19f5e053-b8c8-4250-b07d-233a445dc717-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f5e053-b8c8-4250-b07d-233a445dc717-istio-podinfo\") pod \"19f5e053-b8c8-4250-b07d-233a445dc717\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " Oct 30 05:16:35.923292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.923283 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-data\") pod \"19f5e053-b8c8-4250-b07d-233a445dc717\" (UID: \"19f5e053-b8c8-4250-b07d-233a445dc717\") " Oct 30 05:16:35.926089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.925225 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f5e053-b8c8-4250-b07d-233a445dc717-istiod-ca-cert" pod "19f5e053-b8c8-4250-b07d-233a445dc717" (UID: "19f5e053-b8c8-4250-b07d-233a445dc717"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:16:35.926089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.925537 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-socket" pod "19f5e053-b8c8-4250-b07d-233a445dc717" (UID: "19f5e053-b8c8-4250-b07d-233a445dc717"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:35.926089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.925804 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-credential-socket" pod "19f5e053-b8c8-4250-b07d-233a445dc717" (UID: "19f5e053-b8c8-4250-b07d-233a445dc717"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:35.926089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.926038 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-certs" pod "19f5e053-b8c8-4250-b07d-233a445dc717" (UID: "19f5e053-b8c8-4250-b07d-233a445dc717"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:35.926589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.926566 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-data" pod "19f5e053-b8c8-4250-b07d-233a445dc717" (UID: "19f5e053-b8c8-4250-b07d-233a445dc717"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:35.932758 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.931754 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-envoy" pod "19f5e053-b8c8-4250-b07d-233a445dc717" (UID: "19f5e053-b8c8-4250-b07d-233a445dc717"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:35.933522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.933453 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19f5e053-b8c8-4250-b07d-233a445dc717-istio-podinfo" pod "19f5e053-b8c8-4250-b07d-233a445dc717" (UID: "19f5e053-b8c8-4250-b07d-233a445dc717"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:16:35.936344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.936323 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-kube-api-access-bdmkk" pod "19f5e053-b8c8-4250-b07d-233a445dc717" (UID: "19f5e053-b8c8-4250-b07d-233a445dc717"). InnerVolumeSpecName "kube-api-access-bdmkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:35.937662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.937637 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-istio-token" pod "19f5e053-b8c8-4250-b07d-233a445dc717" (UID: "19f5e053-b8c8-4250-b07d-233a445dc717"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:35.946133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:35.946033 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 05:16:36.018378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.018027 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:16:36.028123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.028095 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-socket\") pod \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " Oct 30 05:16:36.028297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.028226 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-envoy\") pod \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " Oct 30 05:16:36.028297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.028267 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istiod-ca-cert\") pod \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " Oct 30 05:16:36.028472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.028310 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-token\" (UniqueName: \"kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-token\") pod \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " Oct 30 05:16:36.028472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.028348 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-podinfo\") pod \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " Oct 30 05:16:36.028472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.028413 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-data\") pod \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " Oct 30 05:16:36.028635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.028504 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-kube-api-access-s2lwp\" (UniqueName: \"kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-kube-api-access-s2lwp\") pod \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " Oct 30 05:16:36.028635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.028598 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-certs\") pod \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " Oct 30 05:16:36.028736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.028691 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-credential-socket\") pod \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\" (UID: \"e53c02d6-a2cb-4d27-b746-10f4a67978b7\") " Oct 30 05:16:36.029674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.029534 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istiod-ca-cert" pod "e53c02d6-a2cb-4d27-b746-10f4a67978b7" (UID: "e53c02d6-a2cb-4d27-b746-10f4a67978b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:16:36.029863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.029842 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-certs" pod "e53c02d6-a2cb-4d27-b746-10f4a67978b7" (UID: "e53c02d6-a2cb-4d27-b746-10f4a67978b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.029863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.029850 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-socket" pod "e53c02d6-a2cb-4d27-b746-10f4a67978b7" (UID: "e53c02d6-a2cb-4d27-b746-10f4a67978b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.030142 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.030108 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-credential-socket" pod "e53c02d6-a2cb-4d27-b746-10f4a67978b7" (UID: "e53c02d6-a2cb-4d27-b746-10f4a67978b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.030142 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.030119 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-data" pod "e53c02d6-a2cb-4d27-b746-10f4a67978b7" (UID: "e53c02d6-a2cb-4d27-b746-10f4a67978b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.032464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.032089 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-token" pod "e53c02d6-a2cb-4d27-b746-10f4a67978b7" (UID: "e53c02d6-a2cb-4d27-b746-10f4a67978b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:36.032464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.032327 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.032464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.032348 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.032464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.032404 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.032464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.032420 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.033226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.033192 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-podinfo" pod "e53c02d6-a2cb-4d27-b746-10f4a67978b7" (UID: "e53c02d6-a2cb-4d27-b746-10f4a67978b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:16:36.033345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.033319 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.033476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.033368 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.033476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.033384 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdmkk\" (UniqueName: \"kubernetes.io/projected/19f5e053-b8c8-4250-b07d-233a445dc717-kube-api-access-bdmkk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.033476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.033402 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f5e053-b8c8-4250-b07d-233a445dc717-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.033476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.033418 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.033476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.033432 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.033476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.033445 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.033476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.033459 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.033476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.033472 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.033886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.033486 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f5e053-b8c8-4250-b07d-233a445dc717-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.033886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.033499 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f5e053-b8c8-4250-b07d-233a445dc717-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.033886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.033607 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-kube-api-access-s2lwp" pod "e53c02d6-a2cb-4d27-b746-10f4a67978b7" (UID: "e53c02d6-a2cb-4d27-b746-10f4a67978b7"). InnerVolumeSpecName "kube-api-access-s2lwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:36.034273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.034249 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-envoy" pod "e53c02d6-a2cb-4d27-b746-10f4a67978b7" (UID: "e53c02d6-a2cb-4d27-b746-10f4a67978b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.083023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.082944 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 05:16:36.145010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.141694 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-certs" pod "e5e4e00a-ee98-4cf5-a016-cbf70647fea9" (UID: "e5e4e00a-ee98-4cf5-a016-cbf70647fea9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.141419 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-certs\") pod \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.152162 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-kube-api-access-lhckd\" (UniqueName: \"kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-kube-api-access-lhckd\") pod \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.152195 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-token\" (UniqueName: \"kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-token\") pod \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.152229 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-kube-api-access-7bqhk\" (UniqueName: \"kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-kube-api-access-7bqhk\") pod \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.152295 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-data\") pod \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.152334 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-credential-socket\") pod \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.152382 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-socket\") pod \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.152427 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-podinfo\") pod \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.152513 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istiod-ca-cert\") pod \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.152564 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istiod-ca-cert\") pod \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.152614 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-credential-socket\") pod \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.152682 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-envoy\") pod \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.152767 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-socket\") pod \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.153037 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-certs\") pod \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.153132 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-podinfo\") pod \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.153210 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-envoy\") pod \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.153251 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-token\" (UniqueName: \"kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-token\") pod \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\" (UID: \"e5e4e00a-ee98-4cf5-a016-cbf70647fea9\") " Oct 30 05:16:36.153992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.153323 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-data\") pod \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\" (UID: \"7fc74df1-27eb-433d-9e9f-0ad481d00df1\") " Oct 30 05:16:36.156768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.154974 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istiod-ca-cert" pod "7fc74df1-27eb-433d-9e9f-0ad481d00df1" (UID: "7fc74df1-27eb-433d-9e9f-0ad481d00df1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:16:36.169185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.168849 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-kube-api-access-lhckd" pod "7fc74df1-27eb-433d-9e9f-0ad481d00df1" (UID: "7fc74df1-27eb-433d-9e9f-0ad481d00df1"). InnerVolumeSpecName "kube-api-access-lhckd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:36.172185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.171809 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-token" pod "7fc74df1-27eb-433d-9e9f-0ad481d00df1" (UID: "7fc74df1-27eb-433d-9e9f-0ad481d00df1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:36.177198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.176981 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-kube-api-access-7bqhk" pod "e5e4e00a-ee98-4cf5-a016-cbf70647fea9" (UID: "e5e4e00a-ee98-4cf5-a016-cbf70647fea9"). InnerVolumeSpecName "kube-api-access-7bqhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:36.177835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.177659 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-data" pod "e5e4e00a-ee98-4cf5-a016-cbf70647fea9" (UID: "e5e4e00a-ee98-4cf5-a016-cbf70647fea9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.178238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.178063 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-credential-socket" pod "7fc74df1-27eb-433d-9e9f-0ad481d00df1" (UID: "7fc74df1-27eb-433d-9e9f-0ad481d00df1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.178762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.178477 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-socket" pod "7fc74df1-27eb-433d-9e9f-0ad481d00df1" (UID: "7fc74df1-27eb-433d-9e9f-0ad481d00df1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.181654 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.181687 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhckd\" (UniqueName: \"kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-kube-api-access-lhckd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.181701 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.181735 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bqhk\" (UniqueName: \"kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-kube-api-access-7bqhk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.181749 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.181762 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.181775 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.181790 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.181803 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53c02d6-a2cb-4d27-b746-10f4a67978b7-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.181817 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.181830 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2lwp\" (UniqueName: \"kubernetes.io/projected/e53c02d6-a2cb-4d27-b746-10f4a67978b7-kube-api-access-s2lwp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.184235 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-credential-socket" pod "e5e4e00a-ee98-4cf5-a016-cbf70647fea9" (UID: "e5e4e00a-ee98-4cf5-a016-cbf70647fea9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.184880 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-podinfo" pod "7fc74df1-27eb-433d-9e9f-0ad481d00df1" (UID: "7fc74df1-27eb-433d-9e9f-0ad481d00df1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.185128 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-data" pod "7fc74df1-27eb-433d-9e9f-0ad481d00df1" (UID: "7fc74df1-27eb-433d-9e9f-0ad481d00df1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.185592 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-envoy" pod "e5e4e00a-ee98-4cf5-a016-cbf70647fea9" (UID: "e5e4e00a-ee98-4cf5-a016-cbf70647fea9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.185925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.185886 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-socket" pod "e5e4e00a-ee98-4cf5-a016-cbf70647fea9" (UID: "e5e4e00a-ee98-4cf5-a016-cbf70647fea9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.187744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.187708 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-podinfo" pod "e5e4e00a-ee98-4cf5-a016-cbf70647fea9" (UID: "e5e4e00a-ee98-4cf5-a016-cbf70647fea9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:16:36.195120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.195076 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-envoy" pod "7fc74df1-27eb-433d-9e9f-0ad481d00df1" (UID: "7fc74df1-27eb-433d-9e9f-0ad481d00df1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.202782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.202734 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-certs" pod "7fc74df1-27eb-433d-9e9f-0ad481d00df1" (UID: "7fc74df1-27eb-433d-9e9f-0ad481d00df1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:36.205199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.205171 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istiod-ca-cert" pod "e5e4e00a-ee98-4cf5-a016-cbf70647fea9" (UID: "e5e4e00a-ee98-4cf5-a016-cbf70647fea9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:16:36.209685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.209530 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-token" pod "e5e4e00a-ee98-4cf5-a016-cbf70647fea9" (UID: "e5e4e00a-ee98-4cf5-a016-cbf70647fea9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:36.283714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.283684 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.283714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.283711 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.283714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.283721 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.284050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.283734 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.284050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.283746 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.284050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.283754 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.284050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.283764 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.284050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.283773 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.284050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.283782 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e4e00a-ee98-4cf5-a016-cbf70647fea9-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.284050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.283789 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc74df1-27eb-433d-9e9f-0ad481d00df1-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:16:36.827519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.827475 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc"] Oct 30 05:16:36.829273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.829236 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" Oct 30 05:16:36.829458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.829442 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" Oct 30 05:16:36.829581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.829570 6587 cpu_manager.go:414] "containerMap: removing container" podUID="8358c921-c2fe-4b5f-8374-c466080e8631" containerName="istio-proxy" Oct 30 05:16:36.829652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.829641 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8358c921-c2fe-4b5f-8374-c466080e8631" containerName="istio-proxy" Oct 30 05:16:36.829757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.829734 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" Oct 30 05:16:36.829757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.829748 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" Oct 30 05:16:36.829945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.829769 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" Oct 30 05:16:36.829945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.829775 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" Oct 30 05:16:36.829945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.829785 6587 cpu_manager.go:414] "containerMap: removing container" podUID="19f5e053-b8c8-4250-b07d-233a445dc717" containerName="istio-proxy" Oct 30 05:16:36.829945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.829789 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f5e053-b8c8-4250-b07d-233a445dc717" containerName="istio-proxy" Oct 30 05:16:36.829945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.829803 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerName="istio-proxy" Oct 30 05:16:36.829945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.829808 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerName="istio-proxy" Oct 30 05:16:36.830237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.829986 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" Oct 30 05:16:36.830237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.830000 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" containerName="istio-proxy" Oct 30 05:16:36.830237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.830012 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f5e053-b8c8-4250-b07d-233a445dc717" containerName="istio-proxy" Oct 30 05:16:36.830237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.830036 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" containerName="istio-proxy" Oct 30 05:16:36.830237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.830055 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" containerName="istio-proxy" Oct 30 05:16:36.830237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.830069 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="8358c921-c2fe-4b5f-8374-c466080e8631" containerName="istio-proxy" Oct 30 05:16:36.833983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.833959 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" Oct 30 05:16:36.836915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.836859 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" Oct 30 05:16:36.837036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.836904 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5" event={"ID":"19f5e053-b8c8-4250-b07d-233a445dc717","Type":"ContainerDied","Data":"313935acdafdc000f408a4e62bf132be485b6559621129509d39337dd22be346"} Oct 30 05:16:36.837036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.836963 6587 scope.go:122] "RemoveContainer" containerID="b43fbc9187a2b82bf1da4e57fe3852a5d1a49de62e728a898dde9152c14376ae" Oct 30 05:16:36.839292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.839267 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" event={"ID":"e53c02d6-a2cb-4d27-b746-10f4a67978b7","Type":"ContainerDied","Data":"444d242516eab07563fedd267443ac73d77dfcbc351093a8bbe944816ddb6483"} Oct 30 05:16:36.839431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.839384 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt" Oct 30 05:16:36.850717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.850665 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" event={"ID":"7fc74df1-27eb-433d-9e9f-0ad481d00df1","Type":"ContainerDied","Data":"977f880a4d585e5c8b8edbb7f610f2b0d4ab0cae560838bfa4a70196a4ff5842"} Oct 30 05:16:36.850907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.850788 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx" Oct 30 05:16:36.854770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.854745 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc"] Oct 30 05:16:36.857536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.857301 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-802b8a\"" Oct 30 05:16:36.858240 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.858216 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" event={"ID":"e5e4e00a-ee98-4cf5-a016-cbf70647fea9","Type":"ContainerDied","Data":"c543fb4fa9f74c5880b543949a646b54a2b766f82b77877a53b46a33c00ba81c"} Oct 30 05:16:36.858465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.858446 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk" Oct 30 05:16:36.861875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.861853 6587 scope.go:122] "RemoveContainer" containerID="3c78bf483706a8240b1864e3a3405aa0cd547f44b1954e4842d5f597d7baff59" Oct 30 05:16:36.876825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.876807 6587 scope.go:122] "RemoveContainer" containerID="4fee2c5ea10ba025fec580594655be2a8a3c76b349e98db7edb7c7df27c06d2f" Oct 30 05:16:36.889001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.888965 6587 scope.go:122] "RemoveContainer" containerID="9e53bd822b72b2677720d4076af28b0df85381bd517fe96b5103efc2f11dfaa4" Oct 30 05:16:36.937085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.937048 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx"] Oct 30 05:16:36.953011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.952977 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx"] Oct 30 05:16:36.970176 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.970056 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7"] Oct 30 05:16:36.987434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:36.984627 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" Oct 30 05:16:37.007391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.006586 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10ce723-0350-4b2f-8180-32bea097068c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc\" (UID: \"b10ce723-0350-4b2f-8180-32bea097068c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" Oct 30 05:16:37.007391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.006638 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw2gt\" (UniqueName: \"kubernetes.io/projected/b10ce723-0350-4b2f-8180-32bea097068c-kube-api-access-jw2gt\") pod \"payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc\" (UID: \"b10ce723-0350-4b2f-8180-32bea097068c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" Oct 30 05:16:37.023380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.021233 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-802b8a-dockercfg-26gpj\"" Oct 30 05:16:37.111451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.110836 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10ce723-0350-4b2f-8180-32bea097068c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc\" (UID: \"b10ce723-0350-4b2f-8180-32bea097068c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" Oct 30 05:16:37.111451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.110890 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw2gt\" (UniqueName: \"kubernetes.io/projected/b10ce723-0350-4b2f-8180-32bea097068c-kube-api-access-jw2gt\") pod \"payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc\" (UID: \"b10ce723-0350-4b2f-8180-32bea097068c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" Oct 30 05:16:37.111451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.110983 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmdw9\" (UniqueName: \"kubernetes.io/projected/aeea73b7-9493-4e07-9ede-23123f96f897-kube-api-access-tmdw9\") pod \"payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7\" (UID: \"aeea73b7-9493-4e07-9ede-23123f96f897\") " pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" Oct 30 05:16:37.111451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.111127 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeea73b7-9493-4e07-9ede-23123f96f897-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7\" (UID: \"aeea73b7-9493-4e07-9ede-23123f96f897\") " pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" Oct 30 05:16:37.124529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.124497 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7"] Oct 30 05:16:37.136567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.136536 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10ce723-0350-4b2f-8180-32bea097068c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc\" (UID: \"b10ce723-0350-4b2f-8180-32bea097068c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" Oct 30 05:16:37.202026 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.201963 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw2gt\" (UniqueName: \"kubernetes.io/projected/b10ce723-0350-4b2f-8180-32bea097068c-kube-api-access-jw2gt\") pod \"payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc\" (UID: \"b10ce723-0350-4b2f-8180-32bea097068c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" Oct 30 05:16:37.208757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.208687 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt"] Oct 30 05:16:37.212498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.212462 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmdw9\" (UniqueName: \"kubernetes.io/projected/aeea73b7-9493-4e07-9ede-23123f96f897-kube-api-access-tmdw9\") pod \"payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7\" (UID: \"aeea73b7-9493-4e07-9ede-23123f96f897\") " pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" Oct 30 05:16:37.212675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.212595 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeea73b7-9493-4e07-9ede-23123f96f897-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7\" (UID: \"aeea73b7-9493-4e07-9ede-23123f96f897\") " pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" Oct 30 05:16:37.213591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.213511 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeea73b7-9493-4e07-9ede-23123f96f897-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7\" (UID: \"aeea73b7-9493-4e07-9ede-23123f96f897\") " pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" Oct 30 05:16:37.225839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.225766 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt"] Oct 30 05:16:37.230148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.230124 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5"] Oct 30 05:16:37.235424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.235401 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5"] Oct 30 05:16:37.240628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.240604 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk"] Oct 30 05:16:37.243738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.243715 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk"] Oct 30 05:16:37.245522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.245300 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmdw9\" (UniqueName: \"kubernetes.io/projected/aeea73b7-9493-4e07-9ede-23123f96f897-kube-api-access-tmdw9\") pod \"payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7\" (UID: \"aeea73b7-9493-4e07-9ede-23123f96f897\") " pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" Oct 30 05:16:37.467703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.467617 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" Oct 30 05:16:37.489484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.489457 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" Oct 30 05:16:37.590044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.589996 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f5e053-b8c8-4250-b07d-233a445dc717" path="/var/lib/kubelet/pods/19f5e053-b8c8-4250-b07d-233a445dc717/volumes" Oct 30 05:16:37.591193 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.591167 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc74df1-27eb-433d-9e9f-0ad481d00df1" path="/var/lib/kubelet/pods/7fc74df1-27eb-433d-9e9f-0ad481d00df1/volumes" Oct 30 05:16:37.592001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.591982 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e53c02d6-a2cb-4d27-b746-10f4a67978b7" path="/var/lib/kubelet/pods/e53c02d6-a2cb-4d27-b746-10f4a67978b7/volumes" Oct 30 05:16:37.593218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.593198 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e4e00a-ee98-4cf5-a016-cbf70647fea9" path="/var/lib/kubelet/pods/e5e4e00a-ee98-4cf5-a016-cbf70647fea9/volumes" Oct 30 05:16:37.774862 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:16:37.774823 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10ce723_0350_4b2f_8180_32bea097068c.slice/crio-8604234b439a6baf4af5c6edde1e74c426f8dc8f997e231f6fba693936c5b89a WatchSource:0}: Error finding container 8604234b439a6baf4af5c6edde1e74c426f8dc8f997e231f6fba693936c5b89a: Status 404 returned error can't find the container with id 8604234b439a6baf4af5c6edde1e74c426f8dc8f997e231f6fba693936c5b89a Oct 30 05:16:37.776473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.776448 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc"] Oct 30 05:16:37.810414 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:16:37.810368 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeea73b7_9493_4e07_9ede_23123f96f897.slice/crio-0eceb360ec6869d128ffcd1427e8f4aac3c704fa6179d1903d0e004abf01dc24 WatchSource:0}: Error finding container 0eceb360ec6869d128ffcd1427e8f4aac3c704fa6179d1903d0e004abf01dc24: Status 404 returned error can't find the container with id 0eceb360ec6869d128ffcd1427e8f4aac3c704fa6179d1903d0e004abf01dc24 Oct 30 05:16:37.813427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.813401 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7"] Oct 30 05:16:37.873083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.873048 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" event={"ID":"b10ce723-0350-4b2f-8180-32bea097068c","Type":"ContainerStarted","Data":"8604234b439a6baf4af5c6edde1e74c426f8dc8f997e231f6fba693936c5b89a"} Oct 30 05:16:37.876093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:37.876066 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" event={"ID":"aeea73b7-9493-4e07-9ede-23123f96f897","Type":"ContainerStarted","Data":"0eceb360ec6869d128ffcd1427e8f4aac3c704fa6179d1903d0e004abf01dc24"} Oct 30 05:16:38.887875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:38.887838 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" event={"ID":"aeea73b7-9493-4e07-9ede-23123f96f897","Type":"ContainerStarted","Data":"ad8656fd05d3ab92806c38deb7628a8d68a27afcec8198923c2907c098de6b21"} Oct 30 05:16:38.888343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:38.888042 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" Oct 30 05:16:38.889741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:38.889715 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" event={"ID":"b10ce723-0350-4b2f-8180-32bea097068c","Type":"ContainerStarted","Data":"9e66fa0ba50ae834f4cfaf302e00d302c7a296e71aa872a94464b64152d95187"} Oct 30 05:16:38.889867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:38.889762 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" Oct 30 05:16:38.932149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:38.932086 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" podStartSLOduration=2.932069132 podStartE2EDuration="2.932069132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:38.931612254 +0000 UTC m=+1923.459379040" watchObservedRunningTime="2026-10-30 05:16:38.932069132 +0000 UTC m=+1923.459835919" Oct 30 05:16:38.977593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:38.977543 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" podStartSLOduration=2.977528885 podStartE2EDuration="2.977528885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:38.974525292 +0000 UTC m=+1923.502292079" watchObservedRunningTime="2026-10-30 05:16:38.977528885 +0000 UTC m=+1923.505295671" Oct 30 05:16:39.827383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:39.823202 6587 scope.go:122] "RemoveContainer" containerID="744bc187addfd64ca9887801eb411a945300c7fe92968dad785cd233d0b0ee1a" Oct 30 05:16:39.846954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:39.846926 6587 scope.go:122] "RemoveContainer" containerID="fd94f09d18152ecd726bc1e141d6cb0d22558d73b232c1c7a55be82b7a2155ad" Oct 30 05:16:39.863407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:39.863332 6587 scope.go:122] "RemoveContainer" containerID="07de315e7e3fca9a7a91d6a03f21fc32a9fe41832d4864c8545bf41d09b10b76" Oct 30 05:16:44.033914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:44.033876 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld"] Oct 30 05:16:44.047297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:44.047261 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0a27f4-4rqld"] Oct 30 05:16:45.585171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:45.585135 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa45334-b5d2-4cd7-ad85-91ff4f8953c5" path="/var/lib/kubelet/pods/faa45334-b5d2-4cd7-ad85-91ff4f8953c5/volumes" Oct 30 05:16:46.859168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:46.859122 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s"] Oct 30 05:16:46.865674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:46.865650 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:46.889389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:46.886401 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s"] Oct 30 05:16:46.892014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:46.891990 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-a914e9-openshift-default-dockercfg-zl54b\"" Oct 30 05:16:46.989105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:46.989068 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:46.989390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:46.989345 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv48n\" (UniqueName: \"kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-kube-api-access-dv48n\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:46.989651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:46.989629 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:46.989831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:46.989806 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:46.990587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:46.989960 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:46.990587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:46.990033 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:46.990587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:46.990087 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:46.990587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:46.990165 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:46.990587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:46.990221 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.102383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.096685 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.102383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.096761 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.102383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.096824 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.102383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.096874 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.102383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.096949 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.102383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.096999 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.102383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.097174 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.102383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.097201 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv48n\" (UniqueName: \"kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-kube-api-access-dv48n\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.102383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.097313 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.102383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.101036 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.102383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.101306 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.102383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.101976 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.103149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.102835 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.122500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.116837 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.122500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.120711 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.122500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.121184 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.142407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.137619 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv48n\" (UniqueName: \"kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-kube-api-access-dv48n\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.142407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.142169 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.200383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.199835 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:47.602305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.602273 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s"] Oct 30 05:16:47.603121 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:16:47.603097 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a9f78e_8bc2_4bf0_adf1_eab0ca0bf81d.slice/crio-08c7b22aed689244620d9bbfa6857a934e4ea4bc07635b5da61e6a9cdc4c6712 WatchSource:0}: Error finding container 08c7b22aed689244620d9bbfa6857a934e4ea4bc07635b5da61e6a9cdc4c6712: Status 404 returned error can't find the container with id 08c7b22aed689244620d9bbfa6857a934e4ea4bc07635b5da61e6a9cdc4c6712 Oct 30 05:16:47.606410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.606338 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:16:47.606504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.606449 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:16:47.606504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.606480 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:16:47.979045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.978945 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" event={"ID":"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d","Type":"ContainerStarted","Data":"dd4db8126824b483c87e4f80ff2c90a5acaa3b1f79d4e9729222d31794d52103"} Oct 30 05:16:47.979045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:47.979006 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" event={"ID":"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d","Type":"ContainerStarted","Data":"08c7b22aed689244620d9bbfa6857a934e4ea4bc07635b5da61e6a9cdc4c6712"} Oct 30 05:16:48.029667 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:48.029592 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" podStartSLOduration=2.029572851 podStartE2EDuration="2.029572851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:48.024031002 +0000 UTC m=+1932.551797789" watchObservedRunningTime="2026-10-30 05:16:48.029572851 +0000 UTC m=+1932.557339638" Oct 30 05:16:48.204198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:48.202305 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:48.208891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:48.208861 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:48.987061 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:48.987029 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:48.988209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:48.988179 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:16:49.902987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:49.902961 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" Oct 30 05:16:49.903167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:49.903025 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" Oct 30 05:16:52.305660 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.305624 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm"] Oct 30 05:16:52.309418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.309396 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" Oct 30 05:16:52.338837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.338801 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm"] Oct 30 05:16:52.340091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.340064 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-a914e9\"" Oct 30 05:16:52.366812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.366775 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z"] Oct 30 05:16:52.371746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.371720 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" Oct 30 05:16:52.388348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.388322 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-a914e9-dockercfg-l4knh\"" Oct 30 05:16:52.388835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.388813 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z"] Oct 30 05:16:52.390265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.390236 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhzlj\" (UniqueName: \"kubernetes.io/projected/2492f676-52a6-4199-8401-36f35dd25a66-kube-api-access-xhzlj\") pod \"payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z\" (UID: \"2492f676-52a6-4199-8401-36f35dd25a66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" Oct 30 05:16:52.390368 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.390267 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2492f676-52a6-4199-8401-36f35dd25a66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z\" (UID: \"2492f676-52a6-4199-8401-36f35dd25a66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" Oct 30 05:16:52.390508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.390481 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnw2v\" (UniqueName: \"kubernetes.io/projected/e92a12c9-eb28-4851-b02b-f574892dfb49-kube-api-access-cnw2v\") pod \"payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm\" (UID: \"e92a12c9-eb28-4851-b02b-f574892dfb49\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" Oct 30 05:16:52.390601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.390587 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92a12c9-eb28-4851-b02b-f574892dfb49-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm\" (UID: \"e92a12c9-eb28-4851-b02b-f574892dfb49\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" Oct 30 05:16:52.570435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.559010 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92a12c9-eb28-4851-b02b-f574892dfb49-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm\" (UID: \"e92a12c9-eb28-4851-b02b-f574892dfb49\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" Oct 30 05:16:52.571002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.570968 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhzlj\" (UniqueName: \"kubernetes.io/projected/2492f676-52a6-4199-8401-36f35dd25a66-kube-api-access-xhzlj\") pod \"payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z\" (UID: \"2492f676-52a6-4199-8401-36f35dd25a66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" Oct 30 05:16:52.571183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.571167 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2492f676-52a6-4199-8401-36f35dd25a66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z\" (UID: \"2492f676-52a6-4199-8401-36f35dd25a66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" Oct 30 05:16:52.571434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.571413 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnw2v\" (UniqueName: \"kubernetes.io/projected/e92a12c9-eb28-4851-b02b-f574892dfb49-kube-api-access-cnw2v\") pod \"payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm\" (UID: \"e92a12c9-eb28-4851-b02b-f574892dfb49\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" Oct 30 05:16:52.593451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.572819 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92a12c9-eb28-4851-b02b-f574892dfb49-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm\" (UID: \"e92a12c9-eb28-4851-b02b-f574892dfb49\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" Oct 30 05:16:52.637783 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.586268 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2492f676-52a6-4199-8401-36f35dd25a66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z\" (UID: \"2492f676-52a6-4199-8401-36f35dd25a66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" Oct 30 05:16:52.709439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.709380 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhzlj\" (UniqueName: \"kubernetes.io/projected/2492f676-52a6-4199-8401-36f35dd25a66-kube-api-access-xhzlj\") pod \"payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z\" (UID: \"2492f676-52a6-4199-8401-36f35dd25a66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" Oct 30 05:16:52.712306 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.712233 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnw2v\" (UniqueName: \"kubernetes.io/projected/e92a12c9-eb28-4851-b02b-f574892dfb49-kube-api-access-cnw2v\") pod \"payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm\" (UID: \"e92a12c9-eb28-4851-b02b-f574892dfb49\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" Oct 30 05:16:52.925056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.924965 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" Oct 30 05:16:52.991966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:52.991930 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" Oct 30 05:16:53.231289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:53.231260 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm"] Oct 30 05:16:53.336752 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:16:53.336708 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2492f676_52a6_4199_8401_36f35dd25a66.slice/crio-9c752ea4ba19c9a760b61ae8b16ddd7b75e60fa4b5196a1d2d83beae854d8d84 WatchSource:0}: Error finding container 9c752ea4ba19c9a760b61ae8b16ddd7b75e60fa4b5196a1d2d83beae854d8d84: Status 404 returned error can't find the container with id 9c752ea4ba19c9a760b61ae8b16ddd7b75e60fa4b5196a1d2d83beae854d8d84 Oct 30 05:16:53.339103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:53.339078 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z"] Oct 30 05:16:54.043990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:54.043948 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" event={"ID":"e92a12c9-eb28-4851-b02b-f574892dfb49","Type":"ContainerStarted","Data":"9797ceaef15b7f674b9aca7453396fc1caf944aacbc7a4b5eff184bb18662635"} Oct 30 05:16:54.044194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:54.044015 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" event={"ID":"e92a12c9-eb28-4851-b02b-f574892dfb49","Type":"ContainerStarted","Data":"24d30ba4d20773cd7cc628dabaa0d473c90020f8b9767f87137a39fdcc770295"} Oct 30 05:16:54.044255 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:54.044212 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" Oct 30 05:16:54.046184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:54.046153 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" event={"ID":"2492f676-52a6-4199-8401-36f35dd25a66","Type":"ContainerStarted","Data":"981c84e58bc070fc84bde94ea43ad2bcc53a41082006150e8287148b3fe2896f"} Oct 30 05:16:54.046325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:54.046207 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" event={"ID":"2492f676-52a6-4199-8401-36f35dd25a66","Type":"ContainerStarted","Data":"9c752ea4ba19c9a760b61ae8b16ddd7b75e60fa4b5196a1d2d83beae854d8d84"} Oct 30 05:16:54.046325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:54.046233 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" Oct 30 05:16:54.088693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:54.088645 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" podStartSLOduration=2.088631405 podStartE2EDuration="2.088631405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:54.085400229 +0000 UTC m=+1938.613167006" watchObservedRunningTime="2026-10-30 05:16:54.088631405 +0000 UTC m=+1938.616398192" Oct 30 05:16:54.129719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:16:54.129643 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" podStartSLOduration=2.12962556 podStartE2EDuration="2.12962556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:54.128044477 +0000 UTC m=+1938.655811262" watchObservedRunningTime="2026-10-30 05:16:54.12962556 +0000 UTC m=+1938.657392347" Oct 30 05:16:55.986771 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:16:55.986736 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:17:05.059138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:05.059102 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" Oct 30 05:17:05.059771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:05.059176 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" Oct 30 05:17:25.986353 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:17:25.986315 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:17:38.031428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:38.031387 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t"] Oct 30 05:17:38.046960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:38.046922 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0cb4e82-6667t"] Oct 30 05:17:39.584572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:39.584541 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f8e41c-430f-49a5-ac42-ac192db72560" path="/var/lib/kubelet/pods/97f8e41c-430f-49a5-ac42-ac192db72560/volumes" Oct 30 05:17:39.951104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:39.951029 6587 scope.go:122] "RemoveContainer" containerID="256aae25030a239a61d45db8f58ca3482940b001f751b5eae02c0d08f4dc89fe" Oct 30 05:17:39.963764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:39.963737 6587 scope.go:122] "RemoveContainer" containerID="cac5beb02c6e492329ffb789441d353859a293faf20002c1e2d96868c3dd0457" Oct 30 05:17:39.974157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:39.974135 6587 scope.go:122] "RemoveContainer" containerID="6f332238272d7e975d5e455f7c495b3a11f40427f2db6f770813c49079dcf80b" Oct 30 05:17:41.527472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:41.527433 6587 generic.go:356] "Generic (PLEG): container finished" podID="abdd1519-522f-4302-844d-947822be0960" containerID="35f1f9dff0904c02d65598be5658e7005270f90d83036415c62557860378e5cc" exitCode=0 Oct 30 05:17:41.528041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:41.527511 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" event={"ID":"abdd1519-522f-4302-844d-947822be0960","Type":"ContainerDied","Data":"35f1f9dff0904c02d65598be5658e7005270f90d83036415c62557860378e5cc"} Oct 30 05:17:41.528041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:41.527945 6587 scope.go:122] "RemoveContainer" containerID="35f1f9dff0904c02d65598be5658e7005270f90d83036415c62557860378e5cc" Oct 30 05:17:41.529906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:41.529880 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:17:41.529990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:41.529948 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:17:41.529990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:41.529981 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:17:42.557039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:42.557000 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" event={"ID":"abdd1519-522f-4302-844d-947822be0960","Type":"ContainerStarted","Data":"366a66da5be1ff07fa5ca3138ada6e1e84affcf9d31e4e9c8fc0627be2d5aa4e"} Oct 30 05:17:43.092982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:43.092943 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:17:43.094499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:43.094471 6587 patch_prober.go:29] interesting pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 05:17:43.094639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:43.094518 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" podUID="abdd1519-522f-4302-844d-947822be0960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 05:17:44.093423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:44.093352 6587 patch_prober.go:29] interesting pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 05:17:44.093856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:44.093442 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" podUID="abdd1519-522f-4302-844d-947822be0960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 05:17:45.093173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:45.093138 6587 patch_prober.go:29] interesting pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 05:17:45.093401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:45.093190 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" podUID="abdd1519-522f-4302-844d-947822be0960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 05:17:45.195819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:45.195785 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:17:46.093540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:46.093505 6587 patch_prober.go:29] interesting pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 05:17:46.093726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:46.093560 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" podUID="abdd1519-522f-4302-844d-947822be0960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 05:17:47.093302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:47.093206 6587 patch_prober.go:29] interesting pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 05:17:47.093302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:47.093282 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" podUID="abdd1519-522f-4302-844d-947822be0960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 05:17:47.893268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:47.893218 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676"] Oct 30 05:17:47.898398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:47.898338 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:47.914347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:47.914312 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676"] Oct 30 05:17:47.925406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:47.923794 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2a8f18\"/\"test-model-2a8f18-kserve-self-signed-certs\"" Oct 30 05:17:47.936028 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:47.936000 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2a8f18\"/\"kube-root-ca.crt\"" Oct 30 05:17:47.936325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:47.936280 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2a8f18\"/\"default-dockercfg-5s8gw\"" Oct 30 05:17:47.936581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:47.936562 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2a8f18\"/\"openshift-service-ca.crt\"" Oct 30 05:17:48.060726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.060680 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-dshm\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.060943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.060821 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9zl6\" (UniqueName: \"kubernetes.io/projected/57f4169e-d26c-48d5-bcc8-fbfb3022972a-kube-api-access-r9zl6\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.060943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.060909 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tmp-dir\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.061128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.061030 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-model-cache\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.061196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.061176 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-home\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.061422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.061397 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tls-certs\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.164430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.164292 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tls-certs\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.164430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.164406 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-dshm\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.164969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.164464 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zl6\" (UniqueName: \"kubernetes.io/projected/57f4169e-d26c-48d5-bcc8-fbfb3022972a-kube-api-access-r9zl6\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.164969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.164532 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tmp-dir\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.164969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.164583 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-model-cache\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.164969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.164690 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-home\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.165205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.165187 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-home\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.167385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.166691 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-model-cache\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.173228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.173203 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tls-certs\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.183257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.174575 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tmp-dir\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.185383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.185341 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-dshm\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.198390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.190791 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9zl6\" (UniqueName: \"kubernetes.io/projected/57f4169e-d26c-48d5-bcc8-fbfb3022972a-kube-api-access-r9zl6\") pod \"test-model-2a8f18-kserve-5876b794f5-94676\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.227699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.227578 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:48.311011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.310981 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:17:48.316065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.315816 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:17:48.591782 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:17:48.591749 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57f4169e_d26c_48d5_bcc8_fbfb3022972a.slice/crio-a03b7b86732f128ff762a49a9ed28f1c838384e0fdc52bee310ffbab0f9f8240 WatchSource:0}: Error finding container a03b7b86732f128ff762a49a9ed28f1c838384e0fdc52bee310ffbab0f9f8240: Status 404 returned error can't find the container with id a03b7b86732f128ff762a49a9ed28f1c838384e0fdc52bee310ffbab0f9f8240 Oct 30 05:17:48.593380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.593233 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676"] Oct 30 05:17:48.651139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:48.651108 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" event={"ID":"57f4169e-d26c-48d5-bcc8-fbfb3022972a","Type":"ContainerStarted","Data":"a03b7b86732f128ff762a49a9ed28f1c838384e0fdc52bee310ffbab0f9f8240"} Oct 30 05:17:49.660366 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:49.660318 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" event={"ID":"57f4169e-d26c-48d5-bcc8-fbfb3022972a","Type":"ContainerStarted","Data":"78cf8b4c3cb09fed0be742ef8127d222952d9dc67a4eac78eacafcd346716443"} Oct 30 05:17:49.706948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:49.706891 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" podStartSLOduration=2.706876922 podStartE2EDuration="2.706876922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:17:49.703643101 +0000 UTC m=+1994.231409888" watchObservedRunningTime="2026-10-30 05:17:49.706876922 +0000 UTC m=+1994.234643707" Oct 30 05:17:50.028685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:50.028581 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4"] Oct 30 05:17:50.031947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:50.031917 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72a37b86-nmsp4"] Oct 30 05:17:50.228181 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:50.228139 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:51.584273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:51.584238 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10bbc35b-d12d-41c4-acd9-6d3a484459a8" path="/var/lib/kubelet/pods/10bbc35b-d12d-41c4-acd9-6d3a484459a8/volumes" Oct 30 05:17:54.721957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.721918 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b"] Oct 30 05:17:54.725780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.725757 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" Oct 30 05:17:54.750015 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.749985 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b"] Oct 30 05:17:54.758334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.758305 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:17:54.758531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.758483 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:17:54.759936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.759910 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:17:54.759936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.759921 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:17:54.779036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.779001 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e8efdc2-tw82b\" (UID: \"e9368d63-2c7a-4b6e-a818-e52d05407199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" Oct 30 05:17:54.779183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.779136 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s2vr\" (UniqueName: \"kubernetes.io/projected/e9368d63-2c7a-4b6e-a818-e52d05407199-kube-api-access-9s2vr\") pod \"maas-api-revoke-sub-0e8efdc2-tw82b\" (UID: \"e9368d63-2c7a-4b6e-a818-e52d05407199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" Oct 30 05:17:54.779252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.779213 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e8efdc2-tw82b\" (UID: \"e9368d63-2c7a-4b6e-a818-e52d05407199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" Oct 30 05:17:54.882700 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.882391 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s2vr\" (UniqueName: \"kubernetes.io/projected/e9368d63-2c7a-4b6e-a818-e52d05407199-kube-api-access-9s2vr\") pod \"maas-api-revoke-sub-0e8efdc2-tw82b\" (UID: \"e9368d63-2c7a-4b6e-a818-e52d05407199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" Oct 30 05:17:54.882700 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.882472 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e8efdc2-tw82b\" (UID: \"e9368d63-2c7a-4b6e-a818-e52d05407199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" Oct 30 05:17:54.882978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.882805 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e8efdc2-tw82b\" (UID: \"e9368d63-2c7a-4b6e-a818-e52d05407199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" Oct 30 05:17:54.892380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.883698 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e8efdc2-tw82b\" (UID: \"e9368d63-2c7a-4b6e-a818-e52d05407199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" Oct 30 05:17:54.912404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.910190 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e8efdc2-tw82b\" (UID: \"e9368d63-2c7a-4b6e-a818-e52d05407199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" Oct 30 05:17:54.942380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:54.941666 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s2vr\" (UniqueName: \"kubernetes.io/projected/e9368d63-2c7a-4b6e-a818-e52d05407199-kube-api-access-9s2vr\") pod \"maas-api-revoke-sub-0e8efdc2-tw82b\" (UID: \"e9368d63-2c7a-4b6e-a818-e52d05407199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" Oct 30 05:17:55.043731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:55.043649 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" Oct 30 05:17:55.352015 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:17:55.351982 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9368d63_2c7a_4b6e_a818_e52d05407199.slice/crio-fc2b70cd9511a6a7df09257f419a70e26baf62ac38bb94ee732efac88e241b27 WatchSource:0}: Error finding container fc2b70cd9511a6a7df09257f419a70e26baf62ac38bb94ee732efac88e241b27: Status 404 returned error can't find the container with id fc2b70cd9511a6a7df09257f419a70e26baf62ac38bb94ee732efac88e241b27 Oct 30 05:17:55.352917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:55.352889 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b"] Oct 30 05:17:55.649406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:55.649294 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j"] Oct 30 05:17:55.649570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:55.649503 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" containerID="cri-o://21cdf1c63cef0ef58b0dfd95a16d5ebaf6d6c8af91a66a0d08e1f414aa8dd262" gracePeriod=30 Oct 30 05:17:55.712817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:55.712777 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" event={"ID":"e9368d63-2c7a-4b6e-a818-e52d05407199","Type":"ContainerStarted","Data":"fc2b70cd9511a6a7df09257f419a70e26baf62ac38bb94ee732efac88e241b27"} Oct 30 05:17:56.031117 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:17:56.030990 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:17:56.723267 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:56.723184 6587 generic.go:356] "Generic (PLEG): container finished" podID="e9368d63-2c7a-4b6e-a818-e52d05407199" containerID="d2d8512dd194ce8e4b45208430faec5e9c1642a38a92d65594db63aeb2d2ad4c" exitCode=0 Oct 30 05:17:56.723462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:56.723268 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" event={"ID":"e9368d63-2c7a-4b6e-a818-e52d05407199","Type":"ContainerDied","Data":"d2d8512dd194ce8e4b45208430faec5e9c1642a38a92d65594db63aeb2d2ad4c"} Oct 30 05:17:57.967559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:57.967533 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" Oct 30 05:17:57.978898 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:57.978872 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9368d63-2c7a-4b6e-a818-e52d05407199-kube-api-access-9s2vr\" (UniqueName: \"kubernetes.io/projected/e9368d63-2c7a-4b6e-a818-e52d05407199-kube-api-access-9s2vr\") pod \"e9368d63-2c7a-4b6e-a818-e52d05407199\" (UID: \"e9368d63-2c7a-4b6e-a818-e52d05407199\") " Oct 30 05:17:57.979073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:57.978955 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-openshift-ca\") pod \"e9368d63-2c7a-4b6e-a818-e52d05407199\" (UID: \"e9368d63-2c7a-4b6e-a818-e52d05407199\") " Oct 30 05:17:57.979202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:57.979184 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-xks-ca\") pod \"e9368d63-2c7a-4b6e-a818-e52d05407199\" (UID: \"e9368d63-2c7a-4b6e-a818-e52d05407199\") " Oct 30 05:17:57.980606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:57.980576 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-openshift-ca" pod "e9368d63-2c7a-4b6e-a818-e52d05407199" (UID: "e9368d63-2c7a-4b6e-a818-e52d05407199"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:17:57.981612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:57.981587 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9368d63-2c7a-4b6e-a818-e52d05407199-kube-api-access-9s2vr" pod "e9368d63-2c7a-4b6e-a818-e52d05407199" (UID: "e9368d63-2c7a-4b6e-a818-e52d05407199"). InnerVolumeSpecName "kube-api-access-9s2vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:17:57.982305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:57.982251 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-xks-ca" pod "e9368d63-2c7a-4b6e-a818-e52d05407199" (UID: "e9368d63-2c7a-4b6e-a818-e52d05407199"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:17:58.081619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:58.081584 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s2vr\" (UniqueName: \"kubernetes.io/projected/e9368d63-2c7a-4b6e-a818-e52d05407199-kube-api-access-9s2vr\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:17:58.081619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:58.081617 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:17:58.081619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:58.081627 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9368d63-2c7a-4b6e-a818-e52d05407199-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:17:58.229048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:58.229020 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:58.242512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:58.242435 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:17:58.744623 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:58.744593 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" Oct 30 05:17:58.744844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:58.744625 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b" event={"ID":"e9368d63-2c7a-4b6e-a818-e52d05407199","Type":"ContainerDied","Data":"fc2b70cd9511a6a7df09257f419a70e26baf62ac38bb94ee732efac88e241b27"} Oct 30 05:17:58.744844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:58.744671 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc2b70cd9511a6a7df09257f419a70e26baf62ac38bb94ee732efac88e241b27" Oct 30 05:17:58.762011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:17:58.761985 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:18:00.954963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:00.954938 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 05:18:01.026603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.026567 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-istio-token\" (UniqueName: \"kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-istio-token\") pod \"6df4067a-ac1c-496d-83d1-470dd606212d\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " Oct 30 05:18:01.026794 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.026614 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6df4067a-ac1c-496d-83d1-470dd606212d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df4067a-ac1c-496d-83d1-470dd606212d-istio-podinfo\") pod \"6df4067a-ac1c-496d-83d1-470dd606212d\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " Oct 30 05:18:01.026794 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.026664 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-certs\") pod \"6df4067a-ac1c-496d-83d1-470dd606212d\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " Oct 30 05:18:01.026794 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.026724 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-credential-socket\") pod \"6df4067a-ac1c-496d-83d1-470dd606212d\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " Oct 30 05:18:01.026794 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.026760 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-kube-api-access-7rd84\" (UniqueName: \"kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-kube-api-access-7rd84\") pod \"6df4067a-ac1c-496d-83d1-470dd606212d\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " Oct 30 05:18:01.027010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.026807 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-envoy\") pod \"6df4067a-ac1c-496d-83d1-470dd606212d\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " Oct 30 05:18:01.027010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.026932 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df4067a-ac1c-496d-83d1-470dd606212d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df4067a-ac1c-496d-83d1-470dd606212d-istiod-ca-cert\") pod \"6df4067a-ac1c-496d-83d1-470dd606212d\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " Oct 30 05:18:01.027138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.027098 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-socket\") pod \"6df4067a-ac1c-496d-83d1-470dd606212d\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " Oct 30 05:18:01.027204 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.027144 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-data\") pod \"6df4067a-ac1c-496d-83d1-470dd606212d\" (UID: \"6df4067a-ac1c-496d-83d1-470dd606212d\") " Oct 30 05:18:01.027319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.027288 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-certs" pod "6df4067a-ac1c-496d-83d1-470dd606212d" (UID: "6df4067a-ac1c-496d-83d1-470dd606212d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:01.027585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.027561 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-socket" pod "6df4067a-ac1c-496d-83d1-470dd606212d" (UID: "6df4067a-ac1c-496d-83d1-470dd606212d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:01.027970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.027687 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-data" pod "6df4067a-ac1c-496d-83d1-470dd606212d" (UID: "6df4067a-ac1c-496d-83d1-470dd606212d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:01.027970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.027715 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df4067a-ac1c-496d-83d1-470dd606212d-istiod-ca-cert" pod "6df4067a-ac1c-496d-83d1-470dd606212d" (UID: "6df4067a-ac1c-496d-83d1-470dd606212d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:18:01.027970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.027910 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-credential-socket" pod "6df4067a-ac1c-496d-83d1-470dd606212d" (UID: "6df4067a-ac1c-496d-83d1-470dd606212d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:01.028395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.028338 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.028395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.028379 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.028395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.028393 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.028608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.028406 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.028608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.028420 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df4067a-ac1c-496d-83d1-470dd606212d-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.029314 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.029287 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6df4067a-ac1c-496d-83d1-470dd606212d-istio-podinfo" pod "6df4067a-ac1c-496d-83d1-470dd606212d" (UID: "6df4067a-ac1c-496d-83d1-470dd606212d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:18:01.029470 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.029316 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-istio-token" pod "6df4067a-ac1c-496d-83d1-470dd606212d" (UID: "6df4067a-ac1c-496d-83d1-470dd606212d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:01.029956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.029935 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-kube-api-access-7rd84" pod "6df4067a-ac1c-496d-83d1-470dd606212d" (UID: "6df4067a-ac1c-496d-83d1-470dd606212d"). InnerVolumeSpecName "kube-api-access-7rd84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:01.030034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.029980 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-envoy" pod "6df4067a-ac1c-496d-83d1-470dd606212d" (UID: "6df4067a-ac1c-496d-83d1-470dd606212d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:01.130119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.130086 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.130119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.130116 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df4067a-ac1c-496d-83d1-470dd606212d-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.130119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.130127 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rd84\" (UniqueName: \"kubernetes.io/projected/6df4067a-ac1c-496d-83d1-470dd606212d-kube-api-access-7rd84\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.130479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.130141 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df4067a-ac1c-496d-83d1-470dd606212d-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:01.776956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.776924 6587 generic.go:356] "Generic (PLEG): container finished" podID="6df4067a-ac1c-496d-83d1-470dd606212d" containerID="21cdf1c63cef0ef58b0dfd95a16d5ebaf6d6c8af91a66a0d08e1f414aa8dd262" exitCode=0 Oct 30 05:18:01.777173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.776984 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" Oct 30 05:18:01.777173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.777008 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" event={"ID":"6df4067a-ac1c-496d-83d1-470dd606212d","Type":"ContainerDied","Data":"21cdf1c63cef0ef58b0dfd95a16d5ebaf6d6c8af91a66a0d08e1f414aa8dd262"} Oct 30 05:18:01.777173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.777069 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j" event={"ID":"6df4067a-ac1c-496d-83d1-470dd606212d","Type":"ContainerDied","Data":"3eac9c59af28aeddaa59374fe0c9dc35d864c6d8f585e1fc3ec4fc550c62cffe"} Oct 30 05:18:01.777173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.777080 6587 scope.go:122] "RemoveContainer" containerID="21cdf1c63cef0ef58b0dfd95a16d5ebaf6d6c8af91a66a0d08e1f414aa8dd262" Oct 30 05:18:01.791072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.791048 6587 scope.go:122] "RemoveContainer" containerID="3b6ecc9a07c19efada4e684d89ea5f4d5e2f37d9dbd7e4e2804f4542c7b55b20" Oct 30 05:18:01.801773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.801754 6587 scope.go:122] "RemoveContainer" containerID="21cdf1c63cef0ef58b0dfd95a16d5ebaf6d6c8af91a66a0d08e1f414aa8dd262" Oct 30 05:18:01.802055 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:18:01.802034 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21cdf1c63cef0ef58b0dfd95a16d5ebaf6d6c8af91a66a0d08e1f414aa8dd262\": container with ID starting with 21cdf1c63cef0ef58b0dfd95a16d5ebaf6d6c8af91a66a0d08e1f414aa8dd262 not found: ID does not exist" containerID="21cdf1c63cef0ef58b0dfd95a16d5ebaf6d6c8af91a66a0d08e1f414aa8dd262" Oct 30 05:18:01.802140 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.802061 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21cdf1c63cef0ef58b0dfd95a16d5ebaf6d6c8af91a66a0d08e1f414aa8dd262"} err="failed to get container status \"21cdf1c63cef0ef58b0dfd95a16d5ebaf6d6c8af91a66a0d08e1f414aa8dd262\": rpc error: code = NotFound desc = could not find container \"21cdf1c63cef0ef58b0dfd95a16d5ebaf6d6c8af91a66a0d08e1f414aa8dd262\": container with ID starting with 21cdf1c63cef0ef58b0dfd95a16d5ebaf6d6c8af91a66a0d08e1f414aa8dd262 not found: ID does not exist" Oct 30 05:18:01.802140 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.802079 6587 scope.go:122] "RemoveContainer" containerID="3b6ecc9a07c19efada4e684d89ea5f4d5e2f37d9dbd7e4e2804f4542c7b55b20" Oct 30 05:18:01.802320 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:18:01.802305 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6ecc9a07c19efada4e684d89ea5f4d5e2f37d9dbd7e4e2804f4542c7b55b20\": container with ID starting with 3b6ecc9a07c19efada4e684d89ea5f4d5e2f37d9dbd7e4e2804f4542c7b55b20 not found: ID does not exist" containerID="3b6ecc9a07c19efada4e684d89ea5f4d5e2f37d9dbd7e4e2804f4542c7b55b20" Oct 30 05:18:01.802404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.802326 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6ecc9a07c19efada4e684d89ea5f4d5e2f37d9dbd7e4e2804f4542c7b55b20"} err="failed to get container status \"3b6ecc9a07c19efada4e684d89ea5f4d5e2f37d9dbd7e4e2804f4542c7b55b20\": rpc error: code = NotFound desc = could not find container \"3b6ecc9a07c19efada4e684d89ea5f4d5e2f37d9dbd7e4e2804f4542c7b55b20\": container with ID starting with 3b6ecc9a07c19efada4e684d89ea5f4d5e2f37d9dbd7e4e2804f4542c7b55b20 not found: ID does not exist" Oct 30 05:18:01.843896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.843860 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j"] Oct 30 05:18:01.857196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:01.857132 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j"] Oct 30 05:18:03.585319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:03.585285 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" path="/var/lib/kubelet/pods/6df4067a-ac1c-496d-83d1-470dd606212d/volumes" Oct 30 05:18:12.516490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.516449 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh"] Oct 30 05:18:12.517807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.517689 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e9368d63-2c7a-4b6e-a818-e52d05407199" containerName="revoke-keys" Oct 30 05:18:12.517807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.517713 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9368d63-2c7a-4b6e-a818-e52d05407199" containerName="revoke-keys" Oct 30 05:18:12.517807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.517733 6587 cpu_manager.go:414] "containerMap: removing container" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" Oct 30 05:18:12.517807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.517740 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" Oct 30 05:18:12.517807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.517787 6587 cpu_manager.go:414] "containerMap: removing container" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" Oct 30 05:18:12.517807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.517795 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" Oct 30 05:18:12.518179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.518011 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9368d63-2c7a-4b6e-a818-e52d05407199" containerName="revoke-keys" Oct 30 05:18:12.518179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.518026 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" Oct 30 05:18:12.518179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.518045 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df4067a-ac1c-496d-83d1-470dd606212d" containerName="istio-proxy" Oct 30 05:18:12.524037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.524014 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" Oct 30 05:18:12.540487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.540458 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:18:12.542922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.542894 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh"] Oct 30 05:18:12.549403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.549353 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:18:12.549596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.549579 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:18:12.549596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.549589 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:18:12.624335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.624298 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9405336d-bqfdh\" (UID: \"993542b9-d84d-466b-8cbf-e43c3bee49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" Oct 30 05:18:12.624594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.624471 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md6hx\" (UniqueName: \"kubernetes.io/projected/993542b9-d84d-466b-8cbf-e43c3bee49cf-kube-api-access-md6hx\") pod \"maas-api-revoke-sub-9405336d-bqfdh\" (UID: \"993542b9-d84d-466b-8cbf-e43c3bee49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" Oct 30 05:18:12.624788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.624752 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9405336d-bqfdh\" (UID: \"993542b9-d84d-466b-8cbf-e43c3bee49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" Oct 30 05:18:12.725996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.725955 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md6hx\" (UniqueName: \"kubernetes.io/projected/993542b9-d84d-466b-8cbf-e43c3bee49cf-kube-api-access-md6hx\") pod \"maas-api-revoke-sub-9405336d-bqfdh\" (UID: \"993542b9-d84d-466b-8cbf-e43c3bee49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" Oct 30 05:18:12.726910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.726543 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9405336d-bqfdh\" (UID: \"993542b9-d84d-466b-8cbf-e43c3bee49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" Oct 30 05:18:12.726910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.726715 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9405336d-bqfdh\" (UID: \"993542b9-d84d-466b-8cbf-e43c3bee49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" Oct 30 05:18:12.728312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.728280 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9405336d-bqfdh\" (UID: \"993542b9-d84d-466b-8cbf-e43c3bee49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" Oct 30 05:18:12.744785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.743578 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9405336d-bqfdh\" (UID: \"993542b9-d84d-466b-8cbf-e43c3bee49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" Oct 30 05:18:12.748421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.747863 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md6hx\" (UniqueName: \"kubernetes.io/projected/993542b9-d84d-466b-8cbf-e43c3bee49cf-kube-api-access-md6hx\") pod \"maas-api-revoke-sub-9405336d-bqfdh\" (UID: \"993542b9-d84d-466b-8cbf-e43c3bee49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" Oct 30 05:18:12.842207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:12.842167 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" Oct 30 05:18:13.111178 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.111094 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7"] Oct 30 05:18:13.116969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.116945 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.143177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.143136 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7"] Oct 30 05:18:13.148338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.148304 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-68b893-openshift-default-dockercfg-6ns6m\"" Oct 30 05:18:13.170089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.170056 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh"] Oct 30 05:18:13.172530 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:18:13.172497 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993542b9_d84d_466b_8cbf_e43c3bee49cf.slice/crio-ea1e3e53a36807ee29ce1b37d2ce772e9850513344c4f0a414cfc868a67e1cbd WatchSource:0}: Error finding container ea1e3e53a36807ee29ce1b37d2ce772e9850513344c4f0a414cfc868a67e1cbd: Status 404 returned error can't find the container with id ea1e3e53a36807ee29ce1b37d2ce772e9850513344c4f0a414cfc868a67e1cbd Oct 30 05:18:13.267014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.266654 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.267014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.266758 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040a272c-aca3-4af4-86db-3055cbe66486-istiod-ca-cert\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.267014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.266874 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040a272c-aca3-4af4-86db-3055cbe66486-istio-podinfo\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.267014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.266921 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-data\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.267014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.266963 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-istio-token\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.267484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.267039 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-certs\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.267484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.267077 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-credential-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.267484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.267127 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5cln\" (UniqueName: \"kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-kube-api-access-r5cln\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.267484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.267309 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-envoy\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.371440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.370510 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-certs\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.371440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.370574 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-credential-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.371440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.370633 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5cln\" (UniqueName: \"kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-kube-api-access-r5cln\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.371440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.370807 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-envoy\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.371440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.370890 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.371440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.370966 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040a272c-aca3-4af4-86db-3055cbe66486-istiod-ca-cert\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.371440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.371080 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040a272c-aca3-4af4-86db-3055cbe66486-istio-podinfo\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.371440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.371126 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-data\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.371440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.371165 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-istio-token\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.390386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.372130 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-certs\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.390386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.372406 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-credential-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.390386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.373841 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040a272c-aca3-4af4-86db-3055cbe66486-istiod-ca-cert\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.390386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.374118 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.390386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.376701 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-data\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.390386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.382199 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040a272c-aca3-4af4-86db-3055cbe66486-istio-podinfo\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.390386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.384608 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-envoy\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.415995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.411890 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-istio-token\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.420378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.417117 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5cln\" (UniqueName: \"kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-kube-api-access-r5cln\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.446393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.445715 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:13.877330 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:18:13.877297 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040a272c_aca3_4af4_86db_3055cbe66486.slice/crio-2c41bac0f9d161829f32fc8ff96ae1ae65edbf30bb77fc2546d8b87723dd6080 WatchSource:0}: Error finding container 2c41bac0f9d161829f32fc8ff96ae1ae65edbf30bb77fc2546d8b87723dd6080: Status 404 returned error can't find the container with id 2c41bac0f9d161829f32fc8ff96ae1ae65edbf30bb77fc2546d8b87723dd6080 Oct 30 05:18:13.880172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.880124 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7"] Oct 30 05:18:13.880338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.880309 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:13.880431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.880412 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:13.880472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.880446 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:13.915896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.915864 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" event={"ID":"040a272c-aca3-4af4-86db-3055cbe66486","Type":"ContainerStarted","Data":"2c41bac0f9d161829f32fc8ff96ae1ae65edbf30bb77fc2546d8b87723dd6080"} Oct 30 05:18:13.918257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.918223 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" event={"ID":"993542b9-d84d-466b-8cbf-e43c3bee49cf","Type":"ContainerStarted","Data":"5211f742eb578720d118ba007c8c411b01c155a6e212bb7b7dd66527a008b955"} Oct 30 05:18:13.918413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.918292 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" event={"ID":"993542b9-d84d-466b-8cbf-e43c3bee49cf","Type":"ContainerStarted","Data":"ea1e3e53a36807ee29ce1b37d2ce772e9850513344c4f0a414cfc868a67e1cbd"} Oct 30 05:18:13.960861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:13.960808 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" podStartSLOduration=1.9607914210000001 podStartE2EDuration="1.960791421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:13.958735746 +0000 UTC m=+2018.486502533" watchObservedRunningTime="2026-10-30 05:18:13.960791421 +0000 UTC m=+2018.488558208" Oct 30 05:18:14.932760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:14.932728 6587 generic.go:356] "Generic (PLEG): container finished" podID="993542b9-d84d-466b-8cbf-e43c3bee49cf" containerID="5211f742eb578720d118ba007c8c411b01c155a6e212bb7b7dd66527a008b955" exitCode=0 Oct 30 05:18:14.933232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:14.932864 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" event={"ID":"993542b9-d84d-466b-8cbf-e43c3bee49cf","Type":"ContainerDied","Data":"5211f742eb578720d118ba007c8c411b01c155a6e212bb7b7dd66527a008b955"} Oct 30 05:18:14.935023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:14.934997 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" event={"ID":"040a272c-aca3-4af4-86db-3055cbe66486","Type":"ContainerStarted","Data":"0c00818e6ca0ce776a41115ffd356d70451429ff3d97332404c15ef5167596f9"} Oct 30 05:18:15.017621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:15.017569 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" podStartSLOduration=2.017550379 podStartE2EDuration="2.017550379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:15.016937437 +0000 UTC m=+2019.544704225" watchObservedRunningTime="2026-10-30 05:18:15.017550379 +0000 UTC m=+2019.545317164" Oct 30 05:18:15.446283 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:15.446245 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:15.454069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:15.454039 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:15.946300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:15.945799 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:15.946789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:15.946666 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:18:16.185461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:16.185426 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" Oct 30 05:18:16.234974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:16.234891 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-openshift-ca\") pod \"993542b9-d84d-466b-8cbf-e43c3bee49cf\" (UID: \"993542b9-d84d-466b-8cbf-e43c3bee49cf\") " Oct 30 05:18:16.235203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:16.235180 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993542b9-d84d-466b-8cbf-e43c3bee49cf-kube-api-access-md6hx\" (UniqueName: \"kubernetes.io/projected/993542b9-d84d-466b-8cbf-e43c3bee49cf-kube-api-access-md6hx\") pod \"993542b9-d84d-466b-8cbf-e43c3bee49cf\" (UID: \"993542b9-d84d-466b-8cbf-e43c3bee49cf\") " Oct 30 05:18:16.235455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:16.235419 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-openshift-ca" pod "993542b9-d84d-466b-8cbf-e43c3bee49cf" (UID: "993542b9-d84d-466b-8cbf-e43c3bee49cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:18:16.235704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:16.235684 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-xks-ca\") pod \"993542b9-d84d-466b-8cbf-e43c3bee49cf\" (UID: \"993542b9-d84d-466b-8cbf-e43c3bee49cf\") " Oct 30 05:18:16.236586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:16.236562 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:16.237893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:16.237863 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993542b9-d84d-466b-8cbf-e43c3bee49cf-kube-api-access-md6hx" pod "993542b9-d84d-466b-8cbf-e43c3bee49cf" (UID: "993542b9-d84d-466b-8cbf-e43c3bee49cf"). InnerVolumeSpecName "kube-api-access-md6hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:16.243508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:16.243479 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-xks-ca" pod "993542b9-d84d-466b-8cbf-e43c3bee49cf" (UID: "993542b9-d84d-466b-8cbf-e43c3bee49cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:18:16.338309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:16.338268 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md6hx\" (UniqueName: \"kubernetes.io/projected/993542b9-d84d-466b-8cbf-e43c3bee49cf-kube-api-access-md6hx\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:16.338309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:16.338307 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993542b9-d84d-466b-8cbf-e43c3bee49cf-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:16.957546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:16.957518 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" Oct 30 05:18:16.957945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:16.957558 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh" event={"ID":"993542b9-d84d-466b-8cbf-e43c3bee49cf","Type":"ContainerDied","Data":"ea1e3e53a36807ee29ce1b37d2ce772e9850513344c4f0a414cfc868a67e1cbd"} Oct 30 05:18:16.957945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:16.957616 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea1e3e53a36807ee29ce1b37d2ce772e9850513344c4f0a414cfc868a67e1cbd" Oct 30 05:18:19.265397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.265347 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb"] Oct 30 05:18:19.266204 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.266187 6587 cpu_manager.go:414] "containerMap: removing container" podUID="993542b9-d84d-466b-8cbf-e43c3bee49cf" containerName="revoke-keys" Oct 30 05:18:19.266251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.266204 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993542b9-d84d-466b-8cbf-e43c3bee49cf" containerName="revoke-keys" Oct 30 05:18:19.266324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.266314 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="993542b9-d84d-466b-8cbf-e43c3bee49cf" containerName="revoke-keys" Oct 30 05:18:19.270459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.270439 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" Oct 30 05:18:19.291056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.291024 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb"] Oct 30 05:18:19.293702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.293672 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-68b893\"" Oct 30 05:18:19.293853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.293792 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-68b893-dockercfg-pjgd2\"" Oct 30 05:18:19.402395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.399854 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659aff68-3d79-4f16-b239-23197429890c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb\" (UID: \"659aff68-3d79-4f16-b239-23197429890c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" Oct 30 05:18:19.402395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.400046 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9j8x\" (UniqueName: \"kubernetes.io/projected/659aff68-3d79-4f16-b239-23197429890c-kube-api-access-c9j8x\") pod \"payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb\" (UID: \"659aff68-3d79-4f16-b239-23197429890c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" Oct 30 05:18:19.540436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.507429 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659aff68-3d79-4f16-b239-23197429890c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb\" (UID: \"659aff68-3d79-4f16-b239-23197429890c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" Oct 30 05:18:19.552683 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.552642 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9j8x\" (UniqueName: \"kubernetes.io/projected/659aff68-3d79-4f16-b239-23197429890c-kube-api-access-c9j8x\") pod \"payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb\" (UID: \"659aff68-3d79-4f16-b239-23197429890c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" Oct 30 05:18:19.575433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.530483 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659aff68-3d79-4f16-b239-23197429890c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb\" (UID: \"659aff68-3d79-4f16-b239-23197429890c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" Oct 30 05:18:19.608816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.606214 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9j8x\" (UniqueName: \"kubernetes.io/projected/659aff68-3d79-4f16-b239-23197429890c-kube-api-access-c9j8x\") pod \"payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb\" (UID: \"659aff68-3d79-4f16-b239-23197429890c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" Oct 30 05:18:19.619511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.617216 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6"] Oct 30 05:18:19.644468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.644432 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6"] Oct 30 05:18:19.644672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.644491 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" Oct 30 05:18:19.765307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.765267 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d453e4b7-43e6-4463-869e-20060976c7ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6\" (UID: \"d453e4b7-43e6-4463-869e-20060976c7ca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" Oct 30 05:18:19.765532 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.765386 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwxsj\" (UniqueName: \"kubernetes.io/projected/d453e4b7-43e6-4463-869e-20060976c7ca-kube-api-access-kwxsj\") pod \"payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6\" (UID: \"d453e4b7-43e6-4463-869e-20060976c7ca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" Oct 30 05:18:19.866874 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.866826 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwxsj\" (UniqueName: \"kubernetes.io/projected/d453e4b7-43e6-4463-869e-20060976c7ca-kube-api-access-kwxsj\") pod \"payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6\" (UID: \"d453e4b7-43e6-4463-869e-20060976c7ca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" Oct 30 05:18:19.867151 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.867134 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d453e4b7-43e6-4463-869e-20060976c7ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6\" (UID: \"d453e4b7-43e6-4463-869e-20060976c7ca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" Oct 30 05:18:19.867842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.867822 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d453e4b7-43e6-4463-869e-20060976c7ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6\" (UID: \"d453e4b7-43e6-4463-869e-20060976c7ca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" Oct 30 05:18:19.886452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.886338 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" Oct 30 05:18:19.887392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.887335 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwxsj\" (UniqueName: \"kubernetes.io/projected/d453e4b7-43e6-4463-869e-20060976c7ca-kube-api-access-kwxsj\") pod \"payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6\" (UID: \"d453e4b7-43e6-4463-869e-20060976c7ca\") " pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" Oct 30 05:18:19.964719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:19.964689 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" Oct 30 05:18:20.201078 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:18:20.201035 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659aff68_3d79_4f16_b239_23197429890c.slice/crio-2fbbe8e7a9fb8648a70ee32f3cb5c66c1691984947a776f887554493e95167a0 WatchSource:0}: Error finding container 2fbbe8e7a9fb8648a70ee32f3cb5c66c1691984947a776f887554493e95167a0: Status 404 returned error can't find the container with id 2fbbe8e7a9fb8648a70ee32f3cb5c66c1691984947a776f887554493e95167a0 Oct 30 05:18:20.202948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:20.202927 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb"] Oct 30 05:18:20.291453 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:18:20.291425 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd453e4b7_43e6_4463_869e_20060976c7ca.slice/crio-cc886c43ed7948abfdf708bef6275cd8ef9aad3444aefcdd46f98888e785cb91 WatchSource:0}: Error finding container cc886c43ed7948abfdf708bef6275cd8ef9aad3444aefcdd46f98888e785cb91: Status 404 returned error can't find the container with id cc886c43ed7948abfdf708bef6275cd8ef9aad3444aefcdd46f98888e785cb91 Oct 30 05:18:20.294690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:20.294666 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6"] Oct 30 05:18:21.029873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:21.029502 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" event={"ID":"659aff68-3d79-4f16-b239-23197429890c","Type":"ContainerStarted","Data":"6e12c3df4a138f8c16e2a4b3de3e40fceb674419e40bbc156364b3cbe4ec970f"} Oct 30 05:18:21.029873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:21.029573 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" event={"ID":"659aff68-3d79-4f16-b239-23197429890c","Type":"ContainerStarted","Data":"2fbbe8e7a9fb8648a70ee32f3cb5c66c1691984947a776f887554493e95167a0"} Oct 30 05:18:21.029873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:21.029624 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" Oct 30 05:18:21.031630 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:21.031597 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" event={"ID":"d453e4b7-43e6-4463-869e-20060976c7ca","Type":"ContainerStarted","Data":"e63ef0f1bfc5f3df6be8b80a5cdefd671276f0dd2b8efd9629c3709c0fff31f5"} Oct 30 05:18:21.031784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:21.031650 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" event={"ID":"d453e4b7-43e6-4463-869e-20060976c7ca","Type":"ContainerStarted","Data":"cc886c43ed7948abfdf708bef6275cd8ef9aad3444aefcdd46f98888e785cb91"} Oct 30 05:18:21.031784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:21.031705 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" Oct 30 05:18:21.073514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:21.073452 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" podStartSLOduration=2.073434001 podStartE2EDuration="2.073434001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:21.070538819 +0000 UTC m=+2025.598305604" watchObservedRunningTime="2026-10-30 05:18:21.073434001 +0000 UTC m=+2025.601200789" Oct 30 05:18:21.116573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:21.116522 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" podStartSLOduration=2.116507649 podStartE2EDuration="2.116507649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:21.110516359 +0000 UTC m=+2025.638283144" watchObservedRunningTime="2026-10-30 05:18:21.116507649 +0000 UTC m=+2025.644274440" Oct 30 05:18:25.505917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.505877 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf"] Oct 30 05:18:25.512319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.512291 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" Oct 30 05:18:25.531201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.531162 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf"] Oct 30 05:18:25.547557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.535769 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:18:25.547557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.535778 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:18:25.547557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.535867 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:18:25.547557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.545715 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:18:25.574912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.574874 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf\" (UID: \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" Oct 30 05:18:25.617352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.617310 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c7rw\" (UniqueName: \"kubernetes.io/projected/750eb7e3-21cf-4ccc-b546-dc6385746b2a-kube-api-access-2c7rw\") pod \"maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf\" (UID: \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" Oct 30 05:18:25.641771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.641733 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf\" (UID: \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" Oct 30 05:18:25.753509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.753474 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf\" (UID: \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" Oct 30 05:18:25.755529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.755495 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf\" (UID: \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" Oct 30 05:18:25.755956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.755902 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c7rw\" (UniqueName: \"kubernetes.io/projected/750eb7e3-21cf-4ccc-b546-dc6385746b2a-kube-api-access-2c7rw\") pod \"maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf\" (UID: \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" Oct 30 05:18:25.757427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.757379 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf\" (UID: \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" Oct 30 05:18:25.776156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.776121 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf\" (UID: \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" Oct 30 05:18:25.790951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.790738 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c7rw\" (UniqueName: \"kubernetes.io/projected/750eb7e3-21cf-4ccc-b546-dc6385746b2a-kube-api-access-2c7rw\") pod \"maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf\" (UID: \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" Oct 30 05:18:25.914416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:25.872832 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" Oct 30 05:18:26.184492 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:18:26.184452 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:18:26.985611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:26.985494 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf"] Oct 30 05:18:26.991855 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:18:26.991826 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750eb7e3_21cf_4ccc_b546_dc6385746b2a.slice/crio-2f0c599a35aceff16ada7ae898419dfb9cdc317739c939caa71576cd7906ce6b WatchSource:0}: Error finding container 2f0c599a35aceff16ada7ae898419dfb9cdc317739c939caa71576cd7906ce6b: Status 404 returned error can't find the container with id 2f0c599a35aceff16ada7ae898419dfb9cdc317739c939caa71576cd7906ce6b Oct 30 05:18:27.203230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:27.203196 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" event={"ID":"750eb7e3-21cf-4ccc-b546-dc6385746b2a","Type":"ContainerStarted","Data":"2f0c599a35aceff16ada7ae898419dfb9cdc317739c939caa71576cd7906ce6b"} Oct 30 05:18:28.221383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:28.217032 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" event={"ID":"750eb7e3-21cf-4ccc-b546-dc6385746b2a","Type":"ContainerStarted","Data":"37e18287c32eec62814c0a663916f955ab3fdb9c9a3c8ce6374965813f8052f5"} Oct 30 05:18:28.279674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:28.269939 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" podStartSLOduration=3.269920313 podStartE2EDuration="3.269920313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:28.269787831 +0000 UTC m=+2032.797554617" watchObservedRunningTime="2026-10-30 05:18:28.269920313 +0000 UTC m=+2032.797687100" Oct 30 05:18:29.256418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:29.255947 6587 generic.go:356] "Generic (PLEG): container finished" podID="750eb7e3-21cf-4ccc-b546-dc6385746b2a" containerID="37e18287c32eec62814c0a663916f955ab3fdb9c9a3c8ce6374965813f8052f5" exitCode=0 Oct 30 05:18:29.256418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:29.256082 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" event={"ID":"750eb7e3-21cf-4ccc-b546-dc6385746b2a","Type":"ContainerDied","Data":"37e18287c32eec62814c0a663916f955ab3fdb9c9a3c8ce6374965813f8052f5"} Oct 30 05:18:30.553173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:30.553147 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" Oct 30 05:18:30.615352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:30.615318 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-openshift-ca\") pod \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\" (UID: \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\") " Oct 30 05:18:30.615578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:30.615451 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750eb7e3-21cf-4ccc-b546-dc6385746b2a-kube-api-access-2c7rw\" (UniqueName: \"kubernetes.io/projected/750eb7e3-21cf-4ccc-b546-dc6385746b2a-kube-api-access-2c7rw\") pod \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\" (UID: \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\") " Oct 30 05:18:30.615578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:30.615493 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-xks-ca\") pod \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\" (UID: \"750eb7e3-21cf-4ccc-b546-dc6385746b2a\") " Oct 30 05:18:30.619420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:30.615760 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-openshift-ca" pod "750eb7e3-21cf-4ccc-b546-dc6385746b2a" (UID: "750eb7e3-21cf-4ccc-b546-dc6385746b2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:18:30.619420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:30.616277 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:30.619420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:30.618906 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-xks-ca" pod "750eb7e3-21cf-4ccc-b546-dc6385746b2a" (UID: "750eb7e3-21cf-4ccc-b546-dc6385746b2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:18:30.624847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:30.624819 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750eb7e3-21cf-4ccc-b546-dc6385746b2a-kube-api-access-2c7rw" pod "750eb7e3-21cf-4ccc-b546-dc6385746b2a" (UID: "750eb7e3-21cf-4ccc-b546-dc6385746b2a"). InnerVolumeSpecName "kube-api-access-2c7rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:30.735384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:30.725810 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c7rw\" (UniqueName: \"kubernetes.io/projected/750eb7e3-21cf-4ccc-b546-dc6385746b2a-kube-api-access-2c7rw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:30.735384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:30.725853 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750eb7e3-21cf-4ccc-b546-dc6385746b2a-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:31.278033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:31.278004 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" Oct 30 05:18:31.278220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:31.278039 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf" event={"ID":"750eb7e3-21cf-4ccc-b546-dc6385746b2a","Type":"ContainerDied","Data":"2f0c599a35aceff16ada7ae898419dfb9cdc317739c939caa71576cd7906ce6b"} Oct 30 05:18:31.278220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:31.278083 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f0c599a35aceff16ada7ae898419dfb9cdc317739c939caa71576cd7906ce6b" Oct 30 05:18:32.045201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.045172 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" Oct 30 05:18:32.045763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.045739 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" Oct 30 05:18:32.258014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.257976 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td"] Oct 30 05:18:32.258891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.258870 6587 cpu_manager.go:414] "containerMap: removing container" podUID="750eb7e3-21cf-4ccc-b546-dc6385746b2a" containerName="revoke-keys" Oct 30 05:18:32.258891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.258890 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750eb7e3-21cf-4ccc-b546-dc6385746b2a" containerName="revoke-keys" Oct 30 05:18:32.259090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.259016 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="750eb7e3-21cf-4ccc-b546-dc6385746b2a" containerName="revoke-keys" Oct 30 05:18:32.263050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.263015 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.299130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.295695 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td"] Oct 30 05:18:32.373196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.373149 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.373470 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.373443 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.373573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.373484 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d11c4e59-293e-41b7-9f29-157a813083ad-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.373660 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.373643 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.373724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.373683 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llchj\" (UniqueName: \"kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-kube-api-access-llchj\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.373824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.373808 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.373899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.373886 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.374017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.373989 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d11c4e59-293e-41b7-9f29-157a813083ad-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.374159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.374130 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.476160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.476122 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.476379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.476180 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d11c4e59-293e-41b7-9f29-157a813083ad-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.476379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.476239 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.476379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.476289 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.476576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.476515 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.476576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.476546 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d11c4e59-293e-41b7-9f29-157a813083ad-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.476712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.476618 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.476712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.476624 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.476712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.476655 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llchj\" (UniqueName: \"kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-kube-api-access-llchj\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.476874 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.476766 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.477148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.477094 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d11c4e59-293e-41b7-9f29-157a813083ad-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.477249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.477188 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.477381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.477346 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.477896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.477873 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.479339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.479311 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.480288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.480259 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d11c4e59-293e-41b7-9f29-157a813083ad-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.500005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.499971 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.501930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.501904 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llchj\" (UniqueName: \"kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-kube-api-access-llchj\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:32.601715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:32.601622 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:33.024834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:33.024778 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td"] Oct 30 05:18:33.026479 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:18:33.025489 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11c4e59_293e_41b7_9f29_157a813083ad.slice/crio-a9af2d4b043edce8d3350c38ecbdb546c20206db04ac08be78023af276567996 WatchSource:0}: Error finding container a9af2d4b043edce8d3350c38ecbdb546c20206db04ac08be78023af276567996: Status 404 returned error can't find the container with id a9af2d4b043edce8d3350c38ecbdb546c20206db04ac08be78023af276567996 Oct 30 05:18:33.029904 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:33.029864 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:33.030027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:33.029975 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:33.030085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:33.030024 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:33.323380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:33.319650 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" event={"ID":"d11c4e59-293e-41b7-9f29-157a813083ad","Type":"ContainerStarted","Data":"80974c739aa434b1c7fcefa1218abc9c5b02e559ee6419f006cbb332ad5f9a06"} Oct 30 05:18:33.323380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:33.319721 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" event={"ID":"d11c4e59-293e-41b7-9f29-157a813083ad","Type":"ContainerStarted","Data":"a9af2d4b043edce8d3350c38ecbdb546c20206db04ac08be78023af276567996"} Oct 30 05:18:33.366381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:33.365429 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" podStartSLOduration=1.365409847 podStartE2EDuration="1.365409847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:33.363826976 +0000 UTC m=+2037.891593760" watchObservedRunningTime="2026-10-30 05:18:33.365409847 +0000 UTC m=+2037.893176636" Oct 30 05:18:33.602663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:33.602578 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:34.607408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:34.604120 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 05:18:34.607408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:34.604170 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" podUID="d11c4e59-293e-41b7-9f29-157a813083ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 05:18:35.030444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:35.029465 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:18:35.030444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:35.029547 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:18:35.602952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:35.602905 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 05:18:35.603163 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:35.602975 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" podUID="d11c4e59-293e-41b7-9f29-157a813083ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 05:18:36.053486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:36.053428 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq"] Oct 30 05:18:36.602924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:36.602886 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 05:18:36.603095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:36.602938 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" podUID="d11c4e59-293e-41b7-9f29-157a813083ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 05:18:37.602305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:37.602272 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 05:18:37.602739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:37.602323 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" podUID="d11c4e59-293e-41b7-9f29-157a813083ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 05:18:38.405686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:38.405657 6587 generic.go:356] "Generic (PLEG): container finished" podID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerID="dd4db8126824b483c87e4f80ff2c90a5acaa3b1f79d4e9729222d31794d52103" exitCode=0 Oct 30 05:18:38.405865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:38.405734 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" event={"ID":"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d","Type":"ContainerDied","Data":"dd4db8126824b483c87e4f80ff2c90a5acaa3b1f79d4e9729222d31794d52103"} Oct 30 05:18:38.406273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:38.406255 6587 scope.go:122] "RemoveContainer" containerID="dd4db8126824b483c87e4f80ff2c90a5acaa3b1f79d4e9729222d31794d52103" Oct 30 05:18:38.408893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:38.408863 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:38.409002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:38.408950 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:38.409074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:38.409003 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:38.605083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:38.605044 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 05:18:38.606937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:38.606888 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" podUID="d11c4e59-293e-41b7-9f29-157a813083ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 05:18:39.431378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:39.422806 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" event={"ID":"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d","Type":"ContainerStarted","Data":"d6b099e09fee607d4fc2f46adcafa58f1d9766c952269c0be47116710649b530"} Oct 30 05:18:39.770494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:39.769860 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:39.772420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:39.772395 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:39.775814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:39.775760 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:18:40.045441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:40.044488 6587 scope.go:122] "RemoveContainer" containerID="b8a3c16ed187731d5aa692196e6bc0e9a7cb99bd43e0b754ebad91d65417f976" Oct 30 05:18:40.106262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:40.106227 6587 scope.go:122] "RemoveContainer" containerID="ae777fc4250dba1a9cd9288c432ede9548a1d54d8c12bc939d5999b3f4fe1100" Oct 30 05:18:40.118378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:40.118340 6587 scope.go:122] "RemoveContainer" containerID="159ea297208fb3da5dc19f45a4ed487881a05560cc5a360d026a6c687e57bbf5" Oct 30 05:18:40.201814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:40.201791 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:18:40.202038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:40.202016 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 05:18:40.202123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:40.202071 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 05:18:41.054688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:41.054640 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" podUID="901c74b9-8384-451e-ab88-ad7f7069c48e" containerName="payload-processing" containerID="cri-o://a77d0357ce64ef75cf0234c2152e24604b86709dad459cb565a36ee5ff410bdc" gracePeriod=25 Oct 30 05:18:41.201809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:41.201773 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 05:18:41.201995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:41.201829 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 05:18:42.195902 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:42.195871 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf"] Oct 30 05:18:42.201904 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:42.201874 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 05:18:42.202043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:42.201931 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 05:18:42.208601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:42.208577 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f8eb4-wgqgf"] Oct 30 05:18:43.201590 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:43.201557 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 05:18:43.201991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:43.201612 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 05:18:43.588659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:43.588623 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750eb7e3-21cf-4ccc-b546-dc6385746b2a" path="/var/lib/kubelet/pods/750eb7e3-21cf-4ccc-b546-dc6385746b2a/volumes" Oct 30 05:18:44.201875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:44.201839 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 05:18:44.202353 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:44.201905 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 05:18:45.202128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:45.202090 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 05:18:45.202723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:45.202170 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 05:18:46.207208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:46.207174 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:18:46.207742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:46.207720 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:18:46.208338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:46.208312 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:18:49.047171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.047111 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq"] Oct 30 05:18:49.049147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.049094 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" containerID="cri-o://a3ba2b84b60b972d5e22c7914e643afb7acd4e91f3f8fa2a8e6dc89ac0208995" gracePeriod=30 Oct 30 05:18:49.058740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.058705 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz"] Oct 30 05:18:49.058949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.058918 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" podUID="f58ec340-b565-4048-83a2-342485b1a68d" containerName="istio-proxy" containerID="cri-o://3e471749cc6fa7236e4de7139ed84a4597e45a8392020a50c3a5c59deb660bdc" gracePeriod=30 Oct 30 05:18:49.775417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.775379 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8"] Oct 30 05:18:49.781596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.781566 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" Oct 30 05:18:49.800589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.800559 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8"] Oct 30 05:18:49.807639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.807593 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:18:49.807639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.807614 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:18:49.807834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.807692 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:18:49.807834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.807810 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:18:49.861963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.861927 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8\" (UID: \"db1b77ed-5eae-4118-b987-71f5c8ec7560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" Oct 30 05:18:49.862152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.862102 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8\" (UID: \"db1b77ed-5eae-4118-b987-71f5c8ec7560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" Oct 30 05:18:49.862374 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.862341 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9mvs\" (UniqueName: \"kubernetes.io/projected/db1b77ed-5eae-4118-b987-71f5c8ec7560-kube-api-access-k9mvs\") pod \"maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8\" (UID: \"db1b77ed-5eae-4118-b987-71f5c8ec7560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" Oct 30 05:18:49.964129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.964091 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8\" (UID: \"db1b77ed-5eae-4118-b987-71f5c8ec7560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" Oct 30 05:18:49.964311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.964241 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8\" (UID: \"db1b77ed-5eae-4118-b987-71f5c8ec7560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" Oct 30 05:18:49.964494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.964472 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9mvs\" (UniqueName: \"kubernetes.io/projected/db1b77ed-5eae-4118-b987-71f5c8ec7560-kube-api-access-k9mvs\") pod \"maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8\" (UID: \"db1b77ed-5eae-4118-b987-71f5c8ec7560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" Oct 30 05:18:49.965010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.964863 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8\" (UID: \"db1b77ed-5eae-4118-b987-71f5c8ec7560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" Oct 30 05:18:49.967173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.967143 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8\" (UID: \"db1b77ed-5eae-4118-b987-71f5c8ec7560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" Oct 30 05:18:49.998888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:49.998860 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9mvs\" (UniqueName: \"kubernetes.io/projected/db1b77ed-5eae-4118-b987-71f5c8ec7560-kube-api-access-k9mvs\") pod \"maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8\" (UID: \"db1b77ed-5eae-4118-b987-71f5c8ec7560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" Oct 30 05:18:50.099590 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:50.099556 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" Oct 30 05:18:50.431687 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:18:50.431650 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1b77ed_5eae_4118_b987_71f5c8ec7560.slice/crio-0d22ff6a67e5e6312c43cc2f3678e302e58f95bdcf94990979626c4ba54e5841 WatchSource:0}: Error finding container 0d22ff6a67e5e6312c43cc2f3678e302e58f95bdcf94990979626c4ba54e5841: Status 404 returned error can't find the container with id 0d22ff6a67e5e6312c43cc2f3678e302e58f95bdcf94990979626c4ba54e5841 Oct 30 05:18:50.433371 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:50.433330 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8"] Oct 30 05:18:50.576062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:50.576024 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" event={"ID":"db1b77ed-5eae-4118-b987-71f5c8ec7560","Type":"ContainerStarted","Data":"0d22ff6a67e5e6312c43cc2f3678e302e58f95bdcf94990979626c4ba54e5841"} Oct 30 05:18:50.877294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:50.877256 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" podUID="901c74b9-8384-451e-ab88-ad7f7069c48e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:18:51.110434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.106675 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5"] Oct 30 05:18:51.118008 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.117980 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" Oct 30 05:18:51.127962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.127833 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5"] Oct 30 05:18:51.289314 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.289278 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b6b062e-7s6l5\" (UID: \"835cb573-de48-4f9a-a0d9-a429c76dffdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" Oct 30 05:18:51.289517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.289491 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b6b062e-7s6l5\" (UID: \"835cb573-de48-4f9a-a0d9-a429c76dffdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" Oct 30 05:18:51.289579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.289566 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pchd5\" (UniqueName: \"kubernetes.io/projected/835cb573-de48-4f9a-a0d9-a429c76dffdb-kube-api-access-pchd5\") pod \"maas-api-revoke-sub-4b6b062e-7s6l5\" (UID: \"835cb573-de48-4f9a-a0d9-a429c76dffdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" Oct 30 05:18:51.391595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.391511 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b6b062e-7s6l5\" (UID: \"835cb573-de48-4f9a-a0d9-a429c76dffdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" Oct 30 05:18:51.391766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.391636 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pchd5\" (UniqueName: \"kubernetes.io/projected/835cb573-de48-4f9a-a0d9-a429c76dffdb-kube-api-access-pchd5\") pod \"maas-api-revoke-sub-4b6b062e-7s6l5\" (UID: \"835cb573-de48-4f9a-a0d9-a429c76dffdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" Oct 30 05:18:51.391880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.391859 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b6b062e-7s6l5\" (UID: \"835cb573-de48-4f9a-a0d9-a429c76dffdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" Oct 30 05:18:51.392722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.392696 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b6b062e-7s6l5\" (UID: \"835cb573-de48-4f9a-a0d9-a429c76dffdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" Oct 30 05:18:51.393979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.393951 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b6b062e-7s6l5\" (UID: \"835cb573-de48-4f9a-a0d9-a429c76dffdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" Oct 30 05:18:51.412149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.412126 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pchd5\" (UniqueName: \"kubernetes.io/projected/835cb573-de48-4f9a-a0d9-a429c76dffdb-kube-api-access-pchd5\") pod \"maas-api-revoke-sub-4b6b062e-7s6l5\" (UID: \"835cb573-de48-4f9a-a0d9-a429c76dffdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" Oct 30 05:18:51.439060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.439029 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" Oct 30 05:18:51.597651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.597619 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" event={"ID":"db1b77ed-5eae-4118-b987-71f5c8ec7560","Type":"ContainerStarted","Data":"10714accd08f6fb92d1df617073b90fba4bf79b1268191f156c4bd42fa3cf7a3"} Oct 30 05:18:51.637019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.636965 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" podStartSLOduration=2.636949166 podStartE2EDuration="2.636949166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:51.63563231 +0000 UTC m=+2056.163399097" watchObservedRunningTime="2026-10-30 05:18:51.636949166 +0000 UTC m=+2056.164715952" Oct 30 05:18:51.751493 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:18:51.751459 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835cb573_de48_4f9a_a0d9_a429c76dffdb.slice/crio-91faa2470d25991d33ab60c29a11dba744fd517d950a7c321f61b74659a9caab WatchSource:0}: Error finding container 91faa2470d25991d33ab60c29a11dba744fd517d950a7c321f61b74659a9caab: Status 404 returned error can't find the container with id 91faa2470d25991d33ab60c29a11dba744fd517d950a7c321f61b74659a9caab Oct 30 05:18:51.754735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:51.754712 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5"] Oct 30 05:18:52.604439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:52.604402 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" event={"ID":"835cb573-de48-4f9a-a0d9-a429c76dffdb","Type":"ContainerStarted","Data":"9d29f9997bf06563abd3eefe68822e3388e447a1d9d46a7783a48ea19d59b27e"} Oct 30 05:18:52.604926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:52.604456 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" event={"ID":"835cb573-de48-4f9a-a0d9-a429c76dffdb","Type":"ContainerStarted","Data":"91faa2470d25991d33ab60c29a11dba744fd517d950a7c321f61b74659a9caab"} Oct 30 05:18:52.606136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:52.606105 6587 generic.go:356] "Generic (PLEG): container finished" podID="db1b77ed-5eae-4118-b987-71f5c8ec7560" containerID="10714accd08f6fb92d1df617073b90fba4bf79b1268191f156c4bd42fa3cf7a3" exitCode=0 Oct 30 05:18:52.606278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:52.606180 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" event={"ID":"db1b77ed-5eae-4118-b987-71f5c8ec7560","Type":"ContainerDied","Data":"10714accd08f6fb92d1df617073b90fba4bf79b1268191f156c4bd42fa3cf7a3"} Oct 30 05:18:52.648928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:52.648875 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" podStartSLOduration=1.6488615800000002 podStartE2EDuration="1.64886158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:52.64519288 +0000 UTC m=+2057.172959666" watchObservedRunningTime="2026-10-30 05:18:52.64886158 +0000 UTC m=+2057.176628365" Oct 30 05:18:53.277424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.277346 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj"] Oct 30 05:18:53.281300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.281280 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.302552 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.302529 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj"] Oct 30 05:18:53.310705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.310681 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2a8f18-openshift-default-dockercfg-c6srh\"" Oct 30 05:18:53.323848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.323203 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-podinfo\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.323848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.323437 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ttt\" (UniqueName: \"kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-kube-api-access-f8ttt\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.323848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.323518 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2824ca-4137-4f91-90c8-21e086a336d9-istiod-ca-cert\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.323848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.323551 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-certs\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.323848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.323629 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-token\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.323848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.323700 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-credential-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.323848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.323820 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.324331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.323892 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-data\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.324331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.323946 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-envoy\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.425434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.425404 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.425604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.425459 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-data\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.425604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.425575 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-envoy\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.425765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.425649 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-podinfo\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.425882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.425780 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ttt\" (UniqueName: \"kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-kube-api-access-f8ttt\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.425882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.425842 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-data\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.425882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.425857 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2824ca-4137-4f91-90c8-21e086a336d9-istiod-ca-cert\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.426064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.425875 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.426064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.425893 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-certs\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.426064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.425970 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-token\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.426064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.426043 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-credential-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.426277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.426257 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-certs\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.426528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.426506 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2824ca-4137-4f91-90c8-21e086a336d9-istiod-ca-cert\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.426622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.426506 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-credential-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.428080 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.428047 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-envoy\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.428816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.428791 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-podinfo\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.446788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.446765 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-token\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.447642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.447624 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ttt\" (UniqueName: \"kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-kube-api-access-f8ttt\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.599546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.599518 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:53.617844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.617480 6587 generic.go:356] "Generic (PLEG): container finished" podID="835cb573-de48-4f9a-a0d9-a429c76dffdb" containerID="9d29f9997bf06563abd3eefe68822e3388e447a1d9d46a7783a48ea19d59b27e" exitCode=0 Oct 30 05:18:53.618234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.617840 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" event={"ID":"835cb573-de48-4f9a-a0d9-a429c76dffdb","Type":"ContainerDied","Data":"9d29f9997bf06563abd3eefe68822e3388e447a1d9d46a7783a48ea19d59b27e"} Oct 30 05:18:53.867403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.867332 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" Oct 30 05:18:53.935066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.935036 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-openshift-ca\") pod \"db1b77ed-5eae-4118-b987-71f5c8ec7560\" (UID: \"db1b77ed-5eae-4118-b987-71f5c8ec7560\") " Oct 30 05:18:53.935280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.935092 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-xks-ca\") pod \"db1b77ed-5eae-4118-b987-71f5c8ec7560\" (UID: \"db1b77ed-5eae-4118-b987-71f5c8ec7560\") " Oct 30 05:18:53.935399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.935378 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1b77ed-5eae-4118-b987-71f5c8ec7560-kube-api-access-k9mvs\" (UniqueName: \"kubernetes.io/projected/db1b77ed-5eae-4118-b987-71f5c8ec7560-kube-api-access-k9mvs\") pod \"db1b77ed-5eae-4118-b987-71f5c8ec7560\" (UID: \"db1b77ed-5eae-4118-b987-71f5c8ec7560\") " Oct 30 05:18:53.935485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.935465 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-openshift-ca" pod "db1b77ed-5eae-4118-b987-71f5c8ec7560" (UID: "db1b77ed-5eae-4118-b987-71f5c8ec7560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:18:53.936258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.936231 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:53.937338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.937311 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-xks-ca" pod "db1b77ed-5eae-4118-b987-71f5c8ec7560" (UID: "db1b77ed-5eae-4118-b987-71f5c8ec7560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:18:53.937638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.937617 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1b77ed-5eae-4118-b987-71f5c8ec7560-kube-api-access-k9mvs" pod "db1b77ed-5eae-4118-b987-71f5c8ec7560" (UID: "db1b77ed-5eae-4118-b987-71f5c8ec7560"). InnerVolumeSpecName "kube-api-access-k9mvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:53.945761 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:18:53.945137 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2824ca_4137_4f91_90c8_21e086a336d9.slice/crio-fed4d4d692cb255bf191ef87e24ac81232e9ee3b987e2599f4c81f76fab1b098 WatchSource:0}: Error finding container fed4d4d692cb255bf191ef87e24ac81232e9ee3b987e2599f4c81f76fab1b098: Status 404 returned error can't find the container with id fed4d4d692cb255bf191ef87e24ac81232e9ee3b987e2599f4c81f76fab1b098 Oct 30 05:18:53.948130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.948107 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj"] Oct 30 05:18:53.948992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.948948 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:53.949117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.949037 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:53.949117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:53.949109 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:18:54.038511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.038483 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1b77ed-5eae-4118-b987-71f5c8ec7560-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.038623 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.038515 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9mvs\" (UniqueName: \"kubernetes.io/projected/db1b77ed-5eae-4118-b987-71f5c8ec7560-kube-api-access-k9mvs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.381067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.380998 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:18:54.417810 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.417750 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.449871 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f58ec340-b565-4048-83a2-342485b1a68d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58ec340-b565-4048-83a2-342485b1a68d-istio-podinfo\") pod \"f58ec340-b565-4048-83a2-342485b1a68d\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.449913 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-data\") pod \"f58ec340-b565-4048-83a2-342485b1a68d\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.449975 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-kube-api-access-twnff\" (UniqueName: \"kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-kube-api-access-twnff\") pod \"f58ec340-b565-4048-83a2-342485b1a68d\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450003 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-envoy\") pod \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450046 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-credential-socket\") pod \"f58ec340-b565-4048-83a2-342485b1a68d\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450078 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-credential-socket\") pod \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450110 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-data\") pod \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450137 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istiod-ca-cert\") pod \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450159 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-podinfo\") pod \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450210 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-socket\") pod \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450294 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-socket\") pod \"f58ec340-b565-4048-83a2-342485b1a68d\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450413 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-kube-api-access-nwszq\" (UniqueName: \"kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-kube-api-access-nwszq\") pod \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450451 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-certs\") pod \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450486 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-envoy\") pod \"f58ec340-b565-4048-83a2-342485b1a68d\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450510 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-token\" (UniqueName: \"kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-token\") pod \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\" (UID: \"a2fb6f3a-1091-4baa-be60-aab470fa8e84\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450563 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-certs\") pod \"f58ec340-b565-4048-83a2-342485b1a68d\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450589 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-istio-token\" (UniqueName: \"kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-istio-token\") pod \"f58ec340-b565-4048-83a2-342485b1a68d\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.450631 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58ec340-b565-4048-83a2-342485b1a68d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58ec340-b565-4048-83a2-342485b1a68d-istiod-ca-cert\") pod \"f58ec340-b565-4048-83a2-342485b1a68d\" (UID: \"f58ec340-b565-4048-83a2-342485b1a68d\") " Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.452459 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-socket" pod "f58ec340-b565-4048-83a2-342485b1a68d" (UID: "f58ec340-b565-4048-83a2-342485b1a68d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.452717 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-socket" pod "a2fb6f3a-1091-4baa-be60-aab470fa8e84" (UID: "a2fb6f3a-1091-4baa-be60-aab470fa8e84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.453338 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-certs" pod "a2fb6f3a-1091-4baa-be60-aab470fa8e84" (UID: "a2fb6f3a-1091-4baa-be60-aab470fa8e84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.453424 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-credential-socket" pod "a2fb6f3a-1091-4baa-be60-aab470fa8e84" (UID: "a2fb6f3a-1091-4baa-be60-aab470fa8e84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.453460 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-data" pod "a2fb6f3a-1091-4baa-be60-aab470fa8e84" (UID: "a2fb6f3a-1091-4baa-be60-aab470fa8e84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.453680 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-certs" pod "f58ec340-b565-4048-83a2-342485b1a68d" (UID: "f58ec340-b565-4048-83a2-342485b1a68d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.454215 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-credential-socket" pod "f58ec340-b565-4048-83a2-342485b1a68d" (UID: "f58ec340-b565-4048-83a2-342485b1a68d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.454383 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-data" pod "f58ec340-b565-4048-83a2-342485b1a68d" (UID: "f58ec340-b565-4048-83a2-342485b1a68d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.455905 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58ec340-b565-4048-83a2-342485b1a68d-istiod-ca-cert" pod "f58ec340-b565-4048-83a2-342485b1a68d" (UID: "f58ec340-b565-4048-83a2-342485b1a68d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.455978 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-envoy" pod "f58ec340-b565-4048-83a2-342485b1a68d" (UID: "f58ec340-b565-4048-83a2-342485b1a68d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.456327 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istiod-ca-cert" pod "a2fb6f3a-1091-4baa-be60-aab470fa8e84" (UID: "a2fb6f3a-1091-4baa-be60-aab470fa8e84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.462598 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-envoy" pod "a2fb6f3a-1091-4baa-be60-aab470fa8e84" (UID: "a2fb6f3a-1091-4baa-be60-aab470fa8e84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.462639 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-kube-api-access-nwszq" pod "a2fb6f3a-1091-4baa-be60-aab470fa8e84" (UID: "a2fb6f3a-1091-4baa-be60-aab470fa8e84"). InnerVolumeSpecName "kube-api-access-nwszq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.462717 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-token" pod "a2fb6f3a-1091-4baa-be60-aab470fa8e84" (UID: "a2fb6f3a-1091-4baa-be60-aab470fa8e84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.463832 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-istio-token" pod "f58ec340-b565-4048-83a2-342485b1a68d" (UID: "f58ec340-b565-4048-83a2-342485b1a68d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.464388 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f58ec340-b565-4048-83a2-342485b1a68d-istio-podinfo" pod "f58ec340-b565-4048-83a2-342485b1a68d" (UID: "f58ec340-b565-4048-83a2-342485b1a68d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:18:54.465182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.464671 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-kube-api-access-twnff" pod "f58ec340-b565-4048-83a2-342485b1a68d" (UID: "f58ec340-b565-4048-83a2-342485b1a68d"). InnerVolumeSpecName "kube-api-access-twnff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:54.479633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.479591 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-podinfo" pod "a2fb6f3a-1091-4baa-be60-aab470fa8e84" (UID: "a2fb6f3a-1091-4baa-be60-aab470fa8e84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:18:54.552781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552745 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twnff\" (UniqueName: \"kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-kube-api-access-twnff\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.552781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552778 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.552781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552791 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552800 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552809 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552818 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552827 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552837 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552845 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552854 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwszq\" (UniqueName: \"kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-kube-api-access-nwszq\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552862 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fb6f3a-1091-4baa-be60-aab470fa8e84-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552871 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552878 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fb6f3a-1091-4baa-be60-aab470fa8e84-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552887 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552895 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58ec340-b565-4048-83a2-342485b1a68d-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552903 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58ec340-b565-4048-83a2-342485b1a68d-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552911 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58ec340-b565-4048-83a2-342485b1a68d-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.553053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.552918 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58ec340-b565-4048-83a2-342485b1a68d-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.654885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.654740 6587 generic.go:356] "Generic (PLEG): container finished" podID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerID="a3ba2b84b60b972d5e22c7914e643afb7acd4e91f3f8fa2a8e6dc89ac0208995" exitCode=0 Oct 30 05:18:54.654885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.654783 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" Oct 30 05:18:54.654885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.654812 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" event={"ID":"a2fb6f3a-1091-4baa-be60-aab470fa8e84","Type":"ContainerDied","Data":"a3ba2b84b60b972d5e22c7914e643afb7acd4e91f3f8fa2a8e6dc89ac0208995"} Oct 30 05:18:54.654885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.654866 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq" event={"ID":"a2fb6f3a-1091-4baa-be60-aab470fa8e84","Type":"ContainerDied","Data":"52c2bf3f2c7ddbb6c2b2df10823894cfcb38abaafd06527c81a57ee5ecd164c2"} Oct 30 05:18:54.655489 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.654901 6587 scope.go:122] "RemoveContainer" containerID="a3ba2b84b60b972d5e22c7914e643afb7acd4e91f3f8fa2a8e6dc89ac0208995" Oct 30 05:18:54.656885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.656851 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" event={"ID":"db1b77ed-5eae-4118-b987-71f5c8ec7560","Type":"ContainerDied","Data":"0d22ff6a67e5e6312c43cc2f3678e302e58f95bdcf94990979626c4ba54e5841"} Oct 30 05:18:54.657003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.656894 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d22ff6a67e5e6312c43cc2f3678e302e58f95bdcf94990979626c4ba54e5841" Oct 30 05:18:54.657003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.656858 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8" Oct 30 05:18:54.659081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.659055 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" event={"ID":"fd2824ca-4137-4f91-90c8-21e086a336d9","Type":"ContainerStarted","Data":"b78ec7da36f9f4bd659b1e82422a10945314bd5a50d6937a3a9d6704e929716c"} Oct 30 05:18:54.659321 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.659304 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" event={"ID":"fd2824ca-4137-4f91-90c8-21e086a336d9","Type":"ContainerStarted","Data":"fed4d4d692cb255bf191ef87e24ac81232e9ee3b987e2599f4c81f76fab1b098"} Oct 30 05:18:54.660817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.660765 6587 generic.go:356] "Generic (PLEG): container finished" podID="f58ec340-b565-4048-83a2-342485b1a68d" containerID="3e471749cc6fa7236e4de7139ed84a4597e45a8392020a50c3a5c59deb660bdc" exitCode=0 Oct 30 05:18:54.661642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.661252 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" Oct 30 05:18:54.663578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.663556 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" event={"ID":"f58ec340-b565-4048-83a2-342485b1a68d","Type":"ContainerDied","Data":"3e471749cc6fa7236e4de7139ed84a4597e45a8392020a50c3a5c59deb660bdc"} Oct 30 05:18:54.663673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.663603 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz" event={"ID":"f58ec340-b565-4048-83a2-342485b1a68d","Type":"ContainerDied","Data":"751e4cd00192644b3729e77f52045edb4491f04cfd03d2847db06d80438cd63b"} Oct 30 05:18:54.670758 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.670731 6587 scope.go:122] "RemoveContainer" containerID="b34cc9f6e511e072b08ead79fd4c498b10b19a27db5226970364e2e6e7d8af55" Oct 30 05:18:54.686109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.686081 6587 scope.go:122] "RemoveContainer" containerID="a3ba2b84b60b972d5e22c7914e643afb7acd4e91f3f8fa2a8e6dc89ac0208995" Oct 30 05:18:54.686515 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:18:54.686493 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ba2b84b60b972d5e22c7914e643afb7acd4e91f3f8fa2a8e6dc89ac0208995\": container with ID starting with a3ba2b84b60b972d5e22c7914e643afb7acd4e91f3f8fa2a8e6dc89ac0208995 not found: ID does not exist" containerID="a3ba2b84b60b972d5e22c7914e643afb7acd4e91f3f8fa2a8e6dc89ac0208995" Oct 30 05:18:54.686598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.686523 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ba2b84b60b972d5e22c7914e643afb7acd4e91f3f8fa2a8e6dc89ac0208995"} err="failed to get container status \"a3ba2b84b60b972d5e22c7914e643afb7acd4e91f3f8fa2a8e6dc89ac0208995\": rpc error: code = NotFound desc = could not find container \"a3ba2b84b60b972d5e22c7914e643afb7acd4e91f3f8fa2a8e6dc89ac0208995\": container with ID starting with a3ba2b84b60b972d5e22c7914e643afb7acd4e91f3f8fa2a8e6dc89ac0208995 not found: ID does not exist" Oct 30 05:18:54.686598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.686544 6587 scope.go:122] "RemoveContainer" containerID="b34cc9f6e511e072b08ead79fd4c498b10b19a27db5226970364e2e6e7d8af55" Oct 30 05:18:54.686846 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:18:54.686819 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34cc9f6e511e072b08ead79fd4c498b10b19a27db5226970364e2e6e7d8af55\": container with ID starting with b34cc9f6e511e072b08ead79fd4c498b10b19a27db5226970364e2e6e7d8af55 not found: ID does not exist" containerID="b34cc9f6e511e072b08ead79fd4c498b10b19a27db5226970364e2e6e7d8af55" Oct 30 05:18:54.686938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.686850 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34cc9f6e511e072b08ead79fd4c498b10b19a27db5226970364e2e6e7d8af55"} err="failed to get container status \"b34cc9f6e511e072b08ead79fd4c498b10b19a27db5226970364e2e6e7d8af55\": rpc error: code = NotFound desc = could not find container \"b34cc9f6e511e072b08ead79fd4c498b10b19a27db5226970364e2e6e7d8af55\": container with ID starting with b34cc9f6e511e072b08ead79fd4c498b10b19a27db5226970364e2e6e7d8af55 not found: ID does not exist" Oct 30 05:18:54.686938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.686868 6587 scope.go:122] "RemoveContainer" containerID="3e471749cc6fa7236e4de7139ed84a4597e45a8392020a50c3a5c59deb660bdc" Oct 30 05:18:54.699845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.699824 6587 scope.go:122] "RemoveContainer" containerID="3e471749cc6fa7236e4de7139ed84a4597e45a8392020a50c3a5c59deb660bdc" Oct 30 05:18:54.704744 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:18:54.704711 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e471749cc6fa7236e4de7139ed84a4597e45a8392020a50c3a5c59deb660bdc\": container with ID starting with 3e471749cc6fa7236e4de7139ed84a4597e45a8392020a50c3a5c59deb660bdc not found: ID does not exist" containerID="3e471749cc6fa7236e4de7139ed84a4597e45a8392020a50c3a5c59deb660bdc" Oct 30 05:18:54.704881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.704755 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e471749cc6fa7236e4de7139ed84a4597e45a8392020a50c3a5c59deb660bdc"} err="failed to get container status \"3e471749cc6fa7236e4de7139ed84a4597e45a8392020a50c3a5c59deb660bdc\": rpc error: code = NotFound desc = could not find container \"3e471749cc6fa7236e4de7139ed84a4597e45a8392020a50c3a5c59deb660bdc\": container with ID starting with 3e471749cc6fa7236e4de7139ed84a4597e45a8392020a50c3a5c59deb660bdc not found: ID does not exist" Oct 30 05:18:54.719382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.718601 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" podStartSLOduration=1.71858246 podStartE2EDuration="1.71858246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:54.715442715 +0000 UTC m=+2059.243209502" watchObservedRunningTime="2026-10-30 05:18:54.71858246 +0000 UTC m=+2059.246349247" Oct 30 05:18:54.808971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.808907 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq"] Oct 30 05:18:54.814712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.814688 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq"] Oct 30 05:18:54.870584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.870526 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz"] Oct 30 05:18:54.875755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.875625 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz"] Oct 30 05:18:54.902677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.902646 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" Oct 30 05:18:54.959604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.959524 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-xks-ca\") pod \"835cb573-de48-4f9a-a0d9-a429c76dffdb\" (UID: \"835cb573-de48-4f9a-a0d9-a429c76dffdb\") " Oct 30 05:18:54.959770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.959650 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-openshift-ca\") pod \"835cb573-de48-4f9a-a0d9-a429c76dffdb\" (UID: \"835cb573-de48-4f9a-a0d9-a429c76dffdb\") " Oct 30 05:18:54.959942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.959920 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835cb573-de48-4f9a-a0d9-a429c76dffdb-kube-api-access-pchd5\" (UniqueName: \"kubernetes.io/projected/835cb573-de48-4f9a-a0d9-a429c76dffdb-kube-api-access-pchd5\") pod \"835cb573-de48-4f9a-a0d9-a429c76dffdb\" (UID: \"835cb573-de48-4f9a-a0d9-a429c76dffdb\") " Oct 30 05:18:54.960032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.960001 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-openshift-ca" pod "835cb573-de48-4f9a-a0d9-a429c76dffdb" (UID: "835cb573-de48-4f9a-a0d9-a429c76dffdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:18:54.960524 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.960502 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:54.961884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.961858 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835cb573-de48-4f9a-a0d9-a429c76dffdb-kube-api-access-pchd5" pod "835cb573-de48-4f9a-a0d9-a429c76dffdb" (UID: "835cb573-de48-4f9a-a0d9-a429c76dffdb"). InnerVolumeSpecName "kube-api-access-pchd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:54.961987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:54.961907 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-xks-ca" pod "835cb573-de48-4f9a-a0d9-a429c76dffdb" (UID: "835cb573-de48-4f9a-a0d9-a429c76dffdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:18:55.071998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:55.070285 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pchd5\" (UniqueName: \"kubernetes.io/projected/835cb573-de48-4f9a-a0d9-a429c76dffdb-kube-api-access-pchd5\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:55.071998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:55.070325 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/835cb573-de48-4f9a-a0d9-a429c76dffdb-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:18:55.587310 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:55.587275 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" path="/var/lib/kubelet/pods/a2fb6f3a-1091-4baa-be60-aab470fa8e84/volumes" Oct 30 05:18:55.588004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:55.587978 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58ec340-b565-4048-83a2-342485b1a68d" path="/var/lib/kubelet/pods/f58ec340-b565-4048-83a2-342485b1a68d/volumes" Oct 30 05:18:55.600101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:55.600078 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:18:55.601960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:55.601937 6587 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 05:18:55.602039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:55.601981 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" podUID="fd2824ca-4137-4f91-90c8-21e086a336d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 05:18:55.677384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:55.676609 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" Oct 30 05:18:55.677384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:55.676617 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5" event={"ID":"835cb573-de48-4f9a-a0d9-a429c76dffdb","Type":"ContainerDied","Data":"91faa2470d25991d33ab60c29a11dba744fd517d950a7c321f61b74659a9caab"} Oct 30 05:18:55.677384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:55.676684 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91faa2470d25991d33ab60c29a11dba744fd517d950a7c321f61b74659a9caab" Oct 30 05:18:55.987690 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:18:55.987652 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:18:56.600662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:56.600626 6587 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 05:18:56.600829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:56.600677 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" podUID="fd2824ca-4137-4f91-90c8-21e086a336d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 05:18:57.600760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:57.600724 6587 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 05:18:57.601161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:57.600776 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" podUID="fd2824ca-4137-4f91-90c8-21e086a336d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 05:18:58.600559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:58.600526 6587 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 05:18:58.600741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:58.600577 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" podUID="fd2824ca-4137-4f91-90c8-21e086a336d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 05:18:59.600440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:59.600399 6587 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 05:18:59.600888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:18:59.600462 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" podUID="fd2824ca-4137-4f91-90c8-21e086a336d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 05:19:00.117279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:00.117228 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf"] Oct 30 05:19:00.729878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:00.729845 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:19:00.731664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:00.731641 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:19:00.751283 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:00.733090 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:19:00.971416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:00.970399 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" podUID="901c74b9-8384-451e-ab88-ad7f7069c48e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:19:01.352446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.352336 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" Oct 30 05:19:01.516013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.515973 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901c74b9-8384-451e-ab88-ad7f7069c48e-kube-api-access-6gtnr\" (UniqueName: \"kubernetes.io/projected/901c74b9-8384-451e-ab88-ad7f7069c48e-kube-api-access-6gtnr\") pod \"901c74b9-8384-451e-ab88-ad7f7069c48e\" (UID: \"901c74b9-8384-451e-ab88-ad7f7069c48e\") " Oct 30 05:19:01.516221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.516191 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901c74b9-8384-451e-ab88-ad7f7069c48e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901c74b9-8384-451e-ab88-ad7f7069c48e-plugins-config-volume\") pod \"901c74b9-8384-451e-ab88-ad7f7069c48e\" (UID: \"901c74b9-8384-451e-ab88-ad7f7069c48e\") " Oct 30 05:19:01.520497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.517795 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901c74b9-8384-451e-ab88-ad7f7069c48e-plugins-config-volume" pod "901c74b9-8384-451e-ab88-ad7f7069c48e" (UID: "901c74b9-8384-451e-ab88-ad7f7069c48e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:19:01.532861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.532825 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901c74b9-8384-451e-ab88-ad7f7069c48e-kube-api-access-6gtnr" pod "901c74b9-8384-451e-ab88-ad7f7069c48e" (UID: "901c74b9-8384-451e-ab88-ad7f7069c48e"). InnerVolumeSpecName "kube-api-access-6gtnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:01.619132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.619099 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gtnr\" (UniqueName: \"kubernetes.io/projected/901c74b9-8384-451e-ab88-ad7f7069c48e-kube-api-access-6gtnr\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:01.619266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.619135 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901c74b9-8384-451e-ab88-ad7f7069c48e-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:01.924443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.924332 6587 generic.go:356] "Generic (PLEG): container finished" podID="901c74b9-8384-451e-ab88-ad7f7069c48e" containerID="a77d0357ce64ef75cf0234c2152e24604b86709dad459cb565a36ee5ff410bdc" exitCode=0 Oct 30 05:19:01.924443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.924427 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" Oct 30 05:19:01.924942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.924428 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" event={"ID":"901c74b9-8384-451e-ab88-ad7f7069c48e","Type":"ContainerDied","Data":"a77d0357ce64ef75cf0234c2152e24604b86709dad459cb565a36ee5ff410bdc"} Oct 30 05:19:01.924942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.924561 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq" event={"ID":"901c74b9-8384-451e-ab88-ad7f7069c48e","Type":"ContainerDied","Data":"8d5db8d34726ca2b19a27927e8240fdc01d2ccad3bafd2e09328d9d283881b6b"} Oct 30 05:19:01.924942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.924592 6587 scope.go:122] "RemoveContainer" containerID="a77d0357ce64ef75cf0234c2152e24604b86709dad459cb565a36ee5ff410bdc" Oct 30 05:19:01.935713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.935696 6587 scope.go:122] "RemoveContainer" containerID="a77d0357ce64ef75cf0234c2152e24604b86709dad459cb565a36ee5ff410bdc" Oct 30 05:19:01.936008 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:19:01.935981 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77d0357ce64ef75cf0234c2152e24604b86709dad459cb565a36ee5ff410bdc\": container with ID starting with a77d0357ce64ef75cf0234c2152e24604b86709dad459cb565a36ee5ff410bdc not found: ID does not exist" containerID="a77d0357ce64ef75cf0234c2152e24604b86709dad459cb565a36ee5ff410bdc" Oct 30 05:19:01.936073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.936021 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77d0357ce64ef75cf0234c2152e24604b86709dad459cb565a36ee5ff410bdc"} err="failed to get container status \"a77d0357ce64ef75cf0234c2152e24604b86709dad459cb565a36ee5ff410bdc\": rpc error: code = NotFound desc = could not find container \"a77d0357ce64ef75cf0234c2152e24604b86709dad459cb565a36ee5ff410bdc\": container with ID starting with a77d0357ce64ef75cf0234c2152e24604b86709dad459cb565a36ee5ff410bdc not found: ID does not exist" Oct 30 05:19:01.983814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.983788 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq"] Oct 30 05:19:01.996766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:01.996743 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq"] Oct 30 05:19:03.584631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:03.584599 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901c74b9-8384-451e-ab88-ad7f7069c48e" path="/var/lib/kubelet/pods/901c74b9-8384-451e-ab88-ad7f7069c48e/volumes" Oct 30 05:19:04.267179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.267139 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk"] Oct 30 05:19:04.268028 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268005 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" Oct 30 05:19:04.268166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268029 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" Oct 30 05:19:04.268166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268051 6587 cpu_manager.go:414] "containerMap: removing container" podUID="db1b77ed-5eae-4118-b987-71f5c8ec7560" containerName="revoke-keys" Oct 30 05:19:04.268166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268058 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1b77ed-5eae-4118-b987-71f5c8ec7560" containerName="revoke-keys" Oct 30 05:19:04.268166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268074 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" Oct 30 05:19:04.268166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268082 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" Oct 30 05:19:04.268166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268106 6587 cpu_manager.go:414] "containerMap: removing container" podUID="835cb573-de48-4f9a-a0d9-a429c76dffdb" containerName="revoke-keys" Oct 30 05:19:04.268166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268113 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835cb573-de48-4f9a-a0d9-a429c76dffdb" containerName="revoke-keys" Oct 30 05:19:04.268166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268128 6587 cpu_manager.go:414] "containerMap: removing container" podUID="901c74b9-8384-451e-ab88-ad7f7069c48e" containerName="payload-processing" Oct 30 05:19:04.268166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268135 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901c74b9-8384-451e-ab88-ad7f7069c48e" containerName="payload-processing" Oct 30 05:19:04.268657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268202 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f58ec340-b565-4048-83a2-342485b1a68d" containerName="istio-proxy" Oct 30 05:19:04.268657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268211 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58ec340-b565-4048-83a2-342485b1a68d" containerName="istio-proxy" Oct 30 05:19:04.268657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268425 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" Oct 30 05:19:04.268657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268441 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1b77ed-5eae-4118-b987-71f5c8ec7560" containerName="revoke-keys" Oct 30 05:19:04.268657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268455 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fb6f3a-1091-4baa-be60-aab470fa8e84" containerName="istio-proxy" Oct 30 05:19:04.268657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268470 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="835cb573-de48-4f9a-a0d9-a429c76dffdb" containerName="revoke-keys" Oct 30 05:19:04.268657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268483 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="901c74b9-8384-451e-ab88-ad7f7069c48e" containerName="payload-processing" Oct 30 05:19:04.268657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.268503 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58ec340-b565-4048-83a2-342485b1a68d" containerName="istio-proxy" Oct 30 05:19:04.272053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.272030 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" Oct 30 05:19:04.292203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.292176 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk"] Oct 30 05:19:04.299414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.299334 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:19:04.299414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.299403 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:19:04.302479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.302458 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:19:04.302621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.302481 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:19:04.363947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.363898 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6eb4b85-b8ttk\" (UID: \"4aa0d385-5478-4563-99c2-a027e159bf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" Oct 30 05:19:04.364122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.364032 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6eb4b85-b8ttk\" (UID: \"4aa0d385-5478-4563-99c2-a027e159bf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" Oct 30 05:19:04.364167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.364114 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52dx2\" (UniqueName: \"kubernetes.io/projected/4aa0d385-5478-4563-99c2-a027e159bf06-kube-api-access-52dx2\") pod \"maas-api-revoke-sub-d6eb4b85-b8ttk\" (UID: \"4aa0d385-5478-4563-99c2-a027e159bf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" Oct 30 05:19:04.465386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.465337 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6eb4b85-b8ttk\" (UID: \"4aa0d385-5478-4563-99c2-a027e159bf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" Oct 30 05:19:04.465542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.465430 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6eb4b85-b8ttk\" (UID: \"4aa0d385-5478-4563-99c2-a027e159bf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" Oct 30 05:19:04.465542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.465477 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52dx2\" (UniqueName: \"kubernetes.io/projected/4aa0d385-5478-4563-99c2-a027e159bf06-kube-api-access-52dx2\") pod \"maas-api-revoke-sub-d6eb4b85-b8ttk\" (UID: \"4aa0d385-5478-4563-99c2-a027e159bf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" Oct 30 05:19:04.465994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.465965 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6eb4b85-b8ttk\" (UID: \"4aa0d385-5478-4563-99c2-a027e159bf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" Oct 30 05:19:04.467894 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.467869 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6eb4b85-b8ttk\" (UID: \"4aa0d385-5478-4563-99c2-a027e159bf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" Oct 30 05:19:04.487214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.487187 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52dx2\" (UniqueName: \"kubernetes.io/projected/4aa0d385-5478-4563-99c2-a027e159bf06-kube-api-access-52dx2\") pod \"maas-api-revoke-sub-d6eb4b85-b8ttk\" (UID: \"4aa0d385-5478-4563-99c2-a027e159bf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" Oct 30 05:19:04.587118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.587091 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" Oct 30 05:19:04.893780 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:19:04.893750 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa0d385_5478_4563_99c2_a027e159bf06.slice/crio-f814f4ca7859314d07ef55502da23436706262a6c9106c4f17d970de166d356c WatchSource:0}: Error finding container f814f4ca7859314d07ef55502da23436706262a6c9106c4f17d970de166d356c: Status 404 returned error can't find the container with id f814f4ca7859314d07ef55502da23436706262a6c9106c4f17d970de166d356c Oct 30 05:19:04.894098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.894073 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk"] Oct 30 05:19:04.954106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:04.954075 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" event={"ID":"4aa0d385-5478-4563-99c2-a027e159bf06","Type":"ContainerStarted","Data":"f814f4ca7859314d07ef55502da23436706262a6c9106c4f17d970de166d356c"} Oct 30 05:19:05.117690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.117653 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" podUID="6dc48b09-eba2-4237-be66-16087f00866e" containerName="payload-processing" containerID="cri-o://b93e01627e545b554ce94ce2825de1078d9c5ea22d1e77ed00120dd3fef55934" gracePeriod=25 Oct 30 05:19:05.536212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.536178 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" Oct 30 05:19:05.686726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.686695 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc48b09-eba2-4237-be66-16087f00866e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc48b09-eba2-4237-be66-16087f00866e-plugins-config-volume\") pod \"6dc48b09-eba2-4237-be66-16087f00866e\" (UID: \"6dc48b09-eba2-4237-be66-16087f00866e\") " Oct 30 05:19:05.687102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.686763 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc48b09-eba2-4237-be66-16087f00866e-kube-api-access-q2dth\" (UniqueName: \"kubernetes.io/projected/6dc48b09-eba2-4237-be66-16087f00866e-kube-api-access-q2dth\") pod \"6dc48b09-eba2-4237-be66-16087f00866e\" (UID: \"6dc48b09-eba2-4237-be66-16087f00866e\") " Oct 30 05:19:05.687188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.687137 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc48b09-eba2-4237-be66-16087f00866e-plugins-config-volume" pod "6dc48b09-eba2-4237-be66-16087f00866e" (UID: "6dc48b09-eba2-4237-be66-16087f00866e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:19:05.687534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.687511 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc48b09-eba2-4237-be66-16087f00866e-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:05.689042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.689019 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc48b09-eba2-4237-be66-16087f00866e-kube-api-access-q2dth" pod "6dc48b09-eba2-4237-be66-16087f00866e" (UID: "6dc48b09-eba2-4237-be66-16087f00866e"). InnerVolumeSpecName "kube-api-access-q2dth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:05.789201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.789114 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2dth\" (UniqueName: \"kubernetes.io/projected/6dc48b09-eba2-4237-be66-16087f00866e-kube-api-access-q2dth\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:05.967928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.967895 6587 generic.go:356] "Generic (PLEG): container finished" podID="6dc48b09-eba2-4237-be66-16087f00866e" containerID="b93e01627e545b554ce94ce2825de1078d9c5ea22d1e77ed00120dd3fef55934" exitCode=0 Oct 30 05:19:05.968058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.967942 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" Oct 30 05:19:05.968058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.967979 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" event={"ID":"6dc48b09-eba2-4237-be66-16087f00866e","Type":"ContainerDied","Data":"b93e01627e545b554ce94ce2825de1078d9c5ea22d1e77ed00120dd3fef55934"} Oct 30 05:19:05.968058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.968048 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf" event={"ID":"6dc48b09-eba2-4237-be66-16087f00866e","Type":"ContainerDied","Data":"326f0d37ece95b88def8616c2fdfd376339a419c40a23cd36a6747661a0a0717"} Oct 30 05:19:05.968231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.968079 6587 scope.go:122] "RemoveContainer" containerID="b93e01627e545b554ce94ce2825de1078d9c5ea22d1e77ed00120dd3fef55934" Oct 30 05:19:05.971662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.971637 6587 generic.go:356] "Generic (PLEG): container finished" podID="4aa0d385-5478-4563-99c2-a027e159bf06" containerID="c6b96ea85922e1f71d04281b917a4f3da2188e2debcc46689b7c94af98e5e7c8" exitCode=0 Oct 30 05:19:05.971832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:05.971799 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" event={"ID":"4aa0d385-5478-4563-99c2-a027e159bf06","Type":"ContainerDied","Data":"c6b96ea85922e1f71d04281b917a4f3da2188e2debcc46689b7c94af98e5e7c8"} Oct 30 05:19:06.009414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:06.009309 6587 scope.go:122] "RemoveContainer" containerID="b93e01627e545b554ce94ce2825de1078d9c5ea22d1e77ed00120dd3fef55934" Oct 30 05:19:06.009874 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:19:06.009823 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93e01627e545b554ce94ce2825de1078d9c5ea22d1e77ed00120dd3fef55934\": container with ID starting with b93e01627e545b554ce94ce2825de1078d9c5ea22d1e77ed00120dd3fef55934 not found: ID does not exist" containerID="b93e01627e545b554ce94ce2825de1078d9c5ea22d1e77ed00120dd3fef55934" Oct 30 05:19:06.009990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:06.009886 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93e01627e545b554ce94ce2825de1078d9c5ea22d1e77ed00120dd3fef55934"} err="failed to get container status \"b93e01627e545b554ce94ce2825de1078d9c5ea22d1e77ed00120dd3fef55934\": rpc error: code = NotFound desc = could not find container \"b93e01627e545b554ce94ce2825de1078d9c5ea22d1e77ed00120dd3fef55934\": container with ID starting with b93e01627e545b554ce94ce2825de1078d9c5ea22d1e77ed00120dd3fef55934 not found: ID does not exist" Oct 30 05:19:06.069311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:06.069284 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf"] Oct 30 05:19:06.073188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:06.073161 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf"] Oct 30 05:19:07.265431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.265241 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" Oct 30 05:19:07.412777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.412739 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-openshift-ca\") pod \"4aa0d385-5478-4563-99c2-a027e159bf06\" (UID: \"4aa0d385-5478-4563-99c2-a027e159bf06\") " Oct 30 05:19:07.412971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.412952 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-xks-ca\") pod \"4aa0d385-5478-4563-99c2-a027e159bf06\" (UID: \"4aa0d385-5478-4563-99c2-a027e159bf06\") " Oct 30 05:19:07.413108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.413083 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-openshift-ca" pod "4aa0d385-5478-4563-99c2-a027e159bf06" (UID: "4aa0d385-5478-4563-99c2-a027e159bf06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:19:07.413179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.413161 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa0d385-5478-4563-99c2-a027e159bf06-kube-api-access-52dx2\" (UniqueName: \"kubernetes.io/projected/4aa0d385-5478-4563-99c2-a027e159bf06-kube-api-access-52dx2\") pod \"4aa0d385-5478-4563-99c2-a027e159bf06\" (UID: \"4aa0d385-5478-4563-99c2-a027e159bf06\") " Oct 30 05:19:07.413824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.413801 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:07.415046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.415020 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-xks-ca" pod "4aa0d385-5478-4563-99c2-a027e159bf06" (UID: "4aa0d385-5478-4563-99c2-a027e159bf06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:19:07.415208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.415188 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa0d385-5478-4563-99c2-a027e159bf06-kube-api-access-52dx2" pod "4aa0d385-5478-4563-99c2-a027e159bf06" (UID: "4aa0d385-5478-4563-99c2-a027e159bf06"). InnerVolumeSpecName "kube-api-access-52dx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:07.525106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.525008 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52dx2\" (UniqueName: \"kubernetes.io/projected/4aa0d385-5478-4563-99c2-a027e159bf06-kube-api-access-52dx2\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:07.525106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.525042 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa0d385-5478-4563-99c2-a027e159bf06-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:07.588295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.588263 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc48b09-eba2-4237-be66-16087f00866e" path="/var/lib/kubelet/pods/6dc48b09-eba2-4237-be66-16087f00866e/volumes" Oct 30 05:19:07.998001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.997966 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" Oct 30 05:19:07.998001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.997993 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk" event={"ID":"4aa0d385-5478-4563-99c2-a027e159bf06","Type":"ContainerDied","Data":"f814f4ca7859314d07ef55502da23436706262a6c9106c4f17d970de166d356c"} Oct 30 05:19:07.998225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:07.998046 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f814f4ca7859314d07ef55502da23436706262a6c9106c4f17d970de166d356c" Oct 30 05:19:22.345935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.345847 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl"] Oct 30 05:19:22.346926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.346167 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" podUID="4bbc1851-59c4-451d-9345-aff6d78e5a2d" containerName="discovery" containerID="cri-o://b6a9ec40fe70f95a79109d61ddac1df214d7ddc2e0f999c4f6213d8d6f3210df" gracePeriod=30 Oct 30 05:19:22.367220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.364904 6587 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:8080/ready\": read tcp 10.134.0.2:36872->10.134.0.115:8080: read: connection reset by peer" start-of-body= Oct 30 05:19:22.367220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.364956 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" podUID="4bbc1851-59c4-451d-9345-aff6d78e5a2d" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.115:8080/ready\": read tcp 10.134.0.2:36872->10.134.0.115:8080: read: connection reset by peer" Oct 30 05:19:22.665140 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.665112 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:19:22.703285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.703253 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-dns-cert\") pod \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " Oct 30 05:19:22.703483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.703320 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-kube-api-access-wgchb\" (UniqueName: \"kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-kube-api-access-wgchb\") pod \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " Oct 30 05:19:22.703483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.703477 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-cacerts\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-cacerts\") pod \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " Oct 30 05:19:22.703586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.703534 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbc1851-59c4-451d-9345-aff6d78e5a2d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbc1851-59c4-451d-9345-aff6d78e5a2d-local-certs\") pod \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " Oct 30 05:19:22.703806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.703591 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-kubeconfig\") pod \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " Oct 30 05:19:22.703806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.703659 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-ca-configmap\") pod \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " Oct 30 05:19:22.703806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.703788 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-token\" (UniqueName: \"kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-token\") pod \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\" (UID: \"4bbc1851-59c4-451d-9345-aff6d78e5a2d\") " Oct 30 05:19:22.704549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.704343 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-ca-configmap" pod "4bbc1851-59c4-451d-9345-aff6d78e5a2d" (UID: "4bbc1851-59c4-451d-9345-aff6d78e5a2d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:19:22.704817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.704788 6587 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-ca-configmap\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:22.706298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.706269 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-dns-cert" pod "4bbc1851-59c4-451d-9345-aff6d78e5a2d" (UID: "4bbc1851-59c4-451d-9345-aff6d78e5a2d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:19:22.706298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.706283 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-kubeconfig" pod "4bbc1851-59c4-451d-9345-aff6d78e5a2d" (UID: "4bbc1851-59c4-451d-9345-aff6d78e5a2d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:19:22.706817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.706763 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-cacerts" pod "4bbc1851-59c4-451d-9345-aff6d78e5a2d" (UID: "4bbc1851-59c4-451d-9345-aff6d78e5a2d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:19:22.707466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.707441 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-kube-api-access-wgchb" pod "4bbc1851-59c4-451d-9345-aff6d78e5a2d" (UID: "4bbc1851-59c4-451d-9345-aff6d78e5a2d"). InnerVolumeSpecName "kube-api-access-wgchb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:22.708656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.708628 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbc1851-59c4-451d-9345-aff6d78e5a2d-local-certs" pod "4bbc1851-59c4-451d-9345-aff6d78e5a2d" (UID: "4bbc1851-59c4-451d-9345-aff6d78e5a2d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:22.708816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.708650 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-token" pod "4bbc1851-59c4-451d-9345-aff6d78e5a2d" (UID: "4bbc1851-59c4-451d-9345-aff6d78e5a2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:22.807225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.807188 6587 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-cacerts\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:22.807225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.807225 6587 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbc1851-59c4-451d-9345-aff6d78e5a2d-local-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:22.807615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.807242 6587 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-kubeconfig\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:22.807615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.807257 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:22.807615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.807272 6587 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4bbc1851-59c4-451d-9345-aff6d78e5a2d-istio-csr-dns-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:22.807615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:22.807286 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgchb\" (UniqueName: \"kubernetes.io/projected/4bbc1851-59c4-451d-9345-aff6d78e5a2d-kube-api-access-wgchb\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:23.149250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:23.149202 6587 generic.go:356] "Generic (PLEG): container finished" podID="4bbc1851-59c4-451d-9345-aff6d78e5a2d" containerID="b6a9ec40fe70f95a79109d61ddac1df214d7ddc2e0f999c4f6213d8d6f3210df" exitCode=0 Oct 30 05:19:23.149250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:23.149261 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" Oct 30 05:19:23.149579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:23.149310 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" event={"ID":"4bbc1851-59c4-451d-9345-aff6d78e5a2d","Type":"ContainerDied","Data":"b6a9ec40fe70f95a79109d61ddac1df214d7ddc2e0f999c4f6213d8d6f3210df"} Oct 30 05:19:23.149579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:23.149397 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl" event={"ID":"4bbc1851-59c4-451d-9345-aff6d78e5a2d","Type":"ContainerDied","Data":"9aaf80eb68bc7c82801bf1eedcb000643f33e99dfe4088d97625bd52f0901e91"} Oct 30 05:19:23.149579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:23.149429 6587 scope.go:122] "RemoveContainer" containerID="b6a9ec40fe70f95a79109d61ddac1df214d7ddc2e0f999c4f6213d8d6f3210df" Oct 30 05:19:23.164412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:23.164385 6587 scope.go:122] "RemoveContainer" containerID="b6a9ec40fe70f95a79109d61ddac1df214d7ddc2e0f999c4f6213d8d6f3210df" Oct 30 05:19:23.164727 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:19:23.164702 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a9ec40fe70f95a79109d61ddac1df214d7ddc2e0f999c4f6213d8d6f3210df\": container with ID starting with b6a9ec40fe70f95a79109d61ddac1df214d7ddc2e0f999c4f6213d8d6f3210df not found: ID does not exist" containerID="b6a9ec40fe70f95a79109d61ddac1df214d7ddc2e0f999c4f6213d8d6f3210df" Oct 30 05:19:23.164809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:23.164740 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a9ec40fe70f95a79109d61ddac1df214d7ddc2e0f999c4f6213d8d6f3210df"} err="failed to get container status \"b6a9ec40fe70f95a79109d61ddac1df214d7ddc2e0f999c4f6213d8d6f3210df\": rpc error: code = NotFound desc = could not find container \"b6a9ec40fe70f95a79109d61ddac1df214d7ddc2e0f999c4f6213d8d6f3210df\": container with ID starting with b6a9ec40fe70f95a79109d61ddac1df214d7ddc2e0f999c4f6213d8d6f3210df not found: ID does not exist" Oct 30 05:19:23.207039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:23.207009 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl"] Oct 30 05:19:23.221198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:23.221167 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl"] Oct 30 05:19:23.585060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:23.585030 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bbc1851-59c4-451d-9345-aff6d78e5a2d" path="/var/lib/kubelet/pods/4bbc1851-59c4-451d-9345-aff6d78e5a2d/volumes" Oct 30 05:19:25.990892 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:19:25.990851 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:19:26.207659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:26.207626 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj"] Oct 30 05:19:26.207897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:26.207875 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" podUID="a5a3df96-4e74-44bd-a810-f65702e88fd5" containerName="istio-proxy" containerID="cri-o://64e21138c69a7d847d684cfe232ff61f1ce1b23a9949a9f23dc550ad8893f27e" gracePeriod=30 Oct 30 05:19:31.506744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.506707 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:19:31.604746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.604710 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-credential-socket\") pod \"a5a3df96-4e74-44bd-a810-f65702e88fd5\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " Oct 30 05:19:31.604920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.604763 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a3df96-4e74-44bd-a810-f65702e88fd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a3df96-4e74-44bd-a810-f65702e88fd5-istiod-ca-cert\") pod \"a5a3df96-4e74-44bd-a810-f65702e88fd5\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " Oct 30 05:19:31.604920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.604831 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-data\") pod \"a5a3df96-4e74-44bd-a810-f65702e88fd5\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " Oct 30 05:19:31.604920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.604873 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-envoy\") pod \"a5a3df96-4e74-44bd-a810-f65702e88fd5\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " Oct 30 05:19:31.604920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.604911 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-token\" (UniqueName: \"kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-token\") pod \"a5a3df96-4e74-44bd-a810-f65702e88fd5\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " Oct 30 05:19:31.605069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.604980 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-certs\") pod \"a5a3df96-4e74-44bd-a810-f65702e88fd5\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " Oct 30 05:19:31.605069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.605057 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-podinfo\") pod \"a5a3df96-4e74-44bd-a810-f65702e88fd5\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " Oct 30 05:19:31.605159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.605105 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-socket\") pod \"a5a3df96-4e74-44bd-a810-f65702e88fd5\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " Oct 30 05:19:31.605201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.605161 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-kube-api-access-8thkk\" (UniqueName: \"kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-kube-api-access-8thkk\") pod \"a5a3df96-4e74-44bd-a810-f65702e88fd5\" (UID: \"a5a3df96-4e74-44bd-a810-f65702e88fd5\") " Oct 30 05:19:31.605815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.605611 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-certs" pod "a5a3df96-4e74-44bd-a810-f65702e88fd5" (UID: "a5a3df96-4e74-44bd-a810-f65702e88fd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:31.605815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.605690 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a3df96-4e74-44bd-a810-f65702e88fd5-istiod-ca-cert" pod "a5a3df96-4e74-44bd-a810-f65702e88fd5" (UID: "a5a3df96-4e74-44bd-a810-f65702e88fd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:19:31.606025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.605884 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-data" pod "a5a3df96-4e74-44bd-a810-f65702e88fd5" (UID: "a5a3df96-4e74-44bd-a810-f65702e88fd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:31.606025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.605919 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-socket" pod "a5a3df96-4e74-44bd-a810-f65702e88fd5" (UID: "a5a3df96-4e74-44bd-a810-f65702e88fd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:31.606025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.606010 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-credential-socket" pod "a5a3df96-4e74-44bd-a810-f65702e88fd5" (UID: "a5a3df96-4e74-44bd-a810-f65702e88fd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:31.606288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.606266 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:31.606388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.606291 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a3df96-4e74-44bd-a810-f65702e88fd5-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:31.606388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.606320 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:31.606388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.606334 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:31.606388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.606348 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:31.615874 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.615844 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-podinfo" pod "a5a3df96-4e74-44bd-a810-f65702e88fd5" (UID: "a5a3df96-4e74-44bd-a810-f65702e88fd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:19:31.616170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.616139 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-token" pod "a5a3df96-4e74-44bd-a810-f65702e88fd5" (UID: "a5a3df96-4e74-44bd-a810-f65702e88fd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:31.616291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.616179 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-kube-api-access-8thkk" pod "a5a3df96-4e74-44bd-a810-f65702e88fd5" (UID: "a5a3df96-4e74-44bd-a810-f65702e88fd5"). InnerVolumeSpecName "kube-api-access-8thkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:31.616412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.616389 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-envoy" pod "a5a3df96-4e74-44bd-a810-f65702e88fd5" (UID: "a5a3df96-4e74-44bd-a810-f65702e88fd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:31.708205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.708169 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8thkk\" (UniqueName: \"kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-kube-api-access-8thkk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:31.708205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.708199 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:31.708205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.708213 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:31.708493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:31.708224 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a3df96-4e74-44bd-a810-f65702e88fd5-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:32.290306 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:32.290264 6587 generic.go:356] "Generic (PLEG): container finished" podID="a5a3df96-4e74-44bd-a810-f65702e88fd5" containerID="64e21138c69a7d847d684cfe232ff61f1ce1b23a9949a9f23dc550ad8893f27e" exitCode=0 Oct 30 05:19:32.290306 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:32.290306 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" Oct 30 05:19:32.290622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:32.290350 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" event={"ID":"a5a3df96-4e74-44bd-a810-f65702e88fd5","Type":"ContainerDied","Data":"64e21138c69a7d847d684cfe232ff61f1ce1b23a9949a9f23dc550ad8893f27e"} Oct 30 05:19:32.290622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:32.290429 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj" event={"ID":"a5a3df96-4e74-44bd-a810-f65702e88fd5","Type":"ContainerDied","Data":"4635f67bc4c377ccb72a988ce5e356a2e42bd66cd118648d5c5c51646af8f44a"} Oct 30 05:19:32.290622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:32.290458 6587 scope.go:122] "RemoveContainer" containerID="64e21138c69a7d847d684cfe232ff61f1ce1b23a9949a9f23dc550ad8893f27e" Oct 30 05:19:32.302798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:32.302781 6587 scope.go:122] "RemoveContainer" containerID="64e21138c69a7d847d684cfe232ff61f1ce1b23a9949a9f23dc550ad8893f27e" Oct 30 05:19:32.303074 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:19:32.303056 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e21138c69a7d847d684cfe232ff61f1ce1b23a9949a9f23dc550ad8893f27e\": container with ID starting with 64e21138c69a7d847d684cfe232ff61f1ce1b23a9949a9f23dc550ad8893f27e not found: ID does not exist" containerID="64e21138c69a7d847d684cfe232ff61f1ce1b23a9949a9f23dc550ad8893f27e" Oct 30 05:19:32.303115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:32.303082 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e21138c69a7d847d684cfe232ff61f1ce1b23a9949a9f23dc550ad8893f27e"} err="failed to get container status \"64e21138c69a7d847d684cfe232ff61f1ce1b23a9949a9f23dc550ad8893f27e\": rpc error: code = NotFound desc = could not find container \"64e21138c69a7d847d684cfe232ff61f1ce1b23a9949a9f23dc550ad8893f27e\": container with ID starting with 64e21138c69a7d847d684cfe232ff61f1ce1b23a9949a9f23dc550ad8893f27e not found: ID does not exist" Oct 30 05:19:32.364574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:32.364505 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj"] Oct 30 05:19:32.375758 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:32.375722 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj"] Oct 30 05:19:33.584896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.584854 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a3df96-4e74-44bd-a810-f65702e88fd5" path="/var/lib/kubelet/pods/a5a3df96-4e74-44bd-a810-f65702e88fd5/volumes" Oct 30 05:19:33.884875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.884794 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch"] Oct 30 05:19:33.885616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.885599 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4bbc1851-59c4-451d-9345-aff6d78e5a2d" containerName="discovery" Oct 30 05:19:33.885673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.885617 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bbc1851-59c4-451d-9345-aff6d78e5a2d" containerName="discovery" Oct 30 05:19:33.885673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.885630 6587 cpu_manager.go:414] "containerMap: removing container" podUID="6dc48b09-eba2-4237-be66-16087f00866e" containerName="payload-processing" Oct 30 05:19:33.885673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.885635 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc48b09-eba2-4237-be66-16087f00866e" containerName="payload-processing" Oct 30 05:19:33.885673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.885673 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a5a3df96-4e74-44bd-a810-f65702e88fd5" containerName="istio-proxy" Oct 30 05:19:33.885797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.885680 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a3df96-4e74-44bd-a810-f65702e88fd5" containerName="istio-proxy" Oct 30 05:19:33.885797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.885690 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4aa0d385-5478-4563-99c2-a027e159bf06" containerName="revoke-keys" Oct 30 05:19:33.885797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.885695 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa0d385-5478-4563-99c2-a027e159bf06" containerName="revoke-keys" Oct 30 05:19:33.885918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.885825 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bbc1851-59c4-451d-9345-aff6d78e5a2d" containerName="discovery" Oct 30 05:19:33.885918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.885832 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc48b09-eba2-4237-be66-16087f00866e" containerName="payload-processing" Oct 30 05:19:33.885918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.885845 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a3df96-4e74-44bd-a810-f65702e88fd5" containerName="istio-proxy" Oct 30 05:19:33.885918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.885851 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa0d385-5478-4563-99c2-a027e159bf06" containerName="revoke-keys" Oct 30 05:19:33.891145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.891123 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" Oct 30 05:19:33.908792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.908764 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch"] Oct 30 05:19:33.917403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.916798 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:19:33.917403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.916797 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:19:33.917403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.916825 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:19:33.923454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:33.920937 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:19:34.081422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:34.081388 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-010013-4rhch\" (UID: \"612263aa-4f8c-463a-bde7-da8b0c879b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" Oct 30 05:19:34.081626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:34.081430 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-010013-4rhch\" (UID: \"612263aa-4f8c-463a-bde7-da8b0c879b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" Oct 30 05:19:34.081626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:34.081564 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9c7w\" (UniqueName: \"kubernetes.io/projected/612263aa-4f8c-463a-bde7-da8b0c879b6a-kube-api-access-k9c7w\") pod \"maas-api-revoke-keys-e2e-inf-a-010013-4rhch\" (UID: \"612263aa-4f8c-463a-bde7-da8b0c879b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" Oct 30 05:19:34.183589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:34.183505 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-010013-4rhch\" (UID: \"612263aa-4f8c-463a-bde7-da8b0c879b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" Oct 30 05:19:34.183589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:34.183549 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-010013-4rhch\" (UID: \"612263aa-4f8c-463a-bde7-da8b0c879b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" Oct 30 05:19:34.183826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:34.183595 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9c7w\" (UniqueName: \"kubernetes.io/projected/612263aa-4f8c-463a-bde7-da8b0c879b6a-kube-api-access-k9c7w\") pod \"maas-api-revoke-keys-e2e-inf-a-010013-4rhch\" (UID: \"612263aa-4f8c-463a-bde7-da8b0c879b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" Oct 30 05:19:34.184314 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:34.184285 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-010013-4rhch\" (UID: \"612263aa-4f8c-463a-bde7-da8b0c879b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" Oct 30 05:19:34.186032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:34.186006 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-010013-4rhch\" (UID: \"612263aa-4f8c-463a-bde7-da8b0c879b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" Oct 30 05:19:34.204604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:34.204578 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9c7w\" (UniqueName: \"kubernetes.io/projected/612263aa-4f8c-463a-bde7-da8b0c879b6a-kube-api-access-k9c7w\") pod \"maas-api-revoke-keys-e2e-inf-a-010013-4rhch\" (UID: \"612263aa-4f8c-463a-bde7-da8b0c879b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" Oct 30 05:19:34.206419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:34.206402 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" Oct 30 05:19:34.509097 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:19:34.509048 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612263aa_4f8c_463a_bde7_da8b0c879b6a.slice/crio-db466a2b11baa3862025bf9dff4033d2ff09821c55d7d89e9a0fa6340a621e40 WatchSource:0}: Error finding container db466a2b11baa3862025bf9dff4033d2ff09821c55d7d89e9a0fa6340a621e40: Status 404 returned error can't find the container with id db466a2b11baa3862025bf9dff4033d2ff09821c55d7d89e9a0fa6340a621e40 Oct 30 05:19:34.509973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:34.509945 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch"] Oct 30 05:19:35.307712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.307675 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27"] Oct 30 05:19:35.314116 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.314092 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.344197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.344160 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27"] Oct 30 05:19:35.345098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.345023 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-a914e9\"/\"default-dockercfg-75585\"" Oct 30 05:19:35.345271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.345126 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-a914e9\"/\"auth-test-model-a914e9-kserve-self-signed-certs\"" Oct 30 05:19:35.346911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.346884 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" event={"ID":"612263aa-4f8c-463a-bde7-da8b0c879b6a","Type":"ContainerStarted","Data":"5af747b8a1aeea5a2ea819109c3039c15ff9693c6edb4e675755ec8a675d0a9a"} Oct 30 05:19:35.347023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.346938 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" event={"ID":"612263aa-4f8c-463a-bde7-da8b0c879b6a","Type":"ContainerStarted","Data":"db466a2b11baa3862025bf9dff4033d2ff09821c55d7d89e9a0fa6340a621e40"} Oct 30 05:19:35.347939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.347916 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-a914e9\"/\"kube-root-ca.crt\"" Oct 30 05:19:35.348227 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.348206 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-a914e9\"/\"openshift-service-ca.crt\"" Oct 30 05:19:35.401508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.401468 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5szm\" (UniqueName: \"kubernetes.io/projected/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-kube-api-access-t5szm\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.401738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.401523 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tls-certs\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.401738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.401640 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-dshm\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.401836 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.401755 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-model-cache\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.401836 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.401816 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tmp-dir\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.401954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.401841 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-home\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.415513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.415452 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" podStartSLOduration=2.415434403 podStartE2EDuration="2.415434403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:35.415089182 +0000 UTC m=+2099.942855970" watchObservedRunningTime="2026-10-30 05:19:35.415434403 +0000 UTC m=+2099.943201189" Oct 30 05:19:35.503421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.503353 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tmp-dir\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.503620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.503435 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-home\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.503620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.503615 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5szm\" (UniqueName: \"kubernetes.io/projected/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-kube-api-access-t5szm\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.503765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.503650 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tls-certs\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.503765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.503699 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-dshm\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.503884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.503779 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-model-cache\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.503884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.503813 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tmp-dir\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.503884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.503862 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-home\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.504160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.504138 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-model-cache\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.505987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.505965 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-dshm\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.506233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.506216 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tls-certs\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.525014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.524981 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5szm\" (UniqueName: \"kubernetes.io/projected/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-kube-api-access-t5szm\") pod \"auth-test-model-a914e9-kserve-c5c66466f-zcj27\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:35.677383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.676858 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-a914e9\"/\"default-dockercfg-75585\"" Oct 30 05:19:35.691411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:35.690288 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:36.015312 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:19:36.015278 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9325b4e_ca66_4cff_9104_ab1959ef8e8c.slice/crio-705c153798536bee07da554a627db2a624116744a64ebf389e707288bb667d89 WatchSource:0}: Error finding container 705c153798536bee07da554a627db2a624116744a64ebf389e707288bb667d89: Status 404 returned error can't find the container with id 705c153798536bee07da554a627db2a624116744a64ebf389e707288bb667d89 Oct 30 05:19:36.018652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:36.018626 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27"] Oct 30 05:19:36.042379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:36.033483 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:19:36.042379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:36.035235 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:19:36.358550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:36.358511 6587 generic.go:356] "Generic (PLEG): container finished" podID="612263aa-4f8c-463a-bde7-da8b0c879b6a" containerID="5af747b8a1aeea5a2ea819109c3039c15ff9693c6edb4e675755ec8a675d0a9a" exitCode=0 Oct 30 05:19:36.359074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:36.358592 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" event={"ID":"612263aa-4f8c-463a-bde7-da8b0c879b6a","Type":"ContainerDied","Data":"5af747b8a1aeea5a2ea819109c3039c15ff9693c6edb4e675755ec8a675d0a9a"} Oct 30 05:19:36.360615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:36.360584 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" event={"ID":"e9325b4e-ca66-4cff-9104-ab1959ef8e8c","Type":"ContainerStarted","Data":"51a64992d018b04be588426e3f21c5039eefc4df31337060f5e24a5cee38c8f0"} Oct 30 05:19:36.360755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:36.360632 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" event={"ID":"e9325b4e-ca66-4cff-9104-ab1959ef8e8c","Type":"ContainerStarted","Data":"705c153798536bee07da554a627db2a624116744a64ebf389e707288bb667d89"} Oct 30 05:19:36.446118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:36.446054 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" podStartSLOduration=1.44603531 podStartE2EDuration="1.44603531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:36.442009897 +0000 UTC m=+2100.969776683" watchObservedRunningTime="2026-10-30 05:19:36.44603531 +0000 UTC m=+2100.973802098" Oct 30 05:19:36.691308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:36.691229 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:37.550423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:37.550397 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" Oct 30 05:19:37.650866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:37.650779 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-openshift-ca\") pod \"612263aa-4f8c-463a-bde7-da8b0c879b6a\" (UID: \"612263aa-4f8c-463a-bde7-da8b0c879b6a\") " Oct 30 05:19:37.651040 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:37.650984 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612263aa-4f8c-463a-bde7-da8b0c879b6a-kube-api-access-k9c7w\" (UniqueName: \"kubernetes.io/projected/612263aa-4f8c-463a-bde7-da8b0c879b6a-kube-api-access-k9c7w\") pod \"612263aa-4f8c-463a-bde7-da8b0c879b6a\" (UID: \"612263aa-4f8c-463a-bde7-da8b0c879b6a\") " Oct 30 05:19:37.651118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:37.651096 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-openshift-ca" pod "612263aa-4f8c-463a-bde7-da8b0c879b6a" (UID: "612263aa-4f8c-463a-bde7-da8b0c879b6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:19:37.651179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:37.651129 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-xks-ca\") pod \"612263aa-4f8c-463a-bde7-da8b0c879b6a\" (UID: \"612263aa-4f8c-463a-bde7-da8b0c879b6a\") " Oct 30 05:19:37.651970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:37.651906 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:37.653286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:37.653253 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612263aa-4f8c-463a-bde7-da8b0c879b6a-kube-api-access-k9c7w" pod "612263aa-4f8c-463a-bde7-da8b0c879b6a" (UID: "612263aa-4f8c-463a-bde7-da8b0c879b6a"). InnerVolumeSpecName "kube-api-access-k9c7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:37.653521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:37.653388 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-xks-ca" pod "612263aa-4f8c-463a-bde7-da8b0c879b6a" (UID: "612263aa-4f8c-463a-bde7-da8b0c879b6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:19:37.753951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:37.753911 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9c7w\" (UniqueName: \"kubernetes.io/projected/612263aa-4f8c-463a-bde7-da8b0c879b6a-kube-api-access-k9c7w\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:37.753951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:37.753951 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612263aa-4f8c-463a-bde7-da8b0c879b6a-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:38.385324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:38.385292 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" Oct 30 05:19:38.385555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:38.385430 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch" event={"ID":"612263aa-4f8c-463a-bde7-da8b0c879b6a","Type":"ContainerDied","Data":"db466a2b11baa3862025bf9dff4033d2ff09821c55d7d89e9a0fa6340a621e40"} Oct 30 05:19:38.385555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:38.385471 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db466a2b11baa3862025bf9dff4033d2ff09821c55d7d89e9a0fa6340a621e40" Oct 30 05:19:39.134837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:39.134455 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8"] Oct 30 05:19:39.135842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:39.135816 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-36fd3c-jv9h8"] Oct 30 05:19:39.585299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:39.585265 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1b77ed-5eae-4118-b987-71f5c8ec7560" path="/var/lib/kubelet/pods/db1b77ed-5eae-4118-b987-71f5c8ec7560/volumes" Oct 30 05:19:40.204222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:40.204190 6587 scope.go:122] "RemoveContainer" containerID="ab1d4a55e7a0f7bfc14dc709b70e8e68f4a56e09eeac3c75d9dcd8c49011de63" Oct 30 05:19:40.215142 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:40.215121 6587 scope.go:122] "RemoveContainer" containerID="af3a7576ffd4d4081caa1b0fc351b9e13c74ca1d91fc8e8ce7592fd53572f5a4" Oct 30 05:19:40.230720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:40.230691 6587 scope.go:122] "RemoveContainer" containerID="8e66baba563954f8df31237655ef91a8d3b03a492c1d6b5a1f6add54d53d2ddf" Oct 30 05:19:40.240769 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:40.240747 6587 scope.go:122] "RemoveContainer" containerID="f78f3c2d2abc54711b6bc1c9ead1a801bb6dada84c1979caa44e454e945d78f7" Oct 30 05:19:41.161569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:41.157521 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk"] Oct 30 05:19:41.161569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:41.157842 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" podUID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" containerName="istio-proxy" containerID="cri-o://78dd06b19f73cd4d87261952ef85d88d14a79f6898eb8fa9ab8af52849b341b3" gracePeriod=30 Oct 30 05:19:43.668969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.668939 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5"] Oct 30 05:19:43.669777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.669761 6587 cpu_manager.go:414] "containerMap: removing container" podUID="612263aa-4f8c-463a-bde7-da8b0c879b6a" containerName="revoke-keys" Oct 30 05:19:43.669777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.669778 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612263aa-4f8c-463a-bde7-da8b0c879b6a" containerName="revoke-keys" Oct 30 05:19:43.669910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.669901 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="612263aa-4f8c-463a-bde7-da8b0c879b6a" containerName="revoke-keys" Oct 30 05:19:43.675795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.675780 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" Oct 30 05:19:43.693837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.693810 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5"] Oct 30 05:19:43.703526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.703501 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:19:43.703706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.703682 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:19:43.703787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.703765 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:19:43.703846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.703805 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:19:43.757090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.757050 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dbccfc3-c2nk5\" (UID: \"d639fb54-565e-4146-b979-6e52acaed8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" Oct 30 05:19:43.757507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.757483 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgdpd\" (UniqueName: \"kubernetes.io/projected/d639fb54-565e-4146-b979-6e52acaed8d3-kube-api-access-mgdpd\") pod \"maas-api-revoke-sub-9dbccfc3-c2nk5\" (UID: \"d639fb54-565e-4146-b979-6e52acaed8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" Oct 30 05:19:43.757647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.757523 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dbccfc3-c2nk5\" (UID: \"d639fb54-565e-4146-b979-6e52acaed8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" Oct 30 05:19:43.859316 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.859274 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dbccfc3-c2nk5\" (UID: \"d639fb54-565e-4146-b979-6e52acaed8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" Oct 30 05:19:43.859507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.859434 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dbccfc3-c2nk5\" (UID: \"d639fb54-565e-4146-b979-6e52acaed8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" Oct 30 05:19:43.859763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.859744 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgdpd\" (UniqueName: \"kubernetes.io/projected/d639fb54-565e-4146-b979-6e52acaed8d3-kube-api-access-mgdpd\") pod \"maas-api-revoke-sub-9dbccfc3-c2nk5\" (UID: \"d639fb54-565e-4146-b979-6e52acaed8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" Oct 30 05:19:43.860133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.860103 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dbccfc3-c2nk5\" (UID: \"d639fb54-565e-4146-b979-6e52acaed8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" Oct 30 05:19:43.861908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.861881 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dbccfc3-c2nk5\" (UID: \"d639fb54-565e-4146-b979-6e52acaed8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" Oct 30 05:19:43.880735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.880714 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgdpd\" (UniqueName: \"kubernetes.io/projected/d639fb54-565e-4146-b979-6e52acaed8d3-kube-api-access-mgdpd\") pod \"maas-api-revoke-sub-9dbccfc3-c2nk5\" (UID: \"d639fb54-565e-4146-b979-6e52acaed8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" Oct 30 05:19:43.998083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:43.998003 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" Oct 30 05:19:44.304999 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:19:44.304962 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd639fb54_565e_4146_b979_6e52acaed8d3.slice/crio-c6d1fa21dd9b29aaca632ccce38d1c285b97993ca3648cd99e916fdf21e4bc02 WatchSource:0}: Error finding container c6d1fa21dd9b29aaca632ccce38d1c285b97993ca3648cd99e916fdf21e4bc02: Status 404 returned error can't find the container with id c6d1fa21dd9b29aaca632ccce38d1c285b97993ca3648cd99e916fdf21e4bc02 Oct 30 05:19:44.307302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:44.307275 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5"] Oct 30 05:19:44.444928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:44.444876 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" event={"ID":"d639fb54-565e-4146-b979-6e52acaed8d3","Type":"ContainerStarted","Data":"c6d1fa21dd9b29aaca632ccce38d1c285b97993ca3648cd99e916fdf21e4bc02"} Oct 30 05:19:45.455824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:45.455780 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" event={"ID":"d639fb54-565e-4146-b979-6e52acaed8d3","Type":"ContainerStarted","Data":"d29ffc2c9cd06d155a68f0f4703c652662a02cee93dacaf115873e27933a34ee"} Oct 30 05:19:45.497807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:45.497755 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" podStartSLOduration=2.49774169 podStartE2EDuration="2.49774169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:45.49680269 +0000 UTC m=+2110.024569476" watchObservedRunningTime="2026-10-30 05:19:45.49774169 +0000 UTC m=+2110.025508475" Oct 30 05:19:45.691588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:45.691558 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:45.704244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:45.704212 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:46.467924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.467888 6587 generic.go:356] "Generic (PLEG): container finished" podID="d639fb54-565e-4146-b979-6e52acaed8d3" containerID="d29ffc2c9cd06d155a68f0f4703c652662a02cee93dacaf115873e27933a34ee" exitCode=0 Oct 30 05:19:46.468374 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.467973 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" event={"ID":"d639fb54-565e-4146-b979-6e52acaed8d3","Type":"ContainerDied","Data":"d29ffc2c9cd06d155a68f0f4703c652662a02cee93dacaf115873e27933a34ee"} Oct 30 05:19:46.469643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.469619 6587 generic.go:356] "Generic (PLEG): container finished" podID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" containerID="78dd06b19f73cd4d87261952ef85d88d14a79f6898eb8fa9ab8af52849b341b3" exitCode=0 Oct 30 05:19:46.469772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.469746 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" event={"ID":"eb42bb9d-5a6f-4124-a5c1-b200eee2901d","Type":"ContainerDied","Data":"78dd06b19f73cd4d87261952ef85d88d14a79f6898eb8fa9ab8af52849b341b3"} Oct 30 05:19:46.469840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.469803 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" event={"ID":"eb42bb9d-5a6f-4124-a5c1-b200eee2901d","Type":"ContainerDied","Data":"45c16902a6c30c6a0da9f48af8ecf7f085cba85b1c13c768711e4b4a6b743dab"} Oct 30 05:19:46.469840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.469831 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45c16902a6c30c6a0da9f48af8ecf7f085cba85b1c13c768711e4b4a6b743dab" Oct 30 05:19:46.482319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.482295 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:19:46.492093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.492073 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:19:46.615474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.615396 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-credential-socket\") pod \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " Oct 30 05:19:46.616230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.616091 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istiod-ca-cert\") pod \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " Oct 30 05:19:46.616230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.616135 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-podinfo\") pod \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " Oct 30 05:19:46.616230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.616166 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-data\") pod \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " Oct 30 05:19:46.617046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.616315 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-certs\") pod \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " Oct 30 05:19:46.617046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.616415 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-kube-api-access-8ttfg\" (UniqueName: \"kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-kube-api-access-8ttfg\") pod \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " Oct 30 05:19:46.617046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.616462 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-envoy\") pod \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " Oct 30 05:19:46.617046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.616663 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-socket\") pod \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " Oct 30 05:19:46.617046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.616704 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-token\" (UniqueName: \"kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-token\") pod \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\" (UID: \"eb42bb9d-5a6f-4124-a5c1-b200eee2901d\") " Oct 30 05:19:46.618278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.615960 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-credential-socket" pod "eb42bb9d-5a6f-4124-a5c1-b200eee2901d" (UID: "eb42bb9d-5a6f-4124-a5c1-b200eee2901d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:46.618278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.617686 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-certs" pod "eb42bb9d-5a6f-4124-a5c1-b200eee2901d" (UID: "eb42bb9d-5a6f-4124-a5c1-b200eee2901d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:46.618278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.617699 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-socket" pod "eb42bb9d-5a6f-4124-a5c1-b200eee2901d" (UID: "eb42bb9d-5a6f-4124-a5c1-b200eee2901d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:46.618278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.618239 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:46.618278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.618264 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:46.618278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.618278 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:46.619561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.619325 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istiod-ca-cert" pod "eb42bb9d-5a6f-4124-a5c1-b200eee2901d" (UID: "eb42bb9d-5a6f-4124-a5c1-b200eee2901d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:19:46.619561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.619519 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-data" pod "eb42bb9d-5a6f-4124-a5c1-b200eee2901d" (UID: "eb42bb9d-5a6f-4124-a5c1-b200eee2901d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:46.626669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.626623 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-podinfo" pod "eb42bb9d-5a6f-4124-a5c1-b200eee2901d" (UID: "eb42bb9d-5a6f-4124-a5c1-b200eee2901d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:19:46.653595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.653531 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-envoy" pod "eb42bb9d-5a6f-4124-a5c1-b200eee2901d" (UID: "eb42bb9d-5a6f-4124-a5c1-b200eee2901d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:46.653595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.653572 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-kube-api-access-8ttfg" pod "eb42bb9d-5a6f-4124-a5c1-b200eee2901d" (UID: "eb42bb9d-5a6f-4124-a5c1-b200eee2901d"). InnerVolumeSpecName "kube-api-access-8ttfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:46.653805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.653603 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-token" pod "eb42bb9d-5a6f-4124-a5c1-b200eee2901d" (UID: "eb42bb9d-5a6f-4124-a5c1-b200eee2901d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:46.662541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.662510 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn"] Oct 30 05:19:46.663004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.662974 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerName="istio-proxy" containerID="cri-o://84d4ecc6b6ecf5a2dcfa8e1d9aedf9254e966eac463b790fae718324982688dd" gracePeriod=30 Oct 30 05:19:46.719653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.719571 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ttfg\" (UniqueName: \"kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-kube-api-access-8ttfg\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:46.719653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.719609 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:46.719653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.719624 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:46.719653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.719639 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:46.719653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.719652 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:46.719928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:46.719667 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42bb9d-5a6f-4124-a5c1-b200eee2901d-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:47.478792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.478740 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk" Oct 30 05:19:47.561740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.561702 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk"] Oct 30 05:19:47.596507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.596474 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk"] Oct 30 05:19:47.695568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.695535 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" Oct 30 05:19:47.858409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.858373 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-xks-ca\") pod \"d639fb54-565e-4146-b979-6e52acaed8d3\" (UID: \"d639fb54-565e-4146-b979-6e52acaed8d3\") " Oct 30 05:19:47.858583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.858429 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-openshift-ca\") pod \"d639fb54-565e-4146-b979-6e52acaed8d3\" (UID: \"d639fb54-565e-4146-b979-6e52acaed8d3\") " Oct 30 05:19:47.858583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.858515 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d639fb54-565e-4146-b979-6e52acaed8d3-kube-api-access-mgdpd\" (UniqueName: \"kubernetes.io/projected/d639fb54-565e-4146-b979-6e52acaed8d3-kube-api-access-mgdpd\") pod \"d639fb54-565e-4146-b979-6e52acaed8d3\" (UID: \"d639fb54-565e-4146-b979-6e52acaed8d3\") " Oct 30 05:19:47.858837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.858809 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-openshift-ca" pod "d639fb54-565e-4146-b979-6e52acaed8d3" (UID: "d639fb54-565e-4146-b979-6e52acaed8d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:19:47.859342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.859315 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:47.860659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.860630 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-xks-ca" pod "d639fb54-565e-4146-b979-6e52acaed8d3" (UID: "d639fb54-565e-4146-b979-6e52acaed8d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:19:47.860780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.860759 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d639fb54-565e-4146-b979-6e52acaed8d3-kube-api-access-mgdpd" pod "d639fb54-565e-4146-b979-6e52acaed8d3" (UID: "d639fb54-565e-4146-b979-6e52acaed8d3"). InnerVolumeSpecName "kube-api-access-mgdpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:47.962438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.962405 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d639fb54-565e-4146-b979-6e52acaed8d3-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:47.962438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.962438 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgdpd\" (UniqueName: \"kubernetes.io/projected/d639fb54-565e-4146-b979-6e52acaed8d3-kube-api-access-mgdpd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:47.968673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.968642 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v"] Oct 30 05:19:47.969477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.969459 6587 cpu_manager.go:414] "containerMap: removing container" podUID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" containerName="istio-proxy" Oct 30 05:19:47.969477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.969477 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" containerName="istio-proxy" Oct 30 05:19:47.969616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.969510 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d639fb54-565e-4146-b979-6e52acaed8d3" containerName="revoke-keys" Oct 30 05:19:47.969616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.969515 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d639fb54-565e-4146-b979-6e52acaed8d3" containerName="revoke-keys" Oct 30 05:19:47.969731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.969676 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" containerName="istio-proxy" Oct 30 05:19:47.969731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.969686 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d639fb54-565e-4146-b979-6e52acaed8d3" containerName="revoke-keys" Oct 30 05:19:47.973990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.973963 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:47.998478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:47.994755 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-34722e-openshift-default-dockercfg-nq64s\"" Oct 30 05:19:48.002701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.002673 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v"] Oct 30 05:19:48.064289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.064252 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-certs\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.064481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.064297 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-envoy\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.064571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.064498 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.064571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.064557 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405736d9-0831-4381-9707-a3392b6da104-istio-podinfo\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.064727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.064699 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-credential-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.064869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.064832 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvdm9\" (UniqueName: \"kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-kube-api-access-pvdm9\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.064935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.064896 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-istio-token\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.064935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.064930 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-data\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.065042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.064960 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405736d9-0831-4381-9707-a3392b6da104-istiod-ca-cert\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.166410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.166310 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvdm9\" (UniqueName: \"kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-kube-api-access-pvdm9\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.166410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.166350 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-istio-token\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.166614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.166418 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-data\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.166614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.166445 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405736d9-0831-4381-9707-a3392b6da104-istiod-ca-cert\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.166614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.166526 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-certs\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.166614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.166569 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-envoy\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.166858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.166728 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.166858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.166835 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405736d9-0831-4381-9707-a3392b6da104-istio-podinfo\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.166975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.166854 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-data\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.166975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.166953 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-credential-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.167089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.166987 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-certs\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.167089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.167045 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.167316 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.167291 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405736d9-0831-4381-9707-a3392b6da104-istiod-ca-cert\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.167448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.167298 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-credential-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.168906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.168882 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-envoy\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.169133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.169112 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405736d9-0831-4381-9707-a3392b6da104-istio-podinfo\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.187179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.187154 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-istio-token\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.187314 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.187229 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvdm9\" (UniqueName: \"kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-kube-api-access-pvdm9\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.305393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.304981 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:48.501916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.497517 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" event={"ID":"d639fb54-565e-4146-b979-6e52acaed8d3","Type":"ContainerDied","Data":"c6d1fa21dd9b29aaca632ccce38d1c285b97993ca3648cd99e916fdf21e4bc02"} Oct 30 05:19:48.501916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.497588 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6d1fa21dd9b29aaca632ccce38d1c285b97993ca3648cd99e916fdf21e4bc02" Oct 30 05:19:48.501916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.497653 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5" Oct 30 05:19:48.742052 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:19:48.742020 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405736d9_0831_4381_9707_a3392b6da104.slice/crio-f54de6f7081f05d3c36cd7d656400f2eeec26d7c102548592444a661b8332186 WatchSource:0}: Error finding container f54de6f7081f05d3c36cd7d656400f2eeec26d7c102548592444a661b8332186: Status 404 returned error can't find the container with id f54de6f7081f05d3c36cd7d656400f2eeec26d7c102548592444a661b8332186 Oct 30 05:19:48.745677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.745637 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:19:48.745798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.745748 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:19:48.745867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.745795 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:19:48.750537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:48.750516 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v"] Oct 30 05:19:49.507500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:49.507456 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" event={"ID":"405736d9-0831-4381-9707-a3392b6da104","Type":"ContainerStarted","Data":"99c1ab4fedcc6bb6f05ed8d2ba2610dc30192efaf7147393f45bef1634a1488f"} Oct 30 05:19:49.507896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:49.507524 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" event={"ID":"405736d9-0831-4381-9707-a3392b6da104","Type":"ContainerStarted","Data":"f54de6f7081f05d3c36cd7d656400f2eeec26d7c102548592444a661b8332186"} Oct 30 05:19:49.582848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:49.582778 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" podStartSLOduration=2.582759621 podStartE2EDuration="2.582759621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:49.565472455 +0000 UTC m=+2114.093239243" watchObservedRunningTime="2026-10-30 05:19:49.582759621 +0000 UTC m=+2114.110526409" Oct 30 05:19:49.586290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:49.586260 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb42bb9d-5a6f-4124-a5c1-b200eee2901d" path="/var/lib/kubelet/pods/eb42bb9d-5a6f-4124-a5c1-b200eee2901d/volumes" Oct 30 05:19:50.412845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:50.412447 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:50.436386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:50.435187 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:50.515282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:50.515248 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:50.516301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:50.516277 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:19:51.974075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:51.974049 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:19:52.068602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.068564 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-socket\") pod \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " Oct 30 05:19:52.068796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.068655 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-token\" (UniqueName: \"kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-token\") pod \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " Oct 30 05:19:52.068796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.068767 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-envoy\") pod \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " Oct 30 05:19:52.068957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.068807 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-certs\") pod \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " Oct 30 05:19:52.068957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.068929 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-credential-socket\") pod \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " Oct 30 05:19:52.069101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.068978 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istiod-ca-cert\") pod \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " Oct 30 05:19:52.069101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.069006 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-kube-api-access-4jrxq\" (UniqueName: \"kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-kube-api-access-4jrxq\") pod \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " Oct 30 05:19:52.069101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.069037 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-podinfo\") pod \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " Oct 30 05:19:52.069101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.069084 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-data\") pod \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\" (UID: \"cb57c417-36f4-4a6b-9521-9bcfd52abf8c\") " Oct 30 05:19:52.070488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.070245 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-data" pod "cb57c417-36f4-4a6b-9521-9bcfd52abf8c" (UID: "cb57c417-36f4-4a6b-9521-9bcfd52abf8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:52.070689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.070659 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-certs" pod "cb57c417-36f4-4a6b-9521-9bcfd52abf8c" (UID: "cb57c417-36f4-4a6b-9521-9bcfd52abf8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:52.071123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.071101 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-envoy" pod "cb57c417-36f4-4a6b-9521-9bcfd52abf8c" (UID: "cb57c417-36f4-4a6b-9521-9bcfd52abf8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:52.071289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.071272 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-socket" pod "cb57c417-36f4-4a6b-9521-9bcfd52abf8c" (UID: "cb57c417-36f4-4a6b-9521-9bcfd52abf8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:52.072796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.072769 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istiod-ca-cert" pod "cb57c417-36f4-4a6b-9521-9bcfd52abf8c" (UID: "cb57c417-36f4-4a6b-9521-9bcfd52abf8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:19:52.073332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.073308 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-token" pod "cb57c417-36f4-4a6b-9521-9bcfd52abf8c" (UID: "cb57c417-36f4-4a6b-9521-9bcfd52abf8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:52.074261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.074233 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-credential-socket" pod "cb57c417-36f4-4a6b-9521-9bcfd52abf8c" (UID: "cb57c417-36f4-4a6b-9521-9bcfd52abf8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:52.074522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.074494 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-podinfo" pod "cb57c417-36f4-4a6b-9521-9bcfd52abf8c" (UID: "cb57c417-36f4-4a6b-9521-9bcfd52abf8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:19:52.074821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.074796 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-kube-api-access-4jrxq" pod "cb57c417-36f4-4a6b-9521-9bcfd52abf8c" (UID: "cb57c417-36f4-4a6b-9521-9bcfd52abf8c"). InnerVolumeSpecName "kube-api-access-4jrxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:52.171186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.171154 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:52.171186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.171185 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jrxq\" (UniqueName: \"kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-kube-api-access-4jrxq\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:52.171428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.171198 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:52.171428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.171211 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:52.171428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.171224 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:52.171428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.171234 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:52.171428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.171246 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:52.171428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.171257 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:52.171428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.171268 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb57c417-36f4-4a6b-9521-9bcfd52abf8c-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:19:52.534231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.534195 6587 generic.go:356] "Generic (PLEG): container finished" podID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerID="84d4ecc6b6ecf5a2dcfa8e1d9aedf9254e966eac463b790fae718324982688dd" exitCode=0 Oct 30 05:19:52.534440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.534249 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" Oct 30 05:19:52.534440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.534272 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" event={"ID":"cb57c417-36f4-4a6b-9521-9bcfd52abf8c","Type":"ContainerDied","Data":"84d4ecc6b6ecf5a2dcfa8e1d9aedf9254e966eac463b790fae718324982688dd"} Oct 30 05:19:52.534440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.534340 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn" event={"ID":"cb57c417-36f4-4a6b-9521-9bcfd52abf8c","Type":"ContainerDied","Data":"f28837ee75ac00f5847cb7fc0f5da6cb8b35e80dcfac436b864fc8814f0e4192"} Oct 30 05:19:52.534440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.534383 6587 scope.go:122] "RemoveContainer" containerID="84d4ecc6b6ecf5a2dcfa8e1d9aedf9254e966eac463b790fae718324982688dd" Oct 30 05:19:52.546669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.546649 6587 scope.go:122] "RemoveContainer" containerID="790730cb8d823c19a885dcfe6b420dfb145711d6756de4efe9584410072d187e" Oct 30 05:19:52.556968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.556946 6587 scope.go:122] "RemoveContainer" containerID="84d4ecc6b6ecf5a2dcfa8e1d9aedf9254e966eac463b790fae718324982688dd" Oct 30 05:19:52.557238 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:19:52.557211 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d4ecc6b6ecf5a2dcfa8e1d9aedf9254e966eac463b790fae718324982688dd\": container with ID starting with 84d4ecc6b6ecf5a2dcfa8e1d9aedf9254e966eac463b790fae718324982688dd not found: ID does not exist" containerID="84d4ecc6b6ecf5a2dcfa8e1d9aedf9254e966eac463b790fae718324982688dd" Oct 30 05:19:52.557308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.557245 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d4ecc6b6ecf5a2dcfa8e1d9aedf9254e966eac463b790fae718324982688dd"} err="failed to get container status \"84d4ecc6b6ecf5a2dcfa8e1d9aedf9254e966eac463b790fae718324982688dd\": rpc error: code = NotFound desc = could not find container \"84d4ecc6b6ecf5a2dcfa8e1d9aedf9254e966eac463b790fae718324982688dd\": container with ID starting with 84d4ecc6b6ecf5a2dcfa8e1d9aedf9254e966eac463b790fae718324982688dd not found: ID does not exist" Oct 30 05:19:52.557308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.557264 6587 scope.go:122] "RemoveContainer" containerID="790730cb8d823c19a885dcfe6b420dfb145711d6756de4efe9584410072d187e" Oct 30 05:19:52.557510 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:19:52.557489 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790730cb8d823c19a885dcfe6b420dfb145711d6756de4efe9584410072d187e\": container with ID starting with 790730cb8d823c19a885dcfe6b420dfb145711d6756de4efe9584410072d187e not found: ID does not exist" containerID="790730cb8d823c19a885dcfe6b420dfb145711d6756de4efe9584410072d187e" Oct 30 05:19:52.557563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.557517 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790730cb8d823c19a885dcfe6b420dfb145711d6756de4efe9584410072d187e"} err="failed to get container status \"790730cb8d823c19a885dcfe6b420dfb145711d6756de4efe9584410072d187e\": rpc error: code = NotFound desc = could not find container \"790730cb8d823c19a885dcfe6b420dfb145711d6756de4efe9584410072d187e\": container with ID starting with 790730cb8d823c19a885dcfe6b420dfb145711d6756de4efe9584410072d187e not found: ID does not exist" Oct 30 05:19:52.594257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.594228 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn"] Oct 30 05:19:52.607260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:52.607234 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn"] Oct 30 05:19:53.583468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:53.583436 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" path="/var/lib/kubelet/pods/cb57c417-36f4-4a6b-9521-9bcfd52abf8c/volumes" Oct 30 05:19:54.451524 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.451489 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q"] Oct 30 05:19:54.452287 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.452272 6587 cpu_manager.go:414] "containerMap: removing container" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerName="istio-proxy" Oct 30 05:19:54.452341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.452288 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerName="istio-proxy" Oct 30 05:19:54.452341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.452306 6587 cpu_manager.go:414] "containerMap: removing container" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerName="istio-proxy" Oct 30 05:19:54.452341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.452311 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerName="istio-proxy" Oct 30 05:19:54.452490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.452471 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerName="istio-proxy" Oct 30 05:19:54.452490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.452484 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb57c417-36f4-4a6b-9521-9bcfd52abf8c" containerName="istio-proxy" Oct 30 05:19:54.457977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.457955 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" Oct 30 05:19:54.476428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.476399 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q"] Oct 30 05:19:54.484747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.484718 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-34722e-dockercfg-cjzkt\"" Oct 30 05:19:54.484869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.484788 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-34722e\"" Oct 30 05:19:54.518752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.518722 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz"] Oct 30 05:19:54.523669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.523638 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" Oct 30 05:19:54.544565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.544534 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz"] Oct 30 05:19:54.730791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.686419 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt9sv\" (UniqueName: \"kubernetes.io/projected/d37da4e6-b588-4988-85e8-4658f44dc051-kube-api-access-mt9sv\") pod \"payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q\" (UID: \"d37da4e6-b588-4988-85e8-4658f44dc051\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" Oct 30 05:19:54.730791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.686695 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37da4e6-b588-4988-85e8-4658f44dc051-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q\" (UID: \"d37da4e6-b588-4988-85e8-4658f44dc051\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" Oct 30 05:19:54.730791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.686766 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbgmc\" (UniqueName: \"kubernetes.io/projected/d782c841-b8c5-448c-a60e-2f097ed23724-kube-api-access-hbgmc\") pod \"payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz\" (UID: \"d782c841-b8c5-448c-a60e-2f097ed23724\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" Oct 30 05:19:54.730791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.686892 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d782c841-b8c5-448c-a60e-2f097ed23724-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz\" (UID: \"d782c841-b8c5-448c-a60e-2f097ed23724\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" Oct 30 05:19:54.796505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.793630 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbgmc\" (UniqueName: \"kubernetes.io/projected/d782c841-b8c5-448c-a60e-2f097ed23724-kube-api-access-hbgmc\") pod \"payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz\" (UID: \"d782c841-b8c5-448c-a60e-2f097ed23724\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" Oct 30 05:19:54.796505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.793770 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d782c841-b8c5-448c-a60e-2f097ed23724-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz\" (UID: \"d782c841-b8c5-448c-a60e-2f097ed23724\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" Oct 30 05:19:54.796505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.793852 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt9sv\" (UniqueName: \"kubernetes.io/projected/d37da4e6-b588-4988-85e8-4658f44dc051-kube-api-access-mt9sv\") pod \"payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q\" (UID: \"d37da4e6-b588-4988-85e8-4658f44dc051\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" Oct 30 05:19:54.796505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.794109 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37da4e6-b588-4988-85e8-4658f44dc051-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q\" (UID: \"d37da4e6-b588-4988-85e8-4658f44dc051\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" Oct 30 05:19:54.796505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.794817 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d782c841-b8c5-448c-a60e-2f097ed23724-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz\" (UID: \"d782c841-b8c5-448c-a60e-2f097ed23724\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" Oct 30 05:19:54.796505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.794907 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37da4e6-b588-4988-85e8-4658f44dc051-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q\" (UID: \"d37da4e6-b588-4988-85e8-4658f44dc051\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" Oct 30 05:19:54.815297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.815270 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbgmc\" (UniqueName: \"kubernetes.io/projected/d782c841-b8c5-448c-a60e-2f097ed23724-kube-api-access-hbgmc\") pod \"payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz\" (UID: \"d782c841-b8c5-448c-a60e-2f097ed23724\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" Oct 30 05:19:54.815297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.815289 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt9sv\" (UniqueName: \"kubernetes.io/projected/d37da4e6-b588-4988-85e8-4658f44dc051-kube-api-access-mt9sv\") pod \"payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q\" (UID: \"d37da4e6-b588-4988-85e8-4658f44dc051\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" Oct 30 05:19:54.846054 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:54.846026 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" Oct 30 05:19:55.073908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:55.073873 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" Oct 30 05:19:55.165588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:55.165394 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz"] Oct 30 05:19:55.382555 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:19:55.382526 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37da4e6_b588_4988_85e8_4658f44dc051.slice/crio-d789bf825acdb2f355bcd985dd027f73810ff9fc12f5a25a2150729af4493e75 WatchSource:0}: Error finding container d789bf825acdb2f355bcd985dd027f73810ff9fc12f5a25a2150729af4493e75: Status 404 returned error can't find the container with id d789bf825acdb2f355bcd985dd027f73810ff9fc12f5a25a2150729af4493e75 Oct 30 05:19:55.383597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:55.383563 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q"] Oct 30 05:19:55.843052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:55.843010 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" event={"ID":"d37da4e6-b588-4988-85e8-4658f44dc051","Type":"ContainerStarted","Data":"735cb2773de42be84c7d56e42786f891ce7985bf95f0951a107fa6e33b8e4271"} Oct 30 05:19:55.843664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:55.843636 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" event={"ID":"d37da4e6-b588-4988-85e8-4658f44dc051","Type":"ContainerStarted","Data":"d789bf825acdb2f355bcd985dd027f73810ff9fc12f5a25a2150729af4493e75"} Oct 30 05:19:55.857380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:55.845005 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" Oct 30 05:19:55.894292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:55.894175 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" podStartSLOduration=1.8941550889999998 podStartE2EDuration="1.894155089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:55.891097106 +0000 UTC m=+2120.418863909" watchObservedRunningTime="2026-10-30 05:19:55.894155089 +0000 UTC m=+2120.421921876" Oct 30 05:19:55.902396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:55.902342 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" event={"ID":"d782c841-b8c5-448c-a60e-2f097ed23724","Type":"ContainerStarted","Data":"1eca442a324bafdbfbaf40e45c4ed6e393618b85b8e78c2d10073cd314722707"} Oct 30 05:19:55.903371 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:55.902445 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" event={"ID":"d782c841-b8c5-448c-a60e-2f097ed23724","Type":"ContainerStarted","Data":"fe759dd95936bce3c05872f793e64be7ca6b5ca3340abf0be9bc1468737b2c7b"} Oct 30 05:19:55.904409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:55.904381 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" Oct 30 05:19:55.949324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:55.949270 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" podStartSLOduration=1.949255355 podStartE2EDuration="1.949255355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:55.946595732 +0000 UTC m=+2120.474362519" watchObservedRunningTime="2026-10-30 05:19:55.949255355 +0000 UTC m=+2120.477022138" Oct 30 05:19:55.988388 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:19:55.988339 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:19:56.977419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:56.977380 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck"] Oct 30 05:19:56.982493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:56.982474 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" Oct 30 05:19:57.001112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.001080 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck"] Oct 30 05:19:57.010518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.010490 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:19:57.010676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.010651 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:19:57.040322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.040275 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-892dc917-f86ck\" (UID: \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" Oct 30 05:19:57.040567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.040342 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-przd9\" (UniqueName: \"kubernetes.io/projected/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-kube-api-access-przd9\") pod \"maas-api-revoke-sub-892dc917-f86ck\" (UID: \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" Oct 30 05:19:57.040567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.040502 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-892dc917-f86ck\" (UID: \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" Oct 30 05:19:57.065900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.065871 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:19:57.066049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.065871 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:19:57.142560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.142527 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-892dc917-f86ck\" (UID: \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" Oct 30 05:19:57.142763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.142575 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-przd9\" (UniqueName: \"kubernetes.io/projected/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-kube-api-access-przd9\") pod \"maas-api-revoke-sub-892dc917-f86ck\" (UID: \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" Oct 30 05:19:57.142763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.142633 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-892dc917-f86ck\" (UID: \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" Oct 30 05:19:57.143264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.143223 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-892dc917-f86ck\" (UID: \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" Oct 30 05:19:57.144912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.144888 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-892dc917-f86ck\" (UID: \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" Oct 30 05:19:57.163919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.163882 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-przd9\" (UniqueName: \"kubernetes.io/projected/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-kube-api-access-przd9\") pod \"maas-api-revoke-sub-892dc917-f86ck\" (UID: \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" Oct 30 05:19:57.303161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.303134 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" Oct 30 05:19:57.615019 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:19:57.614987 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18fa2da_c745_43bf_a9cb_a2c8ab8641aa.slice/crio-d2b05870df9c5ecda54d875e884906b45788a7e7a18f621de603041f1f812278 WatchSource:0}: Error finding container d2b05870df9c5ecda54d875e884906b45788a7e7a18f621de603041f1f812278: Status 404 returned error can't find the container with id d2b05870df9c5ecda54d875e884906b45788a7e7a18f621de603041f1f812278 Oct 30 05:19:57.617800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.617771 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck"] Oct 30 05:19:57.921750 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:57.921668 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" event={"ID":"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa","Type":"ContainerStarted","Data":"d2b05870df9c5ecda54d875e884906b45788a7e7a18f621de603041f1f812278"} Oct 30 05:19:58.932891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:58.932856 6587 generic.go:356] "Generic (PLEG): container finished" podID="a18fa2da-c745-43bf-a9cb-a2c8ab8641aa" containerID="d2efa9df0a43aa5b31c4f7ec1bd9f35451abb070693b96b6c3c7928d86e8c378" exitCode=0 Oct 30 05:19:58.933389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:19:58.932961 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" event={"ID":"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa","Type":"ContainerDied","Data":"d2efa9df0a43aa5b31c4f7ec1bd9f35451abb070693b96b6c3c7928d86e8c378"} Oct 30 05:20:00.109234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:00.109202 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" Oct 30 05:20:00.193176 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:00.193097 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-xks-ca\") pod \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\" (UID: \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\") " Oct 30 05:20:00.193421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:00.193178 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-openshift-ca\") pod \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\" (UID: \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\") " Oct 30 05:20:00.193569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:00.193545 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-kube-api-access-przd9\" (UniqueName: \"kubernetes.io/projected/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-kube-api-access-przd9\") pod \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\" (UID: \"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa\") " Oct 30 05:20:00.194938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:00.194907 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-openshift-ca" pod "a18fa2da-c745-43bf-a9cb-a2c8ab8641aa" (UID: "a18fa2da-c745-43bf-a9cb-a2c8ab8641aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:00.195973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:00.195944 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-xks-ca" pod "a18fa2da-c745-43bf-a9cb-a2c8ab8641aa" (UID: "a18fa2da-c745-43bf-a9cb-a2c8ab8641aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:20:00.196623 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:00.196581 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-kube-api-access-przd9" pod "a18fa2da-c745-43bf-a9cb-a2c8ab8641aa" (UID: "a18fa2da-c745-43bf-a9cb-a2c8ab8641aa"). InnerVolumeSpecName "kube-api-access-przd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:00.295292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:00.295259 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-przd9\" (UniqueName: \"kubernetes.io/projected/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-kube-api-access-przd9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:00.295292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:00.295296 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:00.295509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:00.295306 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:00.957860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:00.957825 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" Oct 30 05:20:00.958061 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:00.957865 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck" event={"ID":"a18fa2da-c745-43bf-a9cb-a2c8ab8641aa","Type":"ContainerDied","Data":"d2b05870df9c5ecda54d875e884906b45788a7e7a18f621de603041f1f812278"} Oct 30 05:20:00.958061 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:00.957950 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2b05870df9c5ecda54d875e884906b45788a7e7a18f621de603041f1f812278" Oct 30 05:20:07.925452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:07.925418 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" Oct 30 05:20:07.925915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:07.925873 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" Oct 30 05:20:08.208796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.208703 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5"] Oct 30 05:20:08.211014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.210950 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a18fa2da-c745-43bf-a9cb-a2c8ab8641aa" containerName="revoke-keys" Oct 30 05:20:08.211209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.211190 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18fa2da-c745-43bf-a9cb-a2c8ab8641aa" containerName="revoke-keys" Oct 30 05:20:08.211690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.211670 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18fa2da-c745-43bf-a9cb-a2c8ab8641aa" containerName="revoke-keys" Oct 30 05:20:08.223680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.223648 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" Oct 30 05:20:08.226555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.226528 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5"] Oct 30 05:20:08.253741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.252242 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:20:08.253741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.252411 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:20:08.253741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.252469 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:20:08.253741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.252559 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:20:08.324131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.324092 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5\" (UID: \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" Oct 30 05:20:08.324350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.324233 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5\" (UID: \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" Oct 30 05:20:08.324350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.324318 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvfgv\" (UniqueName: \"kubernetes.io/projected/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-kube-api-access-wvfgv\") pod \"maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5\" (UID: \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" Oct 30 05:20:08.426182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.426138 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5\" (UID: \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" Oct 30 05:20:08.426431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.426226 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5\" (UID: \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" Oct 30 05:20:08.426431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.426298 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvfgv\" (UniqueName: \"kubernetes.io/projected/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-kube-api-access-wvfgv\") pod \"maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5\" (UID: \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" Oct 30 05:20:08.426971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.426946 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5\" (UID: \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" Oct 30 05:20:08.428903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.428881 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5\" (UID: \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" Oct 30 05:20:08.458623 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.458591 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvfgv\" (UniqueName: \"kubernetes.io/projected/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-kube-api-access-wvfgv\") pod \"maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5\" (UID: \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" Oct 30 05:20:08.566539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.566504 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" Oct 30 05:20:08.884293 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:20:08.884263 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda193d20e_a7b3_4470_ae3a_cefaa8f466ff.slice/crio-ee10e3f556ecf735f86d83c17c3540f6073dbecef06dcbb4ddbe633084d2d5a5 WatchSource:0}: Error finding container ee10e3f556ecf735f86d83c17c3540f6073dbecef06dcbb4ddbe633084d2d5a5: Status 404 returned error can't find the container with id ee10e3f556ecf735f86d83c17c3540f6073dbecef06dcbb4ddbe633084d2d5a5 Oct 30 05:20:08.887748 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:08.887719 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5"] Oct 30 05:20:09.033420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:09.033377 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" event={"ID":"a193d20e-a7b3-4470-ae3a-cefaa8f466ff","Type":"ContainerStarted","Data":"ee10e3f556ecf735f86d83c17c3540f6073dbecef06dcbb4ddbe633084d2d5a5"} Oct 30 05:20:10.037119 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:20:10.037089 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda193d20e_a7b3_4470_ae3a_cefaa8f466ff.slice/crio-8c98b567fbaf66daf1704d25ad8ea97375d5c5bb5a9573e8c67fdb655eaa9126.scope\": RecentStats: unable to find data in memory cache]" Oct 30 05:20:10.037585 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:20:10.037215 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda193d20e_a7b3_4470_ae3a_cefaa8f466ff.slice/crio-8c98b567fbaf66daf1704d25ad8ea97375d5c5bb5a9573e8c67fdb655eaa9126.scope\": RecentStats: unable to find data in memory cache]" Oct 30 05:20:10.045029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:10.044994 6587 generic.go:356] "Generic (PLEG): container finished" podID="a193d20e-a7b3-4470-ae3a-cefaa8f466ff" containerID="8c98b567fbaf66daf1704d25ad8ea97375d5c5bb5a9573e8c67fdb655eaa9126" exitCode=0 Oct 30 05:20:10.045188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:10.045136 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" event={"ID":"a193d20e-a7b3-4470-ae3a-cefaa8f466ff","Type":"ContainerDied","Data":"8c98b567fbaf66daf1704d25ad8ea97375d5c5bb5a9573e8c67fdb655eaa9126"} Oct 30 05:20:11.270203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:11.270170 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" Oct 30 05:20:11.379930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:11.379841 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-openshift-ca\") pod \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\" (UID: \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\") " Oct 30 05:20:11.380099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:11.380019 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-kube-api-access-wvfgv\" (UniqueName: \"kubernetes.io/projected/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-kube-api-access-wvfgv\") pod \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\" (UID: \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\") " Oct 30 05:20:11.380099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:11.380040 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-xks-ca\") pod \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\" (UID: \"a193d20e-a7b3-4470-ae3a-cefaa8f466ff\") " Oct 30 05:20:11.380290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:11.380262 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-openshift-ca" pod "a193d20e-a7b3-4470-ae3a-cefaa8f466ff" (UID: "a193d20e-a7b3-4470-ae3a-cefaa8f466ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:11.380760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:11.380737 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:11.382183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:11.382155 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-xks-ca" pod "a193d20e-a7b3-4470-ae3a-cefaa8f466ff" (UID: "a193d20e-a7b3-4470-ae3a-cefaa8f466ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:20:11.382273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:11.382200 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-kube-api-access-wvfgv" pod "a193d20e-a7b3-4470-ae3a-cefaa8f466ff" (UID: "a193d20e-a7b3-4470-ae3a-cefaa8f466ff"). InnerVolumeSpecName "kube-api-access-wvfgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:11.481835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:11.481803 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvfgv\" (UniqueName: \"kubernetes.io/projected/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-kube-api-access-wvfgv\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:11.481835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:11.481832 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a193d20e-a7b3-4470-ae3a-cefaa8f466ff-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:12.064065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:12.064033 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" Oct 30 05:20:12.064269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:12.064072 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5" event={"ID":"a193d20e-a7b3-4470-ae3a-cefaa8f466ff","Type":"ContainerDied","Data":"ee10e3f556ecf735f86d83c17c3540f6073dbecef06dcbb4ddbe633084d2d5a5"} Oct 30 05:20:12.064269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:12.064125 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee10e3f556ecf735f86d83c17c3540f6073dbecef06dcbb4ddbe633084d2d5a5" Oct 30 05:20:13.464320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:13.464280 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch"] Oct 30 05:20:13.495072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:13.495034 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-010013-4rhch"] Oct 30 05:20:13.584435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:13.584396 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="612263aa-4f8c-463a-bde7-da8b0c879b6a" path="/var/lib/kubelet/pods/612263aa-4f8c-463a-bde7-da8b0c879b6a/volumes" Oct 30 05:20:18.682345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:18.682309 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs"] Oct 30 05:20:20.886630 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:20.886573 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv"] Oct 30 05:20:20.887161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:20.886904 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" containerID="cri-o://98c167779fff28559d2cd8c8729bf50ea395c93b3b5ed7d4e58f502caf1b21dd" gracePeriod=30 Oct 30 05:20:21.078030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.077985 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5"] Oct 30 05:20:21.078937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.078913 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a193d20e-a7b3-4470-ae3a-cefaa8f466ff" containerName="revoke-keys" Oct 30 05:20:21.078937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.078937 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a193d20e-a7b3-4470-ae3a-cefaa8f466ff" containerName="revoke-keys" Oct 30 05:20:21.079145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.079121 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a193d20e-a7b3-4470-ae3a-cefaa8f466ff" containerName="revoke-keys" Oct 30 05:20:21.084417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.084397 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" Oct 30 05:20:21.102092 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.102063 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:20:21.102236 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.102128 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:20:21.104736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.104685 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5"] Oct 30 05:20:21.105272 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.105245 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5\" (UID: \"e0c31176-a56f-4b19-bf5d-287db8d23891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" Oct 30 05:20:21.105432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.105287 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxxb4\" (UniqueName: \"kubernetes.io/projected/e0c31176-a56f-4b19-bf5d-287db8d23891-kube-api-access-hxxb4\") pod \"maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5\" (UID: \"e0c31176-a56f-4b19-bf5d-287db8d23891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" Oct 30 05:20:21.105516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.105455 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5\" (UID: \"e0c31176-a56f-4b19-bf5d-287db8d23891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" Oct 30 05:20:21.107037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.106995 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:20:21.201917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.201835 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:20:21.208974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.206978 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5\" (UID: \"e0c31176-a56f-4b19-bf5d-287db8d23891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" Oct 30 05:20:21.208974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.207568 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxxb4\" (UniqueName: \"kubernetes.io/projected/e0c31176-a56f-4b19-bf5d-287db8d23891-kube-api-access-hxxb4\") pod \"maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5\" (UID: \"e0c31176-a56f-4b19-bf5d-287db8d23891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" Oct 30 05:20:21.208974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.207636 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5\" (UID: \"e0c31176-a56f-4b19-bf5d-287db8d23891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" Oct 30 05:20:21.208974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.208442 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5\" (UID: \"e0c31176-a56f-4b19-bf5d-287db8d23891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" Oct 30 05:20:21.210735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.210708 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5\" (UID: \"e0c31176-a56f-4b19-bf5d-287db8d23891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" Oct 30 05:20:21.233403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.231213 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxxb4\" (UniqueName: \"kubernetes.io/projected/e0c31176-a56f-4b19-bf5d-287db8d23891-kube-api-access-hxxb4\") pod \"maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5\" (UID: \"e0c31176-a56f-4b19-bf5d-287db8d23891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" Oct 30 05:20:21.406817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.406779 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" Oct 30 05:20:21.942791 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:20:21.942749 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c31176_a56f_4b19_bf5d_287db8d23891.slice/crio-9ee3c7788695ece49f37c5c2545f3fbeb950a003eae3c34b0408b25cb33b978f WatchSource:0}: Error finding container 9ee3c7788695ece49f37c5c2545f3fbeb950a003eae3c34b0408b25cb33b978f: Status 404 returned error can't find the container with id 9ee3c7788695ece49f37c5c2545f3fbeb950a003eae3c34b0408b25cb33b978f Oct 30 05:20:21.944202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:21.944172 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5"] Oct 30 05:20:22.173002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:22.172968 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" event={"ID":"e0c31176-a56f-4b19-bf5d-287db8d23891","Type":"ContainerStarted","Data":"9ee3c7788695ece49f37c5c2545f3fbeb950a003eae3c34b0408b25cb33b978f"} Oct 30 05:20:23.231787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:23.231257 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" event={"ID":"e0c31176-a56f-4b19-bf5d-287db8d23891","Type":"ContainerStarted","Data":"41e9a278a522be8368a4f4dba7ae69e5e42ef20ea9367767fff13e7e07a5a052"} Oct 30 05:20:23.683172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:23.683096 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" podUID="4012f48b-8a86-4077-9dd5-c6886b47a38d" containerName="payload-pre-processing" containerID="cri-o://c90512f68e5aa1ddf9db108c4e8307d3394487287bf859f6fcd679823bcfc21c" gracePeriod=25 Oct 30 05:20:24.296266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:24.296225 6587 generic.go:356] "Generic (PLEG): container finished" podID="e0c31176-a56f-4b19-bf5d-287db8d23891" containerID="41e9a278a522be8368a4f4dba7ae69e5e42ef20ea9367767fff13e7e07a5a052" exitCode=0 Oct 30 05:20:24.296703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:24.296570 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" event={"ID":"e0c31176-a56f-4b19-bf5d-287db8d23891","Type":"ContainerDied","Data":"41e9a278a522be8368a4f4dba7ae69e5e42ef20ea9367767fff13e7e07a5a052"} Oct 30 05:20:24.556516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:24.556481 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" Oct 30 05:20:24.691835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:24.691803 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c31176-a56f-4b19-bf5d-287db8d23891-kube-api-access-hxxb4\" (UniqueName: \"kubernetes.io/projected/e0c31176-a56f-4b19-bf5d-287db8d23891-kube-api-access-hxxb4\") pod \"e0c31176-a56f-4b19-bf5d-287db8d23891\" (UID: \"e0c31176-a56f-4b19-bf5d-287db8d23891\") " Oct 30 05:20:24.692027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:24.691909 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-openshift-ca\") pod \"e0c31176-a56f-4b19-bf5d-287db8d23891\" (UID: \"e0c31176-a56f-4b19-bf5d-287db8d23891\") " Oct 30 05:20:24.692027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:24.691996 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-xks-ca\") pod \"e0c31176-a56f-4b19-bf5d-287db8d23891\" (UID: \"e0c31176-a56f-4b19-bf5d-287db8d23891\") " Oct 30 05:20:24.692350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:24.692313 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-openshift-ca" pod "e0c31176-a56f-4b19-bf5d-287db8d23891" (UID: "e0c31176-a56f-4b19-bf5d-287db8d23891"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:24.692875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:24.692850 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:24.694385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:24.694247 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c31176-a56f-4b19-bf5d-287db8d23891-kube-api-access-hxxb4" pod "e0c31176-a56f-4b19-bf5d-287db8d23891" (UID: "e0c31176-a56f-4b19-bf5d-287db8d23891"). InnerVolumeSpecName "kube-api-access-hxxb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:24.694385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:24.694326 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-xks-ca" pod "e0c31176-a56f-4b19-bf5d-287db8d23891" (UID: "e0c31176-a56f-4b19-bf5d-287db8d23891"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:20:24.794389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:24.794340 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxxb4\" (UniqueName: \"kubernetes.io/projected/e0c31176-a56f-4b19-bf5d-287db8d23891-kube-api-access-hxxb4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:24.794389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:24.794384 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c31176-a56f-4b19-bf5d-287db8d23891-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:25.305980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:25.305953 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" Oct 30 05:20:25.306433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:25.305953 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5" event={"ID":"e0c31176-a56f-4b19-bf5d-287db8d23891","Type":"ContainerDied","Data":"9ee3c7788695ece49f37c5c2545f3fbeb950a003eae3c34b0408b25cb33b978f"} Oct 30 05:20:25.306433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:25.306077 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ee3c7788695ece49f37c5c2545f3fbeb950a003eae3c34b0408b25cb33b978f" Oct 30 05:20:25.988822 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:20:25.988793 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:20:26.294201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.294172 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:20:26.346714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.346678 6587 generic.go:356] "Generic (PLEG): container finished" podID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerID="98c167779fff28559d2cd8c8729bf50ea395c93b3b5ed7d4e58f502caf1b21dd" exitCode=0 Oct 30 05:20:26.347177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.346733 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" event={"ID":"2ff795b5-eea0-47c6-acba-5aea88e9fea4","Type":"ContainerDied","Data":"98c167779fff28559d2cd8c8729bf50ea395c93b3b5ed7d4e58f502caf1b21dd"} Oct 30 05:20:26.347177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.346763 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" Oct 30 05:20:26.347177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.346782 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv" event={"ID":"2ff795b5-eea0-47c6-acba-5aea88e9fea4","Type":"ContainerDied","Data":"7feaff8fa5eae31d119d879e4cf7242dafb8b3db631f059818e8ffbbee45b76f"} Oct 30 05:20:26.347177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.346813 6587 scope.go:122] "RemoveContainer" containerID="98c167779fff28559d2cd8c8729bf50ea395c93b3b5ed7d4e58f502caf1b21dd" Oct 30 05:20:26.362523 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.362501 6587 scope.go:122] "RemoveContainer" containerID="d33226465478fffc519b132e3e3baa41f9ae6b147d6c9a5af221d3ab12b9824f" Oct 30 05:20:26.373567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.373542 6587 scope.go:122] "RemoveContainer" containerID="98c167779fff28559d2cd8c8729bf50ea395c93b3b5ed7d4e58f502caf1b21dd" Oct 30 05:20:26.373937 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:20:26.373910 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c167779fff28559d2cd8c8729bf50ea395c93b3b5ed7d4e58f502caf1b21dd\": container with ID starting with 98c167779fff28559d2cd8c8729bf50ea395c93b3b5ed7d4e58f502caf1b21dd not found: ID does not exist" containerID="98c167779fff28559d2cd8c8729bf50ea395c93b3b5ed7d4e58f502caf1b21dd" Oct 30 05:20:26.374039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.373948 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c167779fff28559d2cd8c8729bf50ea395c93b3b5ed7d4e58f502caf1b21dd"} err="failed to get container status \"98c167779fff28559d2cd8c8729bf50ea395c93b3b5ed7d4e58f502caf1b21dd\": rpc error: code = NotFound desc = could not find container \"98c167779fff28559d2cd8c8729bf50ea395c93b3b5ed7d4e58f502caf1b21dd\": container with ID starting with 98c167779fff28559d2cd8c8729bf50ea395c93b3b5ed7d4e58f502caf1b21dd not found: ID does not exist" Oct 30 05:20:26.374039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.373975 6587 scope.go:122] "RemoveContainer" containerID="d33226465478fffc519b132e3e3baa41f9ae6b147d6c9a5af221d3ab12b9824f" Oct 30 05:20:26.374426 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:20:26.374395 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d33226465478fffc519b132e3e3baa41f9ae6b147d6c9a5af221d3ab12b9824f\": container with ID starting with d33226465478fffc519b132e3e3baa41f9ae6b147d6c9a5af221d3ab12b9824f not found: ID does not exist" containerID="d33226465478fffc519b132e3e3baa41f9ae6b147d6c9a5af221d3ab12b9824f" Oct 30 05:20:26.374878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.374433 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d33226465478fffc519b132e3e3baa41f9ae6b147d6c9a5af221d3ab12b9824f"} err="failed to get container status \"d33226465478fffc519b132e3e3baa41f9ae6b147d6c9a5af221d3ab12b9824f\": rpc error: code = NotFound desc = could not find container \"d33226465478fffc519b132e3e3baa41f9ae6b147d6c9a5af221d3ab12b9824f\": container with ID starting with d33226465478fffc519b132e3e3baa41f9ae6b147d6c9a5af221d3ab12b9824f not found: ID does not exist" Oct 30 05:20:26.470218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.470182 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-data\") pod \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " Oct 30 05:20:26.470434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.470348 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-credential-socket\") pod \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " Oct 30 05:20:26.470530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.470485 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-envoy\") pod \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " Oct 30 05:20:26.470597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.470527 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-kube-api-access-t7d7m\" (UniqueName: \"kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-kube-api-access-t7d7m\") pod \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " Oct 30 05:20:26.470597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.470562 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-data" pod "2ff795b5-eea0-47c6-acba-5aea88e9fea4" (UID: "2ff795b5-eea0-47c6-acba-5aea88e9fea4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:20:26.470597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.470577 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istiod-ca-cert\") pod \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " Oct 30 05:20:26.470772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.470605 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-socket\") pod \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " Oct 30 05:20:26.470772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.470705 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-credential-socket" pod "2ff795b5-eea0-47c6-acba-5aea88e9fea4" (UID: "2ff795b5-eea0-47c6-acba-5aea88e9fea4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:20:26.470772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.470757 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-certs\") pod \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " Oct 30 05:20:26.470933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.470861 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-token\" (UniqueName: \"kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-token\") pod \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " Oct 30 05:20:26.470991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.470941 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-podinfo\") pod \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\" (UID: \"2ff795b5-eea0-47c6-acba-5aea88e9fea4\") " Oct 30 05:20:26.471056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.470996 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-socket" pod "2ff795b5-eea0-47c6-acba-5aea88e9fea4" (UID: "2ff795b5-eea0-47c6-acba-5aea88e9fea4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:20:26.471056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.471030 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istiod-ca-cert" pod "2ff795b5-eea0-47c6-acba-5aea88e9fea4" (UID: "2ff795b5-eea0-47c6-acba-5aea88e9fea4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:26.471176 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.471155 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-certs" pod "2ff795b5-eea0-47c6-acba-5aea88e9fea4" (UID: "2ff795b5-eea0-47c6-acba-5aea88e9fea4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:20:26.471800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.471774 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:26.471906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.471804 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:26.471906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.471818 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:26.471906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.471842 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:26.471906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.471855 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:26.473349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.473318 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-token" pod "2ff795b5-eea0-47c6-acba-5aea88e9fea4" (UID: "2ff795b5-eea0-47c6-acba-5aea88e9fea4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:26.473504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.473472 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-podinfo" pod "2ff795b5-eea0-47c6-acba-5aea88e9fea4" (UID: "2ff795b5-eea0-47c6-acba-5aea88e9fea4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:20:26.473723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.473701 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-envoy" pod "2ff795b5-eea0-47c6-acba-5aea88e9fea4" (UID: "2ff795b5-eea0-47c6-acba-5aea88e9fea4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:20:26.473849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.473833 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-kube-api-access-t7d7m" pod "2ff795b5-eea0-47c6-acba-5aea88e9fea4" (UID: "2ff795b5-eea0-47c6-acba-5aea88e9fea4"). InnerVolumeSpecName "kube-api-access-t7d7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:26.573579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.573550 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:26.573579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.573579 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7d7m\" (UniqueName: \"kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-kube-api-access-t7d7m\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:26.573579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.573592 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:26.573906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.573605 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff795b5-eea0-47c6-acba-5aea88e9fea4-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:26.701065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.701033 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv"] Oct 30 05:20:26.714649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:26.714622 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv"] Oct 30 05:20:27.044430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.044087 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s"] Oct 30 05:20:27.063476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.056486 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" Oct 30 05:20:27.063476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.056515 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" Oct 30 05:20:27.063476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.056570 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" Oct 30 05:20:27.063476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.056582 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" Oct 30 05:20:27.063476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.056648 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e0c31176-a56f-4b19-bf5d-287db8d23891" containerName="revoke-keys" Oct 30 05:20:27.063476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.056656 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c31176-a56f-4b19-bf5d-287db8d23891" containerName="revoke-keys" Oct 30 05:20:27.063476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.056860 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" Oct 30 05:20:27.063476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.056882 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" containerName="istio-proxy" Oct 30 05:20:27.063476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.056898 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c31176-a56f-4b19-bf5d-287db8d23891" containerName="revoke-keys" Oct 30 05:20:27.073759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.071538 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s"] Oct 30 05:20:27.073759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.071674 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.081962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.081929 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tls-certs\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.082095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.082002 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tmp-dir\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.082146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.082120 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-dshm\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.082210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.082175 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-home\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.082305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.082275 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-model-cache\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.082400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.082341 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvndc\" (UniqueName: \"kubernetes.io/projected/2719613c-cb60-4cc8-a36d-b5cdf835aca3-kube-api-access-kvndc\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.089178 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.089159 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-802b8a\"/\"default-dockercfg-pghfp\"" Oct 30 05:20:27.090468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.089992 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-802b8a\"/\"test-model-802b8a-kserve-self-signed-certs\"" Oct 30 05:20:27.090468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.090064 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-802b8a\"/\"kube-root-ca.crt\"" Oct 30 05:20:27.109839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.109811 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-802b8a\"/\"openshift-service-ca.crt\"" Oct 30 05:20:27.183623 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.183590 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tls-certs\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.183794 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.183694 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tmp-dir\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.183794 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.183763 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-dshm\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.183893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.183798 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-home\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.183956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.183913 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-model-cache\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.184062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.183972 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvndc\" (UniqueName: \"kubernetes.io/projected/2719613c-cb60-4cc8-a36d-b5cdf835aca3-kube-api-access-kvndc\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.184250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.184228 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-home\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.184372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.184331 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-model-cache\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.184444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.184349 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tmp-dir\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.186393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.186368 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tls-certs\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.186488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.186416 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-dshm\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.207038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.207011 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvndc\" (UniqueName: \"kubernetes.io/projected/2719613c-cb60-4cc8-a36d-b5cdf835aca3-kube-api-access-kvndc\") pod \"test-model-802b8a-kserve-5dcdbbcdf-hhm4s\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.397877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.397843 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:27.588259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.588225 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff795b5-eea0-47c6-acba-5aea88e9fea4" path="/var/lib/kubelet/pods/2ff795b5-eea0-47c6-acba-5aea88e9fea4/volumes" Oct 30 05:20:27.787007 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:20:27.786974 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2719613c_cb60_4cc8_a36d_b5cdf835aca3.slice/crio-b990aa82a75aab689e2666fffb2901c12f6eac0b39e29c4fee71053d68a88e2d WatchSource:0}: Error finding container b990aa82a75aab689e2666fffb2901c12f6eac0b39e29c4fee71053d68a88e2d: Status 404 returned error can't find the container with id b990aa82a75aab689e2666fffb2901c12f6eac0b39e29c4fee71053d68a88e2d Oct 30 05:20:27.788807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:27.788783 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s"] Oct 30 05:20:28.408591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:28.408543 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" event={"ID":"2719613c-cb60-4cc8-a36d-b5cdf835aca3","Type":"ContainerStarted","Data":"51495dc25b4cab73630d934cba4b4c28dad0d482d6e948eecb56cfecbfd812ff"} Oct 30 05:20:28.409066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:28.408592 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" event={"ID":"2719613c-cb60-4cc8-a36d-b5cdf835aca3","Type":"ContainerStarted","Data":"b990aa82a75aab689e2666fffb2901c12f6eac0b39e29c4fee71053d68a88e2d"} Oct 30 05:20:28.451954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:28.451892 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" podStartSLOduration=1.451877234 podStartE2EDuration="1.451877234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:20:28.451197541 +0000 UTC m=+2152.978964326" watchObservedRunningTime="2026-10-30 05:20:28.451877234 +0000 UTC m=+2152.979644022" Oct 30 05:20:28.741789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:28.741706 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" podUID="4012f48b-8a86-4077-9dd5-c6886b47a38d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:20:29.399633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:29.399602 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:31.569685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:31.569643 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn"] Oct 30 05:20:31.593048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:31.593013 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn"] Oct 30 05:20:31.986600 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:31.986517 6587 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:20:31.986600 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:31.986588 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:20:32.107281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.107244 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg"] Oct 30 05:20:32.148620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.148585 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg"] Oct 30 05:20:32.148818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.148697 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.166197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.166166 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-68b893\"/\"default-dockercfg-sthnn\"" Oct 30 05:20:32.166458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.166436 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-68b893\"/\"openshift-service-ca.crt\"" Oct 30 05:20:32.166906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.166884 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-68b893\"/\"kube-root-ca.crt\"" Oct 30 05:20:32.167054 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.167039 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-68b893\"/\"test-model-68b893-kserve-self-signed-certs\"" Oct 30 05:20:32.292847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.292764 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-home\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.292847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.292810 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-model-cache\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.294049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.292924 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-tmp-dir\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.294049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.293071 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-dshm\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.294049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.293179 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwq82\" (UniqueName: \"kubernetes.io/projected/427c21bb-df54-419c-a7da-25e015c0dec4-kube-api-access-xwq82\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.294049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.293207 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427c21bb-df54-419c-a7da-25e015c0dec4-tls-certs\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.401381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.394439 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-home\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.401381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.394503 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-model-cache\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.401381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.394596 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-tmp-dir\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.401381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.394747 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-dshm\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.401381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.394866 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwq82\" (UniqueName: \"kubernetes.io/projected/427c21bb-df54-419c-a7da-25e015c0dec4-kube-api-access-xwq82\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.401381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.394899 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427c21bb-df54-419c-a7da-25e015c0dec4-tls-certs\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.401381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.395821 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-tmp-dir\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.401381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.396079 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-home\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.401381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.396371 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-model-cache\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.416378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.408940 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc"] Oct 30 05:20:32.468423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.464612 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.473379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.471640 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427c21bb-df54-419c-a7da-25e015c0dec4-tls-certs\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.486397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.484670 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-dshm\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.491387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.489089 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwq82\" (UniqueName: \"kubernetes.io/projected/427c21bb-df54-419c-a7da-25e015c0dec4-kube-api-access-xwq82\") pod \"test-model-68b893-kserve-55f4f9cbc9-cxqdg\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.506007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.505977 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc"] Oct 30 05:20:32.607764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.606824 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.607764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.606908 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.607764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.606938 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.607764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.607000 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.607764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.607042 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.607764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.607151 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.607764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.607462 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.607764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.607511 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.607764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.607537 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zq5d\" (UniqueName: \"kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-kube-api-access-2zq5d\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.709610 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.709570 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.709827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.709630 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.709827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.709713 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.709972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.709915 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.709972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.709956 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.709972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.709972 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zq5d\" (UniqueName: \"kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-kube-api-access-2zq5d\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.710152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.710111 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.710259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.710236 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.710259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.710252 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.710497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.710313 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.710497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.710347 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.710627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.710563 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.710691 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.710661 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.711051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.711031 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.712884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.712861 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.713059 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.713041 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.731345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.731310 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.732217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.732197 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zq5d\" (UniqueName: \"kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-kube-api-access-2zq5d\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:32.781650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.781618 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:32.923999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:32.923927 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:33.103124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.103093 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg"] Oct 30 05:20:33.289578 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:20:33.289541 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8ac8d3_efa9_4d17_aca4_d5c44e7de484.slice/crio-d50191f105ae06c6386e1f1bd519137459b081efb9e8f47c4ee631cb84d6d0dd WatchSource:0}: Error finding container d50191f105ae06c6386e1f1bd519137459b081efb9e8f47c4ee631cb84d6d0dd: Status 404 returned error can't find the container with id d50191f105ae06c6386e1f1bd519137459b081efb9e8f47c4ee631cb84d6d0dd Oct 30 05:20:33.289766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.289742 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc"] Oct 30 05:20:33.292845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.292815 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:20:33.292938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.292876 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:20:33.292938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.292904 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:20:33.483155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.483118 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" event={"ID":"427c21bb-df54-419c-a7da-25e015c0dec4","Type":"ContainerStarted","Data":"b1588dd7e1bf85401a28516303c0f13cfcad3ca4da56d6026a7e645653a51b00"} Oct 30 05:20:33.483416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.483170 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" event={"ID":"427c21bb-df54-419c-a7da-25e015c0dec4","Type":"ContainerStarted","Data":"d4a99db857c73f22b1eb8731f7a21dc67384068f8b6a3b0a56aea630ed14dd0b"} Oct 30 05:20:33.485888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.485851 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" event={"ID":"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484","Type":"ContainerStarted","Data":"db841bef681b0ca4818facec32f08f8ddf866facb1ee76aae5adc30aea45ceb3"} Oct 30 05:20:33.486034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.485946 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" event={"ID":"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484","Type":"ContainerStarted","Data":"d50191f105ae06c6386e1f1bd519137459b081efb9e8f47c4ee631cb84d6d0dd"} Oct 30 05:20:33.488006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.487979 6587 generic.go:356] "Generic (PLEG): container finished" podID="040a272c-aca3-4af4-86db-3055cbe66486" containerID="0c00818e6ca0ce776a41115ffd356d70451429ff3d97332404c15ef5167596f9" exitCode=0 Oct 30 05:20:33.488115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.488047 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" event={"ID":"040a272c-aca3-4af4-86db-3055cbe66486","Type":"ContainerDied","Data":"0c00818e6ca0ce776a41115ffd356d70451429ff3d97332404c15ef5167596f9"} Oct 30 05:20:33.488478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.488461 6587 scope.go:122] "RemoveContainer" containerID="0c00818e6ca0ce776a41115ffd356d70451429ff3d97332404c15ef5167596f9" Oct 30 05:20:33.490492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.490459 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:20:33.490966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.490544 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:20:33.490966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.490588 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:20:33.534296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.534239 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" podStartSLOduration=1.534221853 podStartE2EDuration="1.534221853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:20:33.532746593 +0000 UTC m=+2158.060513380" watchObservedRunningTime="2026-10-30 05:20:33.534221853 +0000 UTC m=+2158.061988642" Oct 30 05:20:33.584252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.584199 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" podStartSLOduration=1.5841842640000001 podStartE2EDuration="1.584184264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:20:33.579249824 +0000 UTC m=+2158.107016610" watchObservedRunningTime="2026-10-30 05:20:33.584184264 +0000 UTC m=+2158.111951049" Oct 30 05:20:33.782766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.782707 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:33.924896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:33.924805 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:34.500755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:34.500714 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" event={"ID":"040a272c-aca3-4af4-86db-3055cbe66486","Type":"ContainerStarted","Data":"e17c36f726279343488d0bd32a10004ce2e5ced6a7d57f3a6fe0b1df5a0c4454"} Oct 30 05:20:34.942380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:34.928271 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 05:20:34.942380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:34.928335 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" podUID="ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 05:20:35.457382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:35.450739 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:20:35.457382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:35.450865 6587 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 05:20:35.457382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:35.450904 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 05:20:35.938946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:35.938909 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 05:20:35.940336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:35.940302 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" podUID="ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 05:20:36.492158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:36.492110 6587 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 05:20:36.496714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:36.496671 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 05:20:36.578496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:36.578450 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" podUID="c31b88a5-6d36-4588-8505-33ce9292c685" containerName="payload-processing" containerID="cri-o://b9c64a0237c2a1000b4a7aec847dcd8440fba0b99b6093d69f0a6fcc6e1e3de5" gracePeriod=25 Oct 30 05:20:36.594594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:36.594518 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" podUID="c4f882bc-ffd5-4288-bf40-388de69479bf" containerName="payload-pre-processing" containerID="cri-o://97fe5c925156c3c69681fc6f0ea5f07bfb3cc41749e17b183c6c6692363d3a8e" gracePeriod=25 Oct 30 05:20:37.000481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:36.996657 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 05:20:37.000666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.000619 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" podUID="ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 05:20:37.212625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.212596 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" Oct 30 05:20:37.313098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.313065 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31b88a5-6d36-4588-8505-33ce9292c685-kube-api-access-w4bk5\" (UniqueName: \"kubernetes.io/projected/c31b88a5-6d36-4588-8505-33ce9292c685-kube-api-access-w4bk5\") pod \"c31b88a5-6d36-4588-8505-33ce9292c685\" (UID: \"c31b88a5-6d36-4588-8505-33ce9292c685\") " Oct 30 05:20:37.313513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.313491 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31b88a5-6d36-4588-8505-33ce9292c685-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31b88a5-6d36-4588-8505-33ce9292c685-plugins-config-volume\") pod \"c31b88a5-6d36-4588-8505-33ce9292c685\" (UID: \"c31b88a5-6d36-4588-8505-33ce9292c685\") " Oct 30 05:20:37.313917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.313890 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31b88a5-6d36-4588-8505-33ce9292c685-plugins-config-volume" pod "c31b88a5-6d36-4588-8505-33ce9292c685" (UID: "c31b88a5-6d36-4588-8505-33ce9292c685"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:37.314310 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.314293 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31b88a5-6d36-4588-8505-33ce9292c685-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:37.345642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.344104 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31b88a5-6d36-4588-8505-33ce9292c685-kube-api-access-w4bk5" pod "c31b88a5-6d36-4588-8505-33ce9292c685" (UID: "c31b88a5-6d36-4588-8505-33ce9292c685"). InnerVolumeSpecName "kube-api-access-w4bk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:37.399588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.399511 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:37.413404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.413376 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:37.478382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.454673 6587 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 05:20:37.483533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.483491 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 05:20:37.484042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.416000 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4bk5\" (UniqueName: \"kubernetes.io/projected/c31b88a5-6d36-4588-8505-33ce9292c685-kube-api-access-w4bk5\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:37.487585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.485696 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9"] Oct 30 05:20:37.487585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.486980 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c31b88a5-6d36-4588-8505-33ce9292c685" containerName="payload-processing" Oct 30 05:20:37.487585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.487004 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31b88a5-6d36-4588-8505-33ce9292c685" containerName="payload-processing" Oct 30 05:20:37.487585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.487249 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31b88a5-6d36-4588-8505-33ce9292c685" containerName="payload-processing" Oct 30 05:20:37.490462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.490441 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.507483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.507259 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-356cae-openshift-default-dockercfg-ss9zj\"" Oct 30 05:20:37.509064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.508959 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9"] Oct 30 05:20:37.596883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.594715 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-certs\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.596883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.594828 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2ltn\" (UniqueName: \"kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-kube-api-access-t2ltn\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.596883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.594909 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-credential-socket\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.596883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.594938 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-data\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.596883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.594962 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd91a815-98b4-496a-885d-7a0b1d3e4369-istiod-ca-cert\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.596883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.595016 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-token\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.596883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.595128 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-envoy\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.596883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.595177 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-socket\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.596883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.595246 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-podinfo\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.697441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.696779 6587 generic.go:356] "Generic (PLEG): container finished" podID="c31b88a5-6d36-4588-8505-33ce9292c685" containerID="b9c64a0237c2a1000b4a7aec847dcd8440fba0b99b6093d69f0a6fcc6e1e3de5" exitCode=0 Oct 30 05:20:37.697441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.696880 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2ltn\" (UniqueName: \"kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-kube-api-access-t2ltn\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.697441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.697200 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" Oct 30 05:20:37.697441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.697259 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" event={"ID":"c31b88a5-6d36-4588-8505-33ce9292c685","Type":"ContainerDied","Data":"b9c64a0237c2a1000b4a7aec847dcd8440fba0b99b6093d69f0a6fcc6e1e3de5"} Oct 30 05:20:37.697441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.697302 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn" event={"ID":"c31b88a5-6d36-4588-8505-33ce9292c685","Type":"ContainerDied","Data":"1ef7a3953d9318cd68af044e1e619c8ab4b843ef127442a0412d0d0b335dd466"} Oct 30 05:20:37.697441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.697331 6587 scope.go:122] "RemoveContainer" containerID="b9c64a0237c2a1000b4a7aec847dcd8440fba0b99b6093d69f0a6fcc6e1e3de5" Oct 30 05:20:37.702005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.698318 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-credential-socket\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.702005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.698389 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-data\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.702005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.698416 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd91a815-98b4-496a-885d-7a0b1d3e4369-istiod-ca-cert\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.702005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.698475 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-token\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.702005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.698585 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-envoy\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.702005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.698654 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-socket\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.702005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.699088 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-credential-socket\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.702005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.699175 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-data\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.702005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.699453 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-socket\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.702005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.700061 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd91a815-98b4-496a-885d-7a0b1d3e4369-istiod-ca-cert\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.702005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.700523 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-podinfo\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.702005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.700707 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-certs\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.702005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.701041 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-certs\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.765421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.736268 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-envoy\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.765678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.748159 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-podinfo\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.790382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.789794 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:20:37.808023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.800531 6587 scope.go:122] "RemoveContainer" containerID="b9c64a0237c2a1000b4a7aec847dcd8440fba0b99b6093d69f0a6fcc6e1e3de5" Oct 30 05:20:37.808023 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:20:37.800871 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c64a0237c2a1000b4a7aec847dcd8440fba0b99b6093d69f0a6fcc6e1e3de5\": container with ID starting with b9c64a0237c2a1000b4a7aec847dcd8440fba0b99b6093d69f0a6fcc6e1e3de5 not found: ID does not exist" containerID="b9c64a0237c2a1000b4a7aec847dcd8440fba0b99b6093d69f0a6fcc6e1e3de5" Oct 30 05:20:37.808023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.800914 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c64a0237c2a1000b4a7aec847dcd8440fba0b99b6093d69f0a6fcc6e1e3de5"} err="failed to get container status \"b9c64a0237c2a1000b4a7aec847dcd8440fba0b99b6093d69f0a6fcc6e1e3de5\": rpc error: code = NotFound desc = could not find container \"b9c64a0237c2a1000b4a7aec847dcd8440fba0b99b6093d69f0a6fcc6e1e3de5\": container with ID starting with b9c64a0237c2a1000b4a7aec847dcd8440fba0b99b6093d69f0a6fcc6e1e3de5 not found: ID does not exist" Oct 30 05:20:37.808023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.803660 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-token\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.808023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.804548 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2ltn\" (UniqueName: \"kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-kube-api-access-t2ltn\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.808307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.808054 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:37.846637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.830426 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn"] Oct 30 05:20:37.900382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.897290 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn"] Oct 30 05:20:37.930278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.929858 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 05:20:37.930278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:37.929915 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" podUID="ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 05:20:38.449262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:38.448788 6587 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 05:20:38.449262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:38.448847 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 05:20:38.693052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:38.686581 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9"] Oct 30 05:20:38.710231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:38.709680 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:20:38.710231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:38.709773 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:20:38.710231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:38.709813 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:20:38.800578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:38.800534 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" podUID="4012f48b-8a86-4077-9dd5-c6886b47a38d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:20:38.939383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:38.939159 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 05:20:38.939383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:38.939233 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" podUID="ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 05:20:39.466535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:39.461500 6587 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 05:20:39.466535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:39.461566 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 05:20:39.653384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:39.630561 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31b88a5-6d36-4588-8505-33ce9292c685" path="/var/lib/kubelet/pods/c31b88a5-6d36-4588-8505-33ce9292c685/volumes" Oct 30 05:20:39.811888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:39.811804 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" event={"ID":"dd91a815-98b4-496a-885d-7a0b1d3e4369","Type":"ContainerStarted","Data":"ab98be8848449e092352989b59c94bea44699b414ba51761caf6b1fb3685801a"} Oct 30 05:20:39.811888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:39.811880 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" event={"ID":"dd91a815-98b4-496a-885d-7a0b1d3e4369","Type":"ContainerStarted","Data":"9b7f02897e8137bfd0c0919827468f72dc9dd49e83a7c79f6453d9bf14638d78"} Oct 30 05:20:39.884215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:39.884140 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" podStartSLOduration=2.884118747 podStartE2EDuration="2.884118747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:20:39.879877004 +0000 UTC m=+2164.407643792" watchObservedRunningTime="2026-10-30 05:20:39.884118747 +0000 UTC m=+2164.411885535" Oct 30 05:20:39.931261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:39.931213 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 05:20:39.931492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:39.931276 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" podUID="ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 05:20:40.383476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:40.383434 6587 scope.go:122] "RemoveContainer" containerID="cf29b333ab59bc423e9cb9dacd6b4c10fc25fe8cecc76a11ce75e9f4b467d397" Oct 30 05:20:40.406467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:40.406331 6587 scope.go:122] "RemoveContainer" containerID="78dd06b19f73cd4d87261952ef85d88d14a79f6898eb8fa9ab8af52849b341b3" Oct 30 05:20:40.469239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:40.469199 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:20:40.470934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:40.470782 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:20:40.472787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:40.472684 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:20:40.843539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:40.843449 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:40.859591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:40.859559 6587 patch_prober.go:29] interesting pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 05:20:40.859933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:40.859911 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" podUID="dd91a815-98b4-496a-885d-7a0b1d3e4369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 05:20:40.956409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:40.955802 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:40.957154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:40.957021 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:40.957715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:40.957691 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:20:41.808838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:41.808804 6587 patch_prober.go:29] interesting pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 05:20:41.809019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:41.808863 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" podUID="dd91a815-98b4-496a-885d-7a0b1d3e4369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 05:20:42.782269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:42.782237 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:42.794468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:42.794444 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:42.809051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:42.809023 6587 patch_prober.go:29] interesting pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 05:20:42.809190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:42.809069 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" podUID="dd91a815-98b4-496a-885d-7a0b1d3e4369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 05:20:42.901778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:42.901751 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:20:43.237381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:43.233447 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" podUID="c4f882bc-ffd5-4288-bf40-388de69479bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:20:43.809516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:43.809489 6587 patch_prober.go:29] interesting pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 05:20:43.810580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:43.809533 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" podUID="dd91a815-98b4-496a-885d-7a0b1d3e4369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 05:20:43.891017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:43.890984 6587 generic.go:356] "Generic (PLEG): container finished" podID="4012f48b-8a86-4077-9dd5-c6886b47a38d" containerID="c90512f68e5aa1ddf9db108c4e8307d3394487287bf859f6fcd679823bcfc21c" exitCode=0 Oct 30 05:20:43.891199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:43.891055 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" event={"ID":"4012f48b-8a86-4077-9dd5-c6886b47a38d","Type":"ContainerDied","Data":"c90512f68e5aa1ddf9db108c4e8307d3394487287bf859f6fcd679823bcfc21c"} Oct 30 05:20:44.093440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.093415 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" Oct 30 05:20:44.177325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.177290 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4012f48b-8a86-4077-9dd5-c6886b47a38d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4012f48b-8a86-4077-9dd5-c6886b47a38d-plugins-config-volume\") pod \"4012f48b-8a86-4077-9dd5-c6886b47a38d\" (UID: \"4012f48b-8a86-4077-9dd5-c6886b47a38d\") " Oct 30 05:20:44.177537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.177322 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4012f48b-8a86-4077-9dd5-c6886b47a38d-plugins-config-volume" pod "4012f48b-8a86-4077-9dd5-c6886b47a38d" (UID: "4012f48b-8a86-4077-9dd5-c6886b47a38d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:44.177537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.177525 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4012f48b-8a86-4077-9dd5-c6886b47a38d-kube-api-access-9667w\" (UniqueName: \"kubernetes.io/projected/4012f48b-8a86-4077-9dd5-c6886b47a38d-kube-api-access-9667w\") pod \"4012f48b-8a86-4077-9dd5-c6886b47a38d\" (UID: \"4012f48b-8a86-4077-9dd5-c6886b47a38d\") " Oct 30 05:20:44.178320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.178290 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4012f48b-8a86-4077-9dd5-c6886b47a38d-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:44.189551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.189519 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4012f48b-8a86-4077-9dd5-c6886b47a38d-kube-api-access-9667w" pod "4012f48b-8a86-4077-9dd5-c6886b47a38d" (UID: "4012f48b-8a86-4077-9dd5-c6886b47a38d"). InnerVolumeSpecName "kube-api-access-9667w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:44.279675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.279642 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9667w\" (UniqueName: \"kubernetes.io/projected/4012f48b-8a86-4077-9dd5-c6886b47a38d-kube-api-access-9667w\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:44.809197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.809164 6587 patch_prober.go:29] interesting pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 05:20:44.809389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.809237 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" podUID="dd91a815-98b4-496a-885d-7a0b1d3e4369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 05:20:44.901044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.901008 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" Oct 30 05:20:44.901044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.901023 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs" event={"ID":"4012f48b-8a86-4077-9dd5-c6886b47a38d","Type":"ContainerDied","Data":"e594cb2930ae5aa88ac2156ac09d8c8b7f6768e71e053056a5cced5b8d83ad0e"} Oct 30 05:20:44.901562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.901092 6587 scope.go:122] "RemoveContainer" containerID="c90512f68e5aa1ddf9db108c4e8307d3394487287bf859f6fcd679823bcfc21c" Oct 30 05:20:44.989967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.989938 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs"] Oct 30 05:20:44.996962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:44.996930 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs"] Oct 30 05:20:45.583957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:45.583929 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4012f48b-8a86-4077-9dd5-c6886b47a38d" path="/var/lib/kubelet/pods/4012f48b-8a86-4077-9dd5-c6886b47a38d/volumes" Oct 30 05:20:45.809478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:45.809445 6587 patch_prober.go:29] interesting pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 05:20:45.809665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:45.809496 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" podUID="dd91a815-98b4-496a-885d-7a0b1d3e4369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 05:20:46.814556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:46.814473 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:46.815032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:46.814956 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:46.815787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:46.815759 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:20:47.834927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:47.834894 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm"] Oct 30 05:20:47.835751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:47.835723 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4012f48b-8a86-4077-9dd5-c6886b47a38d" containerName="payload-pre-processing" Oct 30 05:20:47.835751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:47.835740 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4012f48b-8a86-4077-9dd5-c6886b47a38d" containerName="payload-pre-processing" Oct 30 05:20:47.835950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:47.835938 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4012f48b-8a86-4077-9dd5-c6886b47a38d" containerName="payload-pre-processing" Oct 30 05:20:47.839474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:47.839444 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" Oct 30 05:20:47.868029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:47.867998 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm"] Oct 30 05:20:47.870508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:47.870419 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:20:47.870508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:47.870504 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:20:47.885414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:47.883535 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:20:47.885414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:47.883630 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:20:47.946167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:47.946129 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5pms\" (UniqueName: \"kubernetes.io/projected/aa477b52-3248-43b7-811e-239378cf4438-kube-api-access-s5pms\") pod \"maas-api-revoke-sub-0b4453ce-2pplm\" (UID: \"aa477b52-3248-43b7-811e-239378cf4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" Oct 30 05:20:47.946388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:47.946354 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa477b52-3248-43b7-811e-239378cf4438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4453ce-2pplm\" (UID: \"aa477b52-3248-43b7-811e-239378cf4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" Oct 30 05:20:47.946452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:47.946429 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa477b52-3248-43b7-811e-239378cf4438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4453ce-2pplm\" (UID: \"aa477b52-3248-43b7-811e-239378cf4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" Oct 30 05:20:48.048001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:48.047954 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa477b52-3248-43b7-811e-239378cf4438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4453ce-2pplm\" (UID: \"aa477b52-3248-43b7-811e-239378cf4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" Oct 30 05:20:48.048187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:48.048031 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa477b52-3248-43b7-811e-239378cf4438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4453ce-2pplm\" (UID: \"aa477b52-3248-43b7-811e-239378cf4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" Oct 30 05:20:48.048331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:48.048307 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5pms\" (UniqueName: \"kubernetes.io/projected/aa477b52-3248-43b7-811e-239378cf4438-kube-api-access-s5pms\") pod \"maas-api-revoke-sub-0b4453ce-2pplm\" (UID: \"aa477b52-3248-43b7-811e-239378cf4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" Oct 30 05:20:48.048646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:48.048619 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa477b52-3248-43b7-811e-239378cf4438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4453ce-2pplm\" (UID: \"aa477b52-3248-43b7-811e-239378cf4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" Oct 30 05:20:48.050588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:48.050556 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa477b52-3248-43b7-811e-239378cf4438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4453ce-2pplm\" (UID: \"aa477b52-3248-43b7-811e-239378cf4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" Oct 30 05:20:48.068889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:48.068857 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5pms\" (UniqueName: \"kubernetes.io/projected/aa477b52-3248-43b7-811e-239378cf4438-kube-api-access-s5pms\") pod \"maas-api-revoke-sub-0b4453ce-2pplm\" (UID: \"aa477b52-3248-43b7-811e-239378cf4438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" Oct 30 05:20:48.160113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:48.160039 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" Oct 30 05:20:48.470199 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:20:48.470166 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa477b52_3248_43b7_811e_239378cf4438.slice/crio-57dfba5a2e5d7ada968417c7ab77c644ddb1aabb5b0d9519697b4cb8bdc86d5a WatchSource:0}: Error finding container 57dfba5a2e5d7ada968417c7ab77c644ddb1aabb5b0d9519697b4cb8bdc86d5a: Status 404 returned error can't find the container with id 57dfba5a2e5d7ada968417c7ab77c644ddb1aabb5b0d9519697b4cb8bdc86d5a Oct 30 05:20:48.470954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:48.470924 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm"] Oct 30 05:20:48.946765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:48.946722 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" event={"ID":"aa477b52-3248-43b7-811e-239378cf4438","Type":"ContainerStarted","Data":"f46e3426009c373d03045ff75bd8873a8c6a2a9e67bc72f4a40f1ed2ebf00ada"} Oct 30 05:20:48.947168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:48.946789 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" event={"ID":"aa477b52-3248-43b7-811e-239378cf4438","Type":"ContainerStarted","Data":"57dfba5a2e5d7ada968417c7ab77c644ddb1aabb5b0d9519697b4cb8bdc86d5a"} Oct 30 05:20:48.989131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:48.989038 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" podStartSLOduration=1.989024134 podStartE2EDuration="1.989024134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:20:48.986596717 +0000 UTC m=+2173.514363504" watchObservedRunningTime="2026-10-30 05:20:48.989024134 +0000 UTC m=+2173.516790942" Oct 30 05:20:49.957188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:49.957154 6587 generic.go:356] "Generic (PLEG): container finished" podID="aa477b52-3248-43b7-811e-239378cf4438" containerID="f46e3426009c373d03045ff75bd8873a8c6a2a9e67bc72f4a40f1ed2ebf00ada" exitCode=0 Oct 30 05:20:49.957188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:49.957201 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" event={"ID":"aa477b52-3248-43b7-811e-239378cf4438","Type":"ContainerDied","Data":"f46e3426009c373d03045ff75bd8873a8c6a2a9e67bc72f4a40f1ed2ebf00ada"} Oct 30 05:20:51.155140 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:51.155116 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" Oct 30 05:20:51.301087 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:51.301054 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa477b52-3248-43b7-811e-239378cf4438-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa477b52-3248-43b7-811e-239378cf4438-maas-api-xks-ca\") pod \"aa477b52-3248-43b7-811e-239378cf4438\" (UID: \"aa477b52-3248-43b7-811e-239378cf4438\") " Oct 30 05:20:51.301263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:51.301136 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa477b52-3248-43b7-811e-239378cf4438-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa477b52-3248-43b7-811e-239378cf4438-maas-api-openshift-ca\") pod \"aa477b52-3248-43b7-811e-239378cf4438\" (UID: \"aa477b52-3248-43b7-811e-239378cf4438\") " Oct 30 05:20:51.301405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:51.301318 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa477b52-3248-43b7-811e-239378cf4438-kube-api-access-s5pms\" (UniqueName: \"kubernetes.io/projected/aa477b52-3248-43b7-811e-239378cf4438-kube-api-access-s5pms\") pod \"aa477b52-3248-43b7-811e-239378cf4438\" (UID: \"aa477b52-3248-43b7-811e-239378cf4438\") " Oct 30 05:20:51.301592 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:51.301562 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa477b52-3248-43b7-811e-239378cf4438-maas-api-openshift-ca" pod "aa477b52-3248-43b7-811e-239378cf4438" (UID: "aa477b52-3248-43b7-811e-239378cf4438"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:51.302273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:51.302245 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa477b52-3248-43b7-811e-239378cf4438-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:51.303395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:51.303335 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa477b52-3248-43b7-811e-239378cf4438-maas-api-xks-ca" pod "aa477b52-3248-43b7-811e-239378cf4438" (UID: "aa477b52-3248-43b7-811e-239378cf4438"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:20:51.303395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:51.303384 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa477b52-3248-43b7-811e-239378cf4438-kube-api-access-s5pms" pod "aa477b52-3248-43b7-811e-239378cf4438" (UID: "aa477b52-3248-43b7-811e-239378cf4438"). InnerVolumeSpecName "kube-api-access-s5pms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:51.404213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:51.404174 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5pms\" (UniqueName: \"kubernetes.io/projected/aa477b52-3248-43b7-811e-239378cf4438-kube-api-access-s5pms\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:51.404213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:51.404214 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa477b52-3248-43b7-811e-239378cf4438-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:51.980157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:51.980068 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" event={"ID":"aa477b52-3248-43b7-811e-239378cf4438","Type":"ContainerDied","Data":"57dfba5a2e5d7ada968417c7ab77c644ddb1aabb5b0d9519697b4cb8bdc86d5a"} Oct 30 05:20:51.980157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:51.980104 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm" Oct 30 05:20:51.980157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:51.980127 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57dfba5a2e5d7ada968417c7ab77c644ddb1aabb5b0d9519697b4cb8bdc86d5a" Oct 30 05:20:53.220531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:53.220484 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" podUID="c4f882bc-ffd5-4288-bf40-388de69479bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:20:56.018109 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:20:56.018065 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:20:56.536566 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:56.536531 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5"] Oct 30 05:20:56.550104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:56.550076 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7f962-vsxp5"] Oct 30 05:20:56.850554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:56.850527 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" Oct 30 05:20:56.916017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:56.915764 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f882bc-ffd5-4288-bf40-388de69479bf-kube-api-access-4mgw4\" (UniqueName: \"kubernetes.io/projected/c4f882bc-ffd5-4288-bf40-388de69479bf-kube-api-access-4mgw4\") pod \"c4f882bc-ffd5-4288-bf40-388de69479bf\" (UID: \"c4f882bc-ffd5-4288-bf40-388de69479bf\") " Oct 30 05:20:56.916196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:56.916035 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f882bc-ffd5-4288-bf40-388de69479bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f882bc-ffd5-4288-bf40-388de69479bf-plugins-config-volume\") pod \"c4f882bc-ffd5-4288-bf40-388de69479bf\" (UID: \"c4f882bc-ffd5-4288-bf40-388de69479bf\") " Oct 30 05:20:56.916418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:56.916389 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f882bc-ffd5-4288-bf40-388de69479bf-plugins-config-volume" pod "c4f882bc-ffd5-4288-bf40-388de69479bf" (UID: "c4f882bc-ffd5-4288-bf40-388de69479bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:56.916728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:56.916704 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f882bc-ffd5-4288-bf40-388de69479bf-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:56.918748 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:56.918725 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f882bc-ffd5-4288-bf40-388de69479bf-kube-api-access-4mgw4" pod "c4f882bc-ffd5-4288-bf40-388de69479bf" (UID: "c4f882bc-ffd5-4288-bf40-388de69479bf"). InnerVolumeSpecName "kube-api-access-4mgw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:57.018582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:57.018545 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mgw4\" (UniqueName: \"kubernetes.io/projected/c4f882bc-ffd5-4288-bf40-388de69479bf-kube-api-access-4mgw4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:20:57.035174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:57.035140 6587 generic.go:356] "Generic (PLEG): container finished" podID="c4f882bc-ffd5-4288-bf40-388de69479bf" containerID="97fe5c925156c3c69681fc6f0ea5f07bfb3cc41749e17b183c6c6692363d3a8e" exitCode=0 Oct 30 05:20:57.035396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:57.035197 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" Oct 30 05:20:57.035396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:57.035225 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" event={"ID":"c4f882bc-ffd5-4288-bf40-388de69479bf","Type":"ContainerDied","Data":"97fe5c925156c3c69681fc6f0ea5f07bfb3cc41749e17b183c6c6692363d3a8e"} Oct 30 05:20:57.035396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:57.035295 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn" event={"ID":"c4f882bc-ffd5-4288-bf40-388de69479bf","Type":"ContainerDied","Data":"32a251c1f9f64e89a6b636380a3614a5409982ec81090cd85a994115b530fce8"} Oct 30 05:20:57.035396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:57.035323 6587 scope.go:122] "RemoveContainer" containerID="97fe5c925156c3c69681fc6f0ea5f07bfb3cc41749e17b183c6c6692363d3a8e" Oct 30 05:20:57.048440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:57.048421 6587 scope.go:122] "RemoveContainer" containerID="97fe5c925156c3c69681fc6f0ea5f07bfb3cc41749e17b183c6c6692363d3a8e" Oct 30 05:20:57.048763 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:20:57.048738 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97fe5c925156c3c69681fc6f0ea5f07bfb3cc41749e17b183c6c6692363d3a8e\": container with ID starting with 97fe5c925156c3c69681fc6f0ea5f07bfb3cc41749e17b183c6c6692363d3a8e not found: ID does not exist" containerID="97fe5c925156c3c69681fc6f0ea5f07bfb3cc41749e17b183c6c6692363d3a8e" Oct 30 05:20:57.048813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:57.048774 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97fe5c925156c3c69681fc6f0ea5f07bfb3cc41749e17b183c6c6692363d3a8e"} err="failed to get container status \"97fe5c925156c3c69681fc6f0ea5f07bfb3cc41749e17b183c6c6692363d3a8e\": rpc error: code = NotFound desc = could not find container \"97fe5c925156c3c69681fc6f0ea5f07bfb3cc41749e17b183c6c6692363d3a8e\": container with ID starting with 97fe5c925156c3c69681fc6f0ea5f07bfb3cc41749e17b183c6c6692363d3a8e not found: ID does not exist" Oct 30 05:20:57.096925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:57.096860 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn"] Oct 30 05:20:57.101013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:57.100984 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn"] Oct 30 05:20:57.585815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:57.585779 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a193d20e-a7b3-4470-ae3a-cefaa8f466ff" path="/var/lib/kubelet/pods/a193d20e-a7b3-4470-ae3a-cefaa8f466ff/volumes" Oct 30 05:20:57.586351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:57.586332 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f882bc-ffd5-4288-bf40-388de69479bf" path="/var/lib/kubelet/pods/c4f882bc-ffd5-4288-bf40-388de69479bf/volumes" Oct 30 05:20:59.259249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.259206 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59"] Oct 30 05:20:59.260038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.260021 6587 cpu_manager.go:414] "containerMap: removing container" podUID="aa477b52-3248-43b7-811e-239378cf4438" containerName="revoke-keys" Oct 30 05:20:59.260085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.260038 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa477b52-3248-43b7-811e-239378cf4438" containerName="revoke-keys" Oct 30 05:20:59.260085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.260072 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c4f882bc-ffd5-4288-bf40-388de69479bf" containerName="payload-pre-processing" Oct 30 05:20:59.260085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.260078 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f882bc-ffd5-4288-bf40-388de69479bf" containerName="payload-pre-processing" Oct 30 05:20:59.260229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.260220 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f882bc-ffd5-4288-bf40-388de69479bf" containerName="payload-pre-processing" Oct 30 05:20:59.260266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.260236 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa477b52-3248-43b7-811e-239378cf4438" containerName="revoke-keys" Oct 30 05:20:59.264419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.264393 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" Oct 30 05:20:59.281818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.281794 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59"] Oct 30 05:20:59.281984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.281878 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:20:59.290627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.290601 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:20:59.290808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.290610 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:20:59.290808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.290595 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:20:59.374510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.374475 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45870a0c-zxm59\" (UID: \"1d624690-2e68-45a4-9e66-529b4de0b3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" Oct 30 05:20:59.374701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.374529 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45870a0c-zxm59\" (UID: \"1d624690-2e68-45a4-9e66-529b4de0b3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" Oct 30 05:20:59.374701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.374673 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw472\" (UniqueName: \"kubernetes.io/projected/1d624690-2e68-45a4-9e66-529b4de0b3a2-kube-api-access-tw472\") pod \"maas-api-revoke-sub-45870a0c-zxm59\" (UID: \"1d624690-2e68-45a4-9e66-529b4de0b3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" Oct 30 05:20:59.476575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.476536 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw472\" (UniqueName: \"kubernetes.io/projected/1d624690-2e68-45a4-9e66-529b4de0b3a2-kube-api-access-tw472\") pod \"maas-api-revoke-sub-45870a0c-zxm59\" (UID: \"1d624690-2e68-45a4-9e66-529b4de0b3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" Oct 30 05:20:59.476786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.476754 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45870a0c-zxm59\" (UID: \"1d624690-2e68-45a4-9e66-529b4de0b3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" Oct 30 05:20:59.476879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.476806 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45870a0c-zxm59\" (UID: \"1d624690-2e68-45a4-9e66-529b4de0b3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" Oct 30 05:20:59.477672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.477643 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45870a0c-zxm59\" (UID: \"1d624690-2e68-45a4-9e66-529b4de0b3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" Oct 30 05:20:59.479201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.479178 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45870a0c-zxm59\" (UID: \"1d624690-2e68-45a4-9e66-529b4de0b3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" Oct 30 05:20:59.499619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.499596 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw472\" (UniqueName: \"kubernetes.io/projected/1d624690-2e68-45a4-9e66-529b4de0b3a2-kube-api-access-tw472\") pod \"maas-api-revoke-sub-45870a0c-zxm59\" (UID: \"1d624690-2e68-45a4-9e66-529b4de0b3a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" Oct 30 05:20:59.578833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.578802 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" Oct 30 05:20:59.890673 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:20:59.890633 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d624690_2e68_45a4_9e66_529b4de0b3a2.slice/crio-f9b9891dd214370a533e9ca8de9d8f49b7ea1f5445df9ff0175088e37803ba21 WatchSource:0}: Error finding container f9b9891dd214370a533e9ca8de9d8f49b7ea1f5445df9ff0175088e37803ba21: Status 404 returned error can't find the container with id f9b9891dd214370a533e9ca8de9d8f49b7ea1f5445df9ff0175088e37803ba21 Oct 30 05:20:59.893396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:20:59.891759 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59"] Oct 30 05:21:00.069636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:00.069599 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" event={"ID":"1d624690-2e68-45a4-9e66-529b4de0b3a2","Type":"ContainerStarted","Data":"f9b9891dd214370a533e9ca8de9d8f49b7ea1f5445df9ff0175088e37803ba21"} Oct 30 05:21:01.079386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:01.079334 6587 generic.go:356] "Generic (PLEG): container finished" podID="1d624690-2e68-45a4-9e66-529b4de0b3a2" containerID="d47242dbff5841b0cf5ec24013200e860871254dab009a6336fac13fee9e9db8" exitCode=0 Oct 30 05:21:01.079833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:01.079430 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" event={"ID":"1d624690-2e68-45a4-9e66-529b4de0b3a2","Type":"ContainerDied","Data":"d47242dbff5841b0cf5ec24013200e860871254dab009a6336fac13fee9e9db8"} Oct 30 05:21:02.283012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:02.282984 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" Oct 30 05:21:02.427515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:02.427417 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-openshift-ca\") pod \"1d624690-2e68-45a4-9e66-529b4de0b3a2\" (UID: \"1d624690-2e68-45a4-9e66-529b4de0b3a2\") " Oct 30 05:21:02.427747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:02.427649 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d624690-2e68-45a4-9e66-529b4de0b3a2-kube-api-access-tw472\" (UniqueName: \"kubernetes.io/projected/1d624690-2e68-45a4-9e66-529b4de0b3a2-kube-api-access-tw472\") pod \"1d624690-2e68-45a4-9e66-529b4de0b3a2\" (UID: \"1d624690-2e68-45a4-9e66-529b4de0b3a2\") " Oct 30 05:21:02.427747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:02.427710 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-xks-ca\") pod \"1d624690-2e68-45a4-9e66-529b4de0b3a2\" (UID: \"1d624690-2e68-45a4-9e66-529b4de0b3a2\") " Oct 30 05:21:02.427902 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:02.427809 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-openshift-ca" pod "1d624690-2e68-45a4-9e66-529b4de0b3a2" (UID: "1d624690-2e68-45a4-9e66-529b4de0b3a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:02.428536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:02.428512 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:02.430181 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:02.430155 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d624690-2e68-45a4-9e66-529b4de0b3a2-kube-api-access-tw472" pod "1d624690-2e68-45a4-9e66-529b4de0b3a2" (UID: "1d624690-2e68-45a4-9e66-529b4de0b3a2"). InnerVolumeSpecName "kube-api-access-tw472". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:02.430267 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:02.430242 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-xks-ca" pod "1d624690-2e68-45a4-9e66-529b4de0b3a2" (UID: "1d624690-2e68-45a4-9e66-529b4de0b3a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:21:02.530782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:02.530745 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw472\" (UniqueName: \"kubernetes.io/projected/1d624690-2e68-45a4-9e66-529b4de0b3a2-kube-api-access-tw472\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:02.530782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:02.530782 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d624690-2e68-45a4-9e66-529b4de0b3a2-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:03.099053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:03.099017 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" Oct 30 05:21:03.099053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:03.099031 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59" event={"ID":"1d624690-2e68-45a4-9e66-529b4de0b3a2","Type":"ContainerDied","Data":"f9b9891dd214370a533e9ca8de9d8f49b7ea1f5445df9ff0175088e37803ba21"} Oct 30 05:21:03.099294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:03.099087 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9b9891dd214370a533e9ca8de9d8f49b7ea1f5445df9ff0175088e37803ba21" Oct 30 05:21:04.092203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.092167 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx"] Oct 30 05:21:04.092985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.092968 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1d624690-2e68-45a4-9e66-529b4de0b3a2" containerName="revoke-keys" Oct 30 05:21:04.093040 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.092987 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d624690-2e68-45a4-9e66-529b4de0b3a2" containerName="revoke-keys" Oct 30 05:21:04.093158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.093149 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d624690-2e68-45a4-9e66-529b4de0b3a2" containerName="revoke-keys" Oct 30 05:21:04.097409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.097390 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" Oct 30 05:21:04.128628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.128598 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:21:04.128997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.128978 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:21:04.129296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.129222 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:21:04.139311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.139285 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:21:04.141496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.141472 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx"] Oct 30 05:21:04.188790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.188758 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-453511-csclx\" (UID: \"fb4d9567-0b72-431a-b33a-5d6570482975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" Oct 30 05:21:04.188935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.188799 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frj7q\" (UniqueName: \"kubernetes.io/projected/fb4d9567-0b72-431a-b33a-5d6570482975-kube-api-access-frj7q\") pod \"maas-api-revoke-keys-e2e-inf-b-453511-csclx\" (UID: \"fb4d9567-0b72-431a-b33a-5d6570482975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" Oct 30 05:21:04.188935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.188927 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-453511-csclx\" (UID: \"fb4d9567-0b72-431a-b33a-5d6570482975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" Oct 30 05:21:04.290103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.290074 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-453511-csclx\" (UID: \"fb4d9567-0b72-431a-b33a-5d6570482975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" Oct 30 05:21:04.290278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.290113 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frj7q\" (UniqueName: \"kubernetes.io/projected/fb4d9567-0b72-431a-b33a-5d6570482975-kube-api-access-frj7q\") pod \"maas-api-revoke-keys-e2e-inf-b-453511-csclx\" (UID: \"fb4d9567-0b72-431a-b33a-5d6570482975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" Oct 30 05:21:04.290278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.290178 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-453511-csclx\" (UID: \"fb4d9567-0b72-431a-b33a-5d6570482975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" Oct 30 05:21:04.290842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.290817 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-453511-csclx\" (UID: \"fb4d9567-0b72-431a-b33a-5d6570482975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" Oct 30 05:21:04.292856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.292833 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-453511-csclx\" (UID: \"fb4d9567-0b72-431a-b33a-5d6570482975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" Oct 30 05:21:04.312152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.312131 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frj7q\" (UniqueName: \"kubernetes.io/projected/fb4d9567-0b72-431a-b33a-5d6570482975-kube-api-access-frj7q\") pod \"maas-api-revoke-keys-e2e-inf-b-453511-csclx\" (UID: \"fb4d9567-0b72-431a-b33a-5d6570482975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" Oct 30 05:21:04.412535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.412462 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" Oct 30 05:21:04.725351 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:21:04.725316 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4d9567_0b72_431a_b33a_5d6570482975.slice/crio-7314f984b3b723c038cc73e66d45b73da4d0d44e650122b3d20105606cdac734 WatchSource:0}: Error finding container 7314f984b3b723c038cc73e66d45b73da4d0d44e650122b3d20105606cdac734: Status 404 returned error can't find the container with id 7314f984b3b723c038cc73e66d45b73da4d0d44e650122b3d20105606cdac734 Oct 30 05:21:04.725795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:04.725772 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx"] Oct 30 05:21:05.121147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:05.121113 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" event={"ID":"fb4d9567-0b72-431a-b33a-5d6570482975","Type":"ContainerStarted","Data":"856a0aa5777eca5db9fc5d51db6aa3000b617435d594b567b6505db9f393e112"} Oct 30 05:21:05.121624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:05.121167 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" event={"ID":"fb4d9567-0b72-431a-b33a-5d6570482975","Type":"ContainerStarted","Data":"7314f984b3b723c038cc73e66d45b73da4d0d44e650122b3d20105606cdac734"} Oct 30 05:21:05.163882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:05.163830 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" podStartSLOduration=1.163816907 podStartE2EDuration="1.163816907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:21:05.161849383 +0000 UTC m=+2189.689616170" watchObservedRunningTime="2026-10-30 05:21:05.163816907 +0000 UTC m=+2189.691583693" Oct 30 05:21:06.133307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:06.133204 6587 generic.go:356] "Generic (PLEG): container finished" podID="fb4d9567-0b72-431a-b33a-5d6570482975" containerID="856a0aa5777eca5db9fc5d51db6aa3000b617435d594b567b6505db9f393e112" exitCode=0 Oct 30 05:21:06.133813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:06.133324 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" event={"ID":"fb4d9567-0b72-431a-b33a-5d6570482975","Type":"ContainerDied","Data":"856a0aa5777eca5db9fc5d51db6aa3000b617435d594b567b6505db9f393e112"} Oct 30 05:21:07.364516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:07.364483 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" Oct 30 05:21:07.452065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:07.451974 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-openshift-ca\") pod \"fb4d9567-0b72-431a-b33a-5d6570482975\" (UID: \"fb4d9567-0b72-431a-b33a-5d6570482975\") " Oct 30 05:21:07.452615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:07.452586 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb4d9567-0b72-431a-b33a-5d6570482975-kube-api-access-frj7q\" (UniqueName: \"kubernetes.io/projected/fb4d9567-0b72-431a-b33a-5d6570482975-kube-api-access-frj7q\") pod \"fb4d9567-0b72-431a-b33a-5d6570482975\" (UID: \"fb4d9567-0b72-431a-b33a-5d6570482975\") " Oct 30 05:21:07.453001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:07.452979 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-xks-ca\") pod \"fb4d9567-0b72-431a-b33a-5d6570482975\" (UID: \"fb4d9567-0b72-431a-b33a-5d6570482975\") " Oct 30 05:21:07.472501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:07.472460 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-openshift-ca" pod "fb4d9567-0b72-431a-b33a-5d6570482975" (UID: "fb4d9567-0b72-431a-b33a-5d6570482975"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:07.499834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:07.498175 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4d9567-0b72-431a-b33a-5d6570482975-kube-api-access-frj7q" pod "fb4d9567-0b72-431a-b33a-5d6570482975" (UID: "fb4d9567-0b72-431a-b33a-5d6570482975"). InnerVolumeSpecName "kube-api-access-frj7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:07.499834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:07.499688 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-xks-ca" pod "fb4d9567-0b72-431a-b33a-5d6570482975" (UID: "fb4d9567-0b72-431a-b33a-5d6570482975"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:21:07.617440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:07.572601 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:07.617440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:07.593196 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d9567-0b72-431a-b33a-5d6570482975-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:07.617440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:07.593214 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frj7q\" (UniqueName: \"kubernetes.io/projected/fb4d9567-0b72-431a-b33a-5d6570482975-kube-api-access-frj7q\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:07.966194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:07.966161 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5"] Oct 30 05:21:07.970096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:07.970071 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8afb26-lxqz5"] Oct 30 05:21:08.159187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:08.159151 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" Oct 30 05:21:08.159187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:08.159167 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx" event={"ID":"fb4d9567-0b72-431a-b33a-5d6570482975","Type":"ContainerDied","Data":"7314f984b3b723c038cc73e66d45b73da4d0d44e650122b3d20105606cdac734"} Oct 30 05:21:08.159441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:08.159225 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7314f984b3b723c038cc73e66d45b73da4d0d44e650122b3d20105606cdac734" Oct 30 05:21:09.586556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:09.586524 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c31176-a56f-4b19-bf5d-287db8d23891" path="/var/lib/kubelet/pods/e0c31176-a56f-4b19-bf5d-287db8d23891/volumes" Oct 30 05:21:14.590890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:14.590856 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2"] Oct 30 05:21:14.625687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:14.625652 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t"] Oct 30 05:21:14.805200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:14.805168 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5"] Oct 30 05:21:14.806336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:14.806305 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" containerName="istio-proxy" containerID="cri-o://7006402bca16b9b843d06477ae2b4d75be437a1679a527d4e97862dcdc1fb39f" gracePeriod=30 Oct 30 05:21:16.232013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.231978 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll"] Oct 30 05:21:16.232928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.232907 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fb4d9567-0b72-431a-b33a-5d6570482975" containerName="revoke-keys" Oct 30 05:21:16.233000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.232930 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb4d9567-0b72-431a-b33a-5d6570482975" containerName="revoke-keys" Oct 30 05:21:16.233091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.233080 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb4d9567-0b72-431a-b33a-5d6570482975" containerName="revoke-keys" Oct 30 05:21:16.241085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.241052 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.266382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.258868 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll"] Oct 30 05:21:16.284675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.277596 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-262c10-openshift-default-dockercfg-gq5xt\"" Oct 30 05:21:16.284675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.279731 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-data\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.284675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.279798 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istiod-ca-cert\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.284675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.279831 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqrs9\" (UniqueName: \"kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-kube-api-access-cqrs9\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.284675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.279948 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-certs\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.284675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.279998 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.284675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.280080 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-token\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.284675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.280110 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-envoy\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.284675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.280184 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-podinfo\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.284675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.280243 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-credential-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.418382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.407522 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-podinfo\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.418382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.407640 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-credential-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.418382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.407699 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-data\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.418382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.407760 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istiod-ca-cert\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.418382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.407790 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqrs9\" (UniqueName: \"kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-kube-api-access-cqrs9\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.418382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.407896 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-certs\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.418382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.407965 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.418382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.408084 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-token\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.418382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.408121 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-envoy\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.433386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.426180 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istiod-ca-cert\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.433386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.430685 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-certs\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.433386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.431251 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.479381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.451421 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-credential-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.479381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.451965 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-data\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.551441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.544532 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqrs9\" (UniqueName: \"kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-kube-api-access-cqrs9\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.563876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.554347 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-token\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.564787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.564754 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-podinfo\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.585378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.574689 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-envoy\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.593279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.593025 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:16.966466 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:21:16.966051 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7f97f0_1587_4cd2_8d3d_eecb48dadc1c.slice/crio-0e06d44bc9d3bca65da2ac3fe20dc456e9fede54121d510a1638ad7ee565bc5a WatchSource:0}: Error finding container 0e06d44bc9d3bca65da2ac3fe20dc456e9fede54121d510a1638ad7ee565bc5a: Status 404 returned error can't find the container with id 0e06d44bc9d3bca65da2ac3fe20dc456e9fede54121d510a1638ad7ee565bc5a Oct 30 05:21:16.968032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.968006 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll"] Oct 30 05:21:16.970075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.970035 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:16.970189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.970101 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:16.970189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:16.970129 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:17.306391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:17.297638 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" event={"ID":"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c","Type":"ContainerStarted","Data":"e48d684c1e26930843b6b216d57b811792f6e1958c5fb2919f8a0756adcc34e0"} Oct 30 05:21:17.306391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:17.297712 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" event={"ID":"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c","Type":"ContainerStarted","Data":"0e06d44bc9d3bca65da2ac3fe20dc456e9fede54121d510a1638ad7ee565bc5a"} Oct 30 05:21:17.360384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:17.352326 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" podStartSLOduration=1.3523066510000001 podStartE2EDuration="1.352306651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:21:17.349840033 +0000 UTC m=+2201.877606819" watchObservedRunningTime="2026-10-30 05:21:17.352306651 +0000 UTC m=+2201.880073437" Oct 30 05:21:17.666203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:17.666122 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:18.606468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:18.606429 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:19.361036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:19.361003 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:19.362063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:19.362041 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:21:19.591661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:19.591625 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" podUID="e2e647db-b480-43b6-a65c-aaa6352cd512" containerName="payload-processing" containerID="cri-o://afba1ab5551d65485e135852974c5a427b9075f653b34bdbb2b54e53d02442e1" gracePeriod=25 Oct 30 05:21:19.629449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:19.626466 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" podUID="472f9495-f7da-43a3-a891-83e3b4894cdd" containerName="payload-pre-processing" containerID="cri-o://08470756e09a4ff18166c9f3f3ea996052d7710b13902c9b52049f304009e8b3" gracePeriod=25 Oct 30 05:21:20.111933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.111908 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:21:20.262737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.262646 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98061727-7ccf-40ba-88a2-1442bcb73855-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98061727-7ccf-40ba-88a2-1442bcb73855-istio-podinfo\") pod \"98061727-7ccf-40ba-88a2-1442bcb73855\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " Oct 30 05:21:20.262946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.262749 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98061727-7ccf-40ba-88a2-1442bcb73855-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98061727-7ccf-40ba-88a2-1442bcb73855-istiod-ca-cert\") pod \"98061727-7ccf-40ba-88a2-1442bcb73855\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " Oct 30 05:21:20.262946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.262806 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-data\") pod \"98061727-7ccf-40ba-88a2-1442bcb73855\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " Oct 30 05:21:20.262946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.262839 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-certs\") pod \"98061727-7ccf-40ba-88a2-1442bcb73855\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " Oct 30 05:21:20.262946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.262859 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-socket\") pod \"98061727-7ccf-40ba-88a2-1442bcb73855\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " Oct 30 05:21:20.262946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.262883 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-kube-api-access-5c8q9\" (UniqueName: \"kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-kube-api-access-5c8q9\") pod \"98061727-7ccf-40ba-88a2-1442bcb73855\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " Oct 30 05:21:20.262946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.262912 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-envoy\") pod \"98061727-7ccf-40ba-88a2-1442bcb73855\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " Oct 30 05:21:20.263186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.263027 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-credential-socket\") pod \"98061727-7ccf-40ba-88a2-1442bcb73855\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " Oct 30 05:21:20.263186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.263047 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-istio-token\" (UniqueName: \"kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-istio-token\") pod \"98061727-7ccf-40ba-88a2-1442bcb73855\" (UID: \"98061727-7ccf-40ba-88a2-1442bcb73855\") " Oct 30 05:21:20.264011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.263990 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-certs" pod "98061727-7ccf-40ba-88a2-1442bcb73855" (UID: "98061727-7ccf-40ba-88a2-1442bcb73855"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:20.264240 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.264212 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-data" pod "98061727-7ccf-40ba-88a2-1442bcb73855" (UID: "98061727-7ccf-40ba-88a2-1442bcb73855"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:20.264631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.264605 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98061727-7ccf-40ba-88a2-1442bcb73855-istiod-ca-cert" pod "98061727-7ccf-40ba-88a2-1442bcb73855" (UID: "98061727-7ccf-40ba-88a2-1442bcb73855"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:20.264954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.264878 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-socket" pod "98061727-7ccf-40ba-88a2-1442bcb73855" (UID: "98061727-7ccf-40ba-88a2-1442bcb73855"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:20.265286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.265267 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-credential-socket" pod "98061727-7ccf-40ba-88a2-1442bcb73855" (UID: "98061727-7ccf-40ba-88a2-1442bcb73855"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:20.266620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.266590 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-kube-api-access-5c8q9" pod "98061727-7ccf-40ba-88a2-1442bcb73855" (UID: "98061727-7ccf-40ba-88a2-1442bcb73855"). InnerVolumeSpecName "kube-api-access-5c8q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:20.267841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.267812 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-envoy" pod "98061727-7ccf-40ba-88a2-1442bcb73855" (UID: "98061727-7ccf-40ba-88a2-1442bcb73855"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:20.268348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.268323 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98061727-7ccf-40ba-88a2-1442bcb73855-istio-podinfo" pod "98061727-7ccf-40ba-88a2-1442bcb73855" (UID: "98061727-7ccf-40ba-88a2-1442bcb73855"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:21:20.269518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.269493 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-istio-token" pod "98061727-7ccf-40ba-88a2-1442bcb73855" (UID: "98061727-7ccf-40ba-88a2-1442bcb73855"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:20.371135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.371104 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.371322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.371311 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.371459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.371397 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98061727-7ccf-40ba-88a2-1442bcb73855-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.371459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.371414 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98061727-7ccf-40ba-88a2-1442bcb73855-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.371459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.371428 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.371459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.371440 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.371459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.371453 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.371721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.371469 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c8q9\" (UniqueName: \"kubernetes.io/projected/98061727-7ccf-40ba-88a2-1442bcb73855-kube-api-access-5c8q9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.371721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.371482 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98061727-7ccf-40ba-88a2-1442bcb73855-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:20.383305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.383272 6587 generic.go:356] "Generic (PLEG): container finished" podID="98061727-7ccf-40ba-88a2-1442bcb73855" containerID="7006402bca16b9b843d06477ae2b4d75be437a1679a527d4e97862dcdc1fb39f" exitCode=0 Oct 30 05:21:20.383498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.383331 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" Oct 30 05:21:20.383498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.383347 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" event={"ID":"98061727-7ccf-40ba-88a2-1442bcb73855","Type":"ContainerDied","Data":"7006402bca16b9b843d06477ae2b4d75be437a1679a527d4e97862dcdc1fb39f"} Oct 30 05:21:20.383498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.383473 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5" event={"ID":"98061727-7ccf-40ba-88a2-1442bcb73855","Type":"ContainerDied","Data":"6148b67b9d700638ba2a34e7f7eee7b74e1964ecfbe88251aff71d79e5818d38"} Oct 30 05:21:20.383670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.383505 6587 scope.go:122] "RemoveContainer" containerID="7006402bca16b9b843d06477ae2b4d75be437a1679a527d4e97862dcdc1fb39f" Oct 30 05:21:20.399289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.399266 6587 scope.go:122] "RemoveContainer" containerID="00ffca31366e48a3b1f02bcd15a884a7c26022813912795ba1f2a9f3d4004859" Oct 30 05:21:20.413538 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.413520 6587 scope.go:122] "RemoveContainer" containerID="7006402bca16b9b843d06477ae2b4d75be437a1679a527d4e97862dcdc1fb39f" Oct 30 05:21:20.413826 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:21:20.413799 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7006402bca16b9b843d06477ae2b4d75be437a1679a527d4e97862dcdc1fb39f\": container with ID starting with 7006402bca16b9b843d06477ae2b4d75be437a1679a527d4e97862dcdc1fb39f not found: ID does not exist" containerID="7006402bca16b9b843d06477ae2b4d75be437a1679a527d4e97862dcdc1fb39f" Oct 30 05:21:20.413919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.413836 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7006402bca16b9b843d06477ae2b4d75be437a1679a527d4e97862dcdc1fb39f"} err="failed to get container status \"7006402bca16b9b843d06477ae2b4d75be437a1679a527d4e97862dcdc1fb39f\": rpc error: code = NotFound desc = could not find container \"7006402bca16b9b843d06477ae2b4d75be437a1679a527d4e97862dcdc1fb39f\": container with ID starting with 7006402bca16b9b843d06477ae2b4d75be437a1679a527d4e97862dcdc1fb39f not found: ID does not exist" Oct 30 05:21:20.413919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.413856 6587 scope.go:122] "RemoveContainer" containerID="00ffca31366e48a3b1f02bcd15a884a7c26022813912795ba1f2a9f3d4004859" Oct 30 05:21:20.414139 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:21:20.414112 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ffca31366e48a3b1f02bcd15a884a7c26022813912795ba1f2a9f3d4004859\": container with ID starting with 00ffca31366e48a3b1f02bcd15a884a7c26022813912795ba1f2a9f3d4004859 not found: ID does not exist" containerID="00ffca31366e48a3b1f02bcd15a884a7c26022813912795ba1f2a9f3d4004859" Oct 30 05:21:20.414188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.414142 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ffca31366e48a3b1f02bcd15a884a7c26022813912795ba1f2a9f3d4004859"} err="failed to get container status \"00ffca31366e48a3b1f02bcd15a884a7c26022813912795ba1f2a9f3d4004859\": rpc error: code = NotFound desc = could not find container \"00ffca31366e48a3b1f02bcd15a884a7c26022813912795ba1f2a9f3d4004859\": container with ID starting with 00ffca31366e48a3b1f02bcd15a884a7c26022813912795ba1f2a9f3d4004859 not found: ID does not exist" Oct 30 05:21:20.455979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.455932 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5"] Oct 30 05:21:20.460261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:20.460230 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5"] Oct 30 05:21:21.194131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.194083 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" podUID="472f9495-f7da-43a3-a891-83e3b4894cdd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:21:21.585875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.585835 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" path="/var/lib/kubelet/pods/98061727-7ccf-40ba-88a2-1442bcb73855/volumes" Oct 30 05:21:21.894819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.894731 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm"] Oct 30 05:21:21.895689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.895665 6587 cpu_manager.go:414] "containerMap: removing container" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" containerName="istio-proxy" Oct 30 05:21:21.895689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.895687 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" containerName="istio-proxy" Oct 30 05:21:21.895875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.895708 6587 cpu_manager.go:414] "containerMap: removing container" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" containerName="istio-proxy" Oct 30 05:21:21.895875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.895716 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" containerName="istio-proxy" Oct 30 05:21:21.895998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.895944 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" containerName="istio-proxy" Oct 30 05:21:21.895998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.895961 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="98061727-7ccf-40ba-88a2-1442bcb73855" containerName="istio-proxy" Oct 30 05:21:21.902205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.902184 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" Oct 30 05:21:21.939786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.925882 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm"] Oct 30 05:21:21.945728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.941212 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-262c10\"" Oct 30 05:21:21.977234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.977194 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h"] Oct 30 05:21:21.982263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.982243 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" Oct 30 05:21:21.999465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.999438 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsp6z\" (UniqueName: \"kubernetes.io/projected/e360f2a8-ce2f-47ae-8e56-a265de776937-kube-api-access-hsp6z\") pod \"payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h\" (UID: \"e360f2a8-ce2f-47ae-8e56-a265de776937\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" Oct 30 05:21:21.999608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.999491 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5684\" (UniqueName: \"kubernetes.io/projected/00485499-909c-4e26-8206-edb1483727ce-kube-api-access-k5684\") pod \"payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm\" (UID: \"00485499-909c-4e26-8206-edb1483727ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" Oct 30 05:21:21.999686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:21.999664 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e360f2a8-ce2f-47ae-8e56-a265de776937-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h\" (UID: \"e360f2a8-ce2f-47ae-8e56-a265de776937\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" Oct 30 05:21:22.003401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.000440 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-262c10-dockercfg-c5f7t\"" Oct 30 05:21:22.003401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.000507 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h"] Oct 30 05:21:22.003401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.001519 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00485499-909c-4e26-8206-edb1483727ce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm\" (UID: \"00485499-909c-4e26-8206-edb1483727ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" Oct 30 05:21:22.347549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.304197 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" podUID="e2e647db-b480-43b6-a65c-aaa6352cd512" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:21:22.347549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.305504 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00485499-909c-4e26-8206-edb1483727ce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm\" (UID: \"00485499-909c-4e26-8206-edb1483727ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" Oct 30 05:21:22.347549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.305690 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsp6z\" (UniqueName: \"kubernetes.io/projected/e360f2a8-ce2f-47ae-8e56-a265de776937-kube-api-access-hsp6z\") pod \"payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h\" (UID: \"e360f2a8-ce2f-47ae-8e56-a265de776937\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" Oct 30 05:21:22.347549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.305749 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5684\" (UniqueName: \"kubernetes.io/projected/00485499-909c-4e26-8206-edb1483727ce-kube-api-access-k5684\") pod \"payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm\" (UID: \"00485499-909c-4e26-8206-edb1483727ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" Oct 30 05:21:22.347549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.305915 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e360f2a8-ce2f-47ae-8e56-a265de776937-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h\" (UID: \"e360f2a8-ce2f-47ae-8e56-a265de776937\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" Oct 30 05:21:22.347549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.307336 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e360f2a8-ce2f-47ae-8e56-a265de776937-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h\" (UID: \"e360f2a8-ce2f-47ae-8e56-a265de776937\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" Oct 30 05:21:22.347549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.308098 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00485499-909c-4e26-8206-edb1483727ce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm\" (UID: \"00485499-909c-4e26-8206-edb1483727ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" Oct 30 05:21:22.372115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.372077 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5684\" (UniqueName: \"kubernetes.io/projected/00485499-909c-4e26-8206-edb1483727ce-kube-api-access-k5684\") pod \"payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm\" (UID: \"00485499-909c-4e26-8206-edb1483727ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" Oct 30 05:21:22.376152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.376119 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsp6z\" (UniqueName: \"kubernetes.io/projected/e360f2a8-ce2f-47ae-8e56-a265de776937-kube-api-access-hsp6z\") pod \"payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h\" (UID: \"e360f2a8-ce2f-47ae-8e56-a265de776937\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" Oct 30 05:21:22.557845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.557784 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" Oct 30 05:21:22.599236 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.599151 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" Oct 30 05:21:22.913332 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:21:22.913296 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00485499_909c_4e26_8206_edb1483727ce.slice/crio-1a2144f37e0c6e455f4f3568b874164e7597cf43734a7712436628a15c19ffcb WatchSource:0}: Error finding container 1a2144f37e0c6e455f4f3568b874164e7597cf43734a7712436628a15c19ffcb: Status 404 returned error can't find the container with id 1a2144f37e0c6e455f4f3568b874164e7597cf43734a7712436628a15c19ffcb Oct 30 05:21:22.939497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:22.939461 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm"] Oct 30 05:21:23.000445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:23.000403 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h"] Oct 30 05:21:23.449540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:23.449306 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" event={"ID":"e360f2a8-ce2f-47ae-8e56-a265de776937","Type":"ContainerStarted","Data":"e218bdb67d883c7e2b607f467c91b8c9758236f1f6d2f52eb8a565733c8ef0cb"} Oct 30 05:21:23.449540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:23.449390 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" event={"ID":"e360f2a8-ce2f-47ae-8e56-a265de776937","Type":"ContainerStarted","Data":"52de7f34792e61e7e9f8ee23e0a9453a7d98c127ec0a59431f85e7f64dd7062b"} Oct 30 05:21:23.449540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:23.449434 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" Oct 30 05:21:23.452050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:23.452017 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" event={"ID":"00485499-909c-4e26-8206-edb1483727ce","Type":"ContainerStarted","Data":"fa141672bc6b8a6c589b34da7fd732178444548712641360a092a93b7e2b58a2"} Oct 30 05:21:23.452208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:23.452074 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" event={"ID":"00485499-909c-4e26-8206-edb1483727ce","Type":"ContainerStarted","Data":"1a2144f37e0c6e455f4f3568b874164e7597cf43734a7712436628a15c19ffcb"} Oct 30 05:21:23.452208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:23.452150 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" Oct 30 05:21:23.493854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:23.493806 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" podStartSLOduration=2.493793701 podStartE2EDuration="2.493793701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:21:23.491878132 +0000 UTC m=+2208.019644930" watchObservedRunningTime="2026-10-30 05:21:23.493793701 +0000 UTC m=+2208.021560486" Oct 30 05:21:25.987655 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:21:25.987604 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:21:28.508278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.508247 6587 generic.go:356] "Generic (PLEG): container finished" podID="405736d9-0831-4381-9707-a3392b6da104" containerID="99c1ab4fedcc6bb6f05ed8d2ba2610dc30192efaf7147393f45bef1634a1488f" exitCode=0 Oct 30 05:21:28.508841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.508278 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" event={"ID":"405736d9-0831-4381-9707-a3392b6da104","Type":"ContainerDied","Data":"99c1ab4fedcc6bb6f05ed8d2ba2610dc30192efaf7147393f45bef1634a1488f"} Oct 30 05:21:28.508841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.508734 6587 scope.go:122] "RemoveContainer" containerID="99c1ab4fedcc6bb6f05ed8d2ba2610dc30192efaf7147393f45bef1634a1488f" Oct 30 05:21:28.510586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.510561 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:28.510668 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.510628 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:28.510668 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.510659 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:28.574754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.574664 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" podStartSLOduration=7.574646063 podStartE2EDuration="7.574646063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:21:23.548321971 +0000 UTC m=+2208.076088759" watchObservedRunningTime="2026-10-30 05:21:28.574646063 +0000 UTC m=+2213.102412851" Oct 30 05:21:28.737430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.737202 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676"] Oct 30 05:21:28.973043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.972823 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g"] Oct 30 05:21:28.979536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.979514 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" Oct 30 05:21:28.993857 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.993346 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g"] Oct 30 05:21:28.998841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.998806 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:21:28.998990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.998967 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:21:28.999077 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.999049 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:21:28.999628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:28.999591 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:21:29.072576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:29.072543 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xk84\" (UniqueName: \"kubernetes.io/projected/4486d33f-de9c-4380-9a1e-c40afef5f1ec-kube-api-access-8xk84\") pod \"maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g\" (UID: \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" Oct 30 05:21:29.072757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:29.072593 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g\" (UID: \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" Oct 30 05:21:29.072757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:29.072727 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g\" (UID: \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" Oct 30 05:21:29.174419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:29.174344 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xk84\" (UniqueName: \"kubernetes.io/projected/4486d33f-de9c-4380-9a1e-c40afef5f1ec-kube-api-access-8xk84\") pod \"maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g\" (UID: \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" Oct 30 05:21:29.174626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:29.174453 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g\" (UID: \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" Oct 30 05:21:29.174626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:29.174552 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g\" (UID: \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" Oct 30 05:21:29.175203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:29.175178 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g\" (UID: \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" Oct 30 05:21:29.176907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:29.176887 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g\" (UID: \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" Oct 30 05:21:29.195921 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:29.195901 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xk84\" (UniqueName: \"kubernetes.io/projected/4486d33f-de9c-4380-9a1e-c40afef5f1ec-kube-api-access-8xk84\") pod \"maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g\" (UID: \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" Oct 30 05:21:29.301944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:29.301920 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" Oct 30 05:21:29.532778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:29.532731 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" event={"ID":"405736d9-0831-4381-9707-a3392b6da104","Type":"ContainerStarted","Data":"1a23fbc2702a8976a4937d6236e8990f7078ac09e4c41d81a554ceca33b8c8ac"} Oct 30 05:21:29.659653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:29.659622 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g"] Oct 30 05:21:30.365123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.365087 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn"] Oct 30 05:21:30.370562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.370537 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.389642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.389563 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-0dba31-openshift-default-dockercfg-9kc5b\"" Oct 30 05:21:30.409399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.409352 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn"] Oct 30 05:21:30.411527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.411500 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:21:30.411690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.411509 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 05:21:30.411690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.411592 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" podUID="405736d9-0831-4381-9707-a3392b6da104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 05:21:30.503763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.503729 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-podinfo\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.503946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.503783 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-data\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.503993 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.503946 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-envoy\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.504035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.504020 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.504072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.504062 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74mz4\" (UniqueName: \"kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-kube-api-access-74mz4\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.504110 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.504096 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-certs\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.504145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.504137 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-credential-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.504224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.504211 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-token\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.504261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.504238 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istiod-ca-cert\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.562489 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.562451 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" event={"ID":"4486d33f-de9c-4380-9a1e-c40afef5f1ec","Type":"ContainerStarted","Data":"b056bee6225a6c1f6ea45e4f07007f37ef6d4dd10f835fb6c7ab86e523926a3f"} Oct 30 05:21:30.562883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.562508 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" event={"ID":"4486d33f-de9c-4380-9a1e-c40afef5f1ec","Type":"ContainerStarted","Data":"bf9958ea2a09fd78e1f2e4eb96eafc3971c6f9cc5458c28bf8869aa9cb847286"} Oct 30 05:21:30.609456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.609404 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-envoy\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.609846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.609806 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.610051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.610031 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74mz4\" (UniqueName: \"kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-kube-api-access-74mz4\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.610249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.610232 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-certs\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.610469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.610443 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-credential-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.610734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.610713 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-token\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.610890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.610874 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istiod-ca-cert\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.611073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.611056 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-podinfo\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.611212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.611196 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-data\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.612937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.612915 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-credential-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.613556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.613535 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.614666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.614633 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-certs\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.633689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.616480 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istiod-ca-cert\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.633928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.623232 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-data\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.657975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.657931 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-token\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.678682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.678647 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-envoy\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.695980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.688669 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" podStartSLOduration=2.688648339 podStartE2EDuration="2.688648339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:21:30.652244529 +0000 UTC m=+2215.180011316" watchObservedRunningTime="2026-10-30 05:21:30.688648339 +0000 UTC m=+2215.216415125" Oct 30 05:21:30.700727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.700635 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-podinfo\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.709380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.702468 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74mz4\" (UniqueName: \"kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-kube-api-access-74mz4\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:30.992467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:30.992391 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:31.198935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:31.198890 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" podUID="472f9495-f7da-43a3-a891-83e3b4894cdd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:21:31.352926 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:21:31.352893 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7930f4_1e78_4bd5_be93_010fe7e02d20.slice/crio-6d420269333e6fb2003d1260b1beef054b740576da554d62bb853dd87e3505c0 WatchSource:0}: Error finding container 6d420269333e6fb2003d1260b1beef054b740576da554d62bb853dd87e3505c0: Status 404 returned error can't find the container with id 6d420269333e6fb2003d1260b1beef054b740576da554d62bb853dd87e3505c0 Oct 30 05:21:31.355971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:31.355944 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn"] Oct 30 05:21:31.357097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:31.357059 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:31.357190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:31.357146 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:31.357270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:31.357188 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:21:31.412119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:31.412096 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 05:21:31.412232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:31.412143 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" podUID="405736d9-0831-4381-9707-a3392b6da104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 05:21:31.588670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:31.588635 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" event={"ID":"2e7930f4-1e78-4bd5-be93-010fe7e02d20","Type":"ContainerStarted","Data":"250dc0b7b4d0e922ebc9cd1e5dac86cf9d68e5cc75841625e541237809aca257"} Oct 30 05:21:31.589203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:31.588683 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" event={"ID":"2e7930f4-1e78-4bd5-be93-010fe7e02d20","Type":"ContainerStarted","Data":"6d420269333e6fb2003d1260b1beef054b740576da554d62bb853dd87e3505c0"} Oct 30 05:21:31.591016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:31.590976 6587 generic.go:356] "Generic (PLEG): container finished" podID="4486d33f-de9c-4380-9a1e-c40afef5f1ec" containerID="b056bee6225a6c1f6ea45e4f07007f37ef6d4dd10f835fb6c7ab86e523926a3f" exitCode=0 Oct 30 05:21:31.591231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:31.591213 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" event={"ID":"4486d33f-de9c-4380-9a1e-c40afef5f1ec","Type":"ContainerDied","Data":"b056bee6225a6c1f6ea45e4f07007f37ef6d4dd10f835fb6c7ab86e523926a3f"} Oct 30 05:21:31.659094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:31.658073 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" podStartSLOduration=1.6580506019999999 podStartE2EDuration="1.658050602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:21:31.641667297 +0000 UTC m=+2216.169434097" watchObservedRunningTime="2026-10-30 05:21:31.658050602 +0000 UTC m=+2216.185817389" Oct 30 05:21:31.992740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:31.992656 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:32.201531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:32.201499 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" podUID="e2e647db-b480-43b6-a65c-aaa6352cd512" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:21:32.412169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:32.412133 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 05:21:32.412429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:32.412191 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" podUID="405736d9-0831-4381-9707-a3392b6da104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 05:21:32.865122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:32.865090 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" Oct 30 05:21:32.960643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:32.960553 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-xks-ca\") pod \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\" (UID: \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\") " Oct 30 05:21:32.960824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:32.960715 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-openshift-ca\") pod \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\" (UID: \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\") " Oct 30 05:21:32.960891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:32.960824 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4486d33f-de9c-4380-9a1e-c40afef5f1ec-kube-api-access-8xk84\" (UniqueName: \"kubernetes.io/projected/4486d33f-de9c-4380-9a1e-c40afef5f1ec-kube-api-access-8xk84\") pod \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\" (UID: \"4486d33f-de9c-4380-9a1e-c40afef5f1ec\") " Oct 30 05:21:32.961287 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:32.961253 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-openshift-ca" pod "4486d33f-de9c-4380-9a1e-c40afef5f1ec" (UID: "4486d33f-de9c-4380-9a1e-c40afef5f1ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:32.961659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:32.961634 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:32.963418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:32.963349 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4486d33f-de9c-4380-9a1e-c40afef5f1ec-kube-api-access-8xk84" pod "4486d33f-de9c-4380-9a1e-c40afef5f1ec" (UID: "4486d33f-de9c-4380-9a1e-c40afef5f1ec"). InnerVolumeSpecName "kube-api-access-8xk84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:32.964651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:32.964627 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-xks-ca" pod "4486d33f-de9c-4380-9a1e-c40afef5f1ec" (UID: "4486d33f-de9c-4380-9a1e-c40afef5f1ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:21:32.999308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:32.999281 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:33.063160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:33.063122 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4486d33f-de9c-4380-9a1e-c40afef5f1ec-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:33.063160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:33.063158 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xk84\" (UniqueName: \"kubernetes.io/projected/4486d33f-de9c-4380-9a1e-c40afef5f1ec-kube-api-access-8xk84\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:33.411987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:33.411954 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 05:21:33.412166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:33.412003 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" podUID="405736d9-0831-4381-9707-a3392b6da104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 05:21:33.648192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:33.642152 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" Oct 30 05:21:33.648192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:33.642713 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g" event={"ID":"4486d33f-de9c-4380-9a1e-c40afef5f1ec","Type":"ContainerDied","Data":"bf9958ea2a09fd78e1f2e4eb96eafc3971c6f9cc5458c28bf8869aa9cb847286"} Oct 30 05:21:33.648192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:33.642765 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf9958ea2a09fd78e1f2e4eb96eafc3971c6f9cc5458c28bf8869aa9cb847286" Oct 30 05:21:33.648192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:33.642788 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:33.648192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:33.647982 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:21:34.412089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:34.412055 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 05:21:34.412516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:34.412117 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" podUID="405736d9-0831-4381-9707-a3392b6da104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 05:21:34.472437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:34.472408 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" Oct 30 05:21:34.472983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:34.472963 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" Oct 30 05:21:35.417328 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:35.417298 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:21:35.417940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:35.417821 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:21:35.418654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:35.418635 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:21:36.244198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.244159 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv"] Oct 30 05:21:36.245103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.245080 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4486d33f-de9c-4380-9a1e-c40afef5f1ec" containerName="revoke-keys" Oct 30 05:21:36.245103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.245102 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4486d33f-de9c-4380-9a1e-c40afef5f1ec" containerName="revoke-keys" Oct 30 05:21:36.245320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.245304 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4486d33f-de9c-4380-9a1e-c40afef5f1ec" containerName="revoke-keys" Oct 30 05:21:36.250616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.250592 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" Oct 30 05:21:36.269168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.269124 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-0dba31-dockercfg-4ssqv\"" Oct 30 05:21:36.270133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.270107 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv"] Oct 30 05:21:36.292961 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.292928 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-0dba31\"" Oct 30 05:21:36.507444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.506941 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpzfk\" (UniqueName: \"kubernetes.io/projected/38da181c-3cfa-48da-b727-08064c3d57fe-kube-api-access-rpzfk\") pod \"payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv\" (UID: \"38da181c-3cfa-48da-b727-08064c3d57fe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" Oct 30 05:21:36.507444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.507055 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38da181c-3cfa-48da-b727-08064c3d57fe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv\" (UID: \"38da181c-3cfa-48da-b727-08064c3d57fe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" Oct 30 05:21:36.609002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.608964 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpzfk\" (UniqueName: \"kubernetes.io/projected/38da181c-3cfa-48da-b727-08064c3d57fe-kube-api-access-rpzfk\") pod \"payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv\" (UID: \"38da181c-3cfa-48da-b727-08064c3d57fe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" Oct 30 05:21:36.609204 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.609050 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38da181c-3cfa-48da-b727-08064c3d57fe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv\" (UID: \"38da181c-3cfa-48da-b727-08064c3d57fe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" Oct 30 05:21:36.609786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.609765 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38da181c-3cfa-48da-b727-08064c3d57fe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv\" (UID: \"38da181c-3cfa-48da-b727-08064c3d57fe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" Oct 30 05:21:36.631699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.631666 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpzfk\" (UniqueName: \"kubernetes.io/projected/38da181c-3cfa-48da-b727-08064c3d57fe-kube-api-access-rpzfk\") pod \"payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv\" (UID: \"38da181c-3cfa-48da-b727-08064c3d57fe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" Oct 30 05:21:36.869998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:36.869967 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" Oct 30 05:21:37.180652 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:21:37.180167 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38da181c_3cfa_48da_b727_08064c3d57fe.slice/crio-914ce144dfa2de8ef6f6ac8685e6994fb548b53c45e4eaa4f44a585beba6034c WatchSource:0}: Error finding container 914ce144dfa2de8ef6f6ac8685e6994fb548b53c45e4eaa4f44a585beba6034c: Status 404 returned error can't find the container with id 914ce144dfa2de8ef6f6ac8685e6994fb548b53c45e4eaa4f44a585beba6034c Oct 30 05:21:37.185310 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:37.184672 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv"] Oct 30 05:21:37.695291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:37.695244 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" event={"ID":"38da181c-3cfa-48da-b727-08064c3d57fe","Type":"ContainerStarted","Data":"b6bd2e82274909902811212bc020634f12a2f9fe6574ad117156183873129f7e"} Oct 30 05:21:37.695776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:37.695318 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" event={"ID":"38da181c-3cfa-48da-b727-08064c3d57fe","Type":"ContainerStarted","Data":"914ce144dfa2de8ef6f6ac8685e6994fb548b53c45e4eaa4f44a585beba6034c"} Oct 30 05:21:37.695776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:37.695343 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" Oct 30 05:21:37.736919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:37.736852 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" podStartSLOduration=1.736835074 podStartE2EDuration="1.736835074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:21:37.735652335 +0000 UTC m=+2222.263419117" watchObservedRunningTime="2026-10-30 05:21:37.736835074 +0000 UTC m=+2222.264601861" Oct 30 05:21:39.760892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:39.760337 6587 generic.go:356] "Generic (PLEG): container finished" podID="472f9495-f7da-43a3-a891-83e3b4894cdd" containerID="08470756e09a4ff18166c9f3f3ea996052d7710b13902c9b52049f304009e8b3" exitCode=0 Oct 30 05:21:39.760892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:39.760784 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" event={"ID":"472f9495-f7da-43a3-a891-83e3b4894cdd","Type":"ContainerDied","Data":"08470756e09a4ff18166c9f3f3ea996052d7710b13902c9b52049f304009e8b3"} Oct 30 05:21:39.764650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:39.764427 6587 generic.go:356] "Generic (PLEG): container finished" podID="e2e647db-b480-43b6-a65c-aaa6352cd512" containerID="afba1ab5551d65485e135852974c5a427b9075f653b34bdbb2b54e53d02442e1" exitCode=0 Oct 30 05:21:39.764650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:39.764586 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" event={"ID":"e2e647db-b480-43b6-a65c-aaa6352cd512","Type":"ContainerDied","Data":"afba1ab5551d65485e135852974c5a427b9075f653b34bdbb2b54e53d02442e1"} Oct 30 05:21:39.961830 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:39.960458 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" Oct 30 05:21:40.008866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.008837 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" Oct 30 05:21:40.079236 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.079201 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e647db-b480-43b6-a65c-aaa6352cd512-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e647db-b480-43b6-a65c-aaa6352cd512-plugins-config-volume\") pod \"e2e647db-b480-43b6-a65c-aaa6352cd512\" (UID: \"e2e647db-b480-43b6-a65c-aaa6352cd512\") " Oct 30 05:21:40.079446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.079282 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472f9495-f7da-43a3-a891-83e3b4894cdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472f9495-f7da-43a3-a891-83e3b4894cdd-plugins-config-volume\") pod \"472f9495-f7da-43a3-a891-83e3b4894cdd\" (UID: \"472f9495-f7da-43a3-a891-83e3b4894cdd\") " Oct 30 05:21:40.079548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.079477 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472f9495-f7da-43a3-a891-83e3b4894cdd-kube-api-access-c88hk\" (UniqueName: \"kubernetes.io/projected/472f9495-f7da-43a3-a891-83e3b4894cdd-kube-api-access-c88hk\") pod \"472f9495-f7da-43a3-a891-83e3b4894cdd\" (UID: \"472f9495-f7da-43a3-a891-83e3b4894cdd\") " Oct 30 05:21:40.079548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.079523 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e647db-b480-43b6-a65c-aaa6352cd512-kube-api-access-t5f2f\" (UniqueName: \"kubernetes.io/projected/e2e647db-b480-43b6-a65c-aaa6352cd512-kube-api-access-t5f2f\") pod \"e2e647db-b480-43b6-a65c-aaa6352cd512\" (UID: \"e2e647db-b480-43b6-a65c-aaa6352cd512\") " Oct 30 05:21:40.079670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.079650 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e647db-b480-43b6-a65c-aaa6352cd512-plugins-config-volume" pod "e2e647db-b480-43b6-a65c-aaa6352cd512" (UID: "e2e647db-b480-43b6-a65c-aaa6352cd512"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:40.079723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.079655 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472f9495-f7da-43a3-a891-83e3b4894cdd-plugins-config-volume" pod "472f9495-f7da-43a3-a891-83e3b4894cdd" (UID: "472f9495-f7da-43a3-a891-83e3b4894cdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:40.080102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.080080 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e647db-b480-43b6-a65c-aaa6352cd512-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:40.080102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.080099 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472f9495-f7da-43a3-a891-83e3b4894cdd-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:40.081781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.081751 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472f9495-f7da-43a3-a891-83e3b4894cdd-kube-api-access-c88hk" pod "472f9495-f7da-43a3-a891-83e3b4894cdd" (UID: "472f9495-f7da-43a3-a891-83e3b4894cdd"). InnerVolumeSpecName "kube-api-access-c88hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:40.081887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.081836 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e647db-b480-43b6-a65c-aaa6352cd512-kube-api-access-t5f2f" pod "e2e647db-b480-43b6-a65c-aaa6352cd512" (UID: "e2e647db-b480-43b6-a65c-aaa6352cd512"). InnerVolumeSpecName "kube-api-access-t5f2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:40.280495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.186779 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c88hk\" (UniqueName: \"kubernetes.io/projected/472f9495-f7da-43a3-a891-83e3b4894cdd-kube-api-access-c88hk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:40.280495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.186819 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5f2f\" (UniqueName: \"kubernetes.io/projected/e2e647db-b480-43b6-a65c-aaa6352cd512-kube-api-access-t5f2f\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:40.515212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.515179 6587 scope.go:122] "RemoveContainer" containerID="2d7dd1708db6c9c4203683c521d15a84503428695ad10257051c886d6eb52089" Oct 30 05:21:40.529603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.529572 6587 scope.go:122] "RemoveContainer" containerID="08470756e09a4ff18166c9f3f3ea996052d7710b13902c9b52049f304009e8b3" Oct 30 05:21:40.540653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.540631 6587 scope.go:122] "RemoveContainer" containerID="afba1ab5551d65485e135852974c5a427b9075f653b34bdbb2b54e53d02442e1" Oct 30 05:21:40.772489 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.772450 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" event={"ID":"472f9495-f7da-43a3-a891-83e3b4894cdd","Type":"ContainerDied","Data":"7077de0185ed2484ff782909c49e532eda224bad44fe46d633c0a14a57457ac3"} Oct 30 05:21:40.772967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.772595 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" event={"ID":"e2e647db-b480-43b6-a65c-aaa6352cd512","Type":"ContainerDied","Data":"34a0efdc41b6ed6942d89276c56afd32987dc2d601976d7eaddb474333667805"} Oct 30 05:21:40.772967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.772655 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2" Oct 30 05:21:40.772967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.772659 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t" Oct 30 05:21:40.894408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.894339 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t"] Oct 30 05:21:40.898626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.898600 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t"] Oct 30 05:21:40.964767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.964325 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2"] Oct 30 05:21:40.975763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:40.975725 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2"] Oct 30 05:21:41.588241 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:41.588210 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472f9495-f7da-43a3-a891-83e3b4894cdd" path="/var/lib/kubelet/pods/472f9495-f7da-43a3-a891-83e3b4894cdd/volumes" Oct 30 05:21:41.588788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:41.588772 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e647db-b480-43b6-a65c-aaa6352cd512" path="/var/lib/kubelet/pods/e2e647db-b480-43b6-a65c-aaa6352cd512/volumes" Oct 30 05:21:42.748659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.748606 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp"] Oct 30 05:21:42.749488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.749465 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e2e647db-b480-43b6-a65c-aaa6352cd512" containerName="payload-processing" Oct 30 05:21:42.749488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.749489 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e647db-b480-43b6-a65c-aaa6352cd512" containerName="payload-processing" Oct 30 05:21:42.749662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.749546 6587 cpu_manager.go:414] "containerMap: removing container" podUID="472f9495-f7da-43a3-a891-83e3b4894cdd" containerName="payload-pre-processing" Oct 30 05:21:42.749662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.749552 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472f9495-f7da-43a3-a891-83e3b4894cdd" containerName="payload-pre-processing" Oct 30 05:21:42.749766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.749698 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e647db-b480-43b6-a65c-aaa6352cd512" containerName="payload-processing" Oct 30 05:21:42.749766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.749713 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="472f9495-f7da-43a3-a891-83e3b4894cdd" containerName="payload-pre-processing" Oct 30 05:21:42.755553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.755531 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" Oct 30 05:21:42.771674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.771650 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp"] Oct 30 05:21:42.779545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.779524 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:21:42.779718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.779604 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:21:42.779718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.779640 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:21:42.780213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.780189 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:21:42.846244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.846208 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758ee42e-xw5bp\" (UID: \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" Oct 30 05:21:42.846441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.846334 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvrs5\" (UniqueName: \"kubernetes.io/projected/2dc232fb-c42a-4876-9aa3-e64af1c1484a-kube-api-access-dvrs5\") pod \"maas-api-revoke-sub-758ee42e-xw5bp\" (UID: \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" Oct 30 05:21:42.846516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.846492 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758ee42e-xw5bp\" (UID: \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" Oct 30 05:21:42.948642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.948603 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758ee42e-xw5bp\" (UID: \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" Oct 30 05:21:42.948819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.948725 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvrs5\" (UniqueName: \"kubernetes.io/projected/2dc232fb-c42a-4876-9aa3-e64af1c1484a-kube-api-access-dvrs5\") pod \"maas-api-revoke-sub-758ee42e-xw5bp\" (UID: \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" Oct 30 05:21:42.948918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.948871 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758ee42e-xw5bp\" (UID: \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" Oct 30 05:21:42.949712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.949681 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758ee42e-xw5bp\" (UID: \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" Oct 30 05:21:42.951054 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.951031 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758ee42e-xw5bp\" (UID: \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" Oct 30 05:21:42.970231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:42.970208 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvrs5\" (UniqueName: \"kubernetes.io/projected/2dc232fb-c42a-4876-9aa3-e64af1c1484a-kube-api-access-dvrs5\") pod \"maas-api-revoke-sub-758ee42e-xw5bp\" (UID: \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" Oct 30 05:21:43.073583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:43.073554 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" Oct 30 05:21:43.378619 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:21:43.378590 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc232fb_c42a_4876_9aa3_e64af1c1484a.slice/crio-62bf8fb74e5a68f292ac07e93f966508e499ee422982a5ca6b5b52d0f5af56ea WatchSource:0}: Error finding container 62bf8fb74e5a68f292ac07e93f966508e499ee422982a5ca6b5b52d0f5af56ea: Status 404 returned error can't find the container with id 62bf8fb74e5a68f292ac07e93f966508e499ee422982a5ca6b5b52d0f5af56ea Oct 30 05:21:43.382182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:43.382158 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp"] Oct 30 05:21:43.797716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:43.797624 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" podUID="57f4169e-d26c-48d5-bcc8-fbfb3022972a" containerName="main" containerID="cri-o://78cf8b4c3cb09fed0be742ef8127d222952d9dc67a4eac78eacafcd346716443" gracePeriod=45 Oct 30 05:21:43.802733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:43.802526 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" event={"ID":"2dc232fb-c42a-4876-9aa3-e64af1c1484a","Type":"ContainerStarted","Data":"62bf8fb74e5a68f292ac07e93f966508e499ee422982a5ca6b5b52d0f5af56ea"} Oct 30 05:21:44.129113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.129088 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:21:44.169160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.169128 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-model-cache\") pod \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " Oct 30 05:21:44.169397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.169177 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-home\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-home\") pod \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " Oct 30 05:21:44.169397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.169369 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f4169e-d26c-48d5-bcc8-fbfb3022972a-kube-api-access-r9zl6\" (UniqueName: \"kubernetes.io/projected/57f4169e-d26c-48d5-bcc8-fbfb3022972a-kube-api-access-r9zl6\") pod \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " Oct 30 05:21:44.169555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.169422 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tls-certs\" (UniqueName: \"kubernetes.io/secret/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tls-certs\") pod \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " Oct 30 05:21:44.169555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.169456 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-dshm\") pod \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " Oct 30 05:21:44.169555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.169444 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-model-cache" pod "57f4169e-d26c-48d5-bcc8-fbfb3022972a" (UID: "57f4169e-d26c-48d5-bcc8-fbfb3022972a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:44.169555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.169505 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-home" pod "57f4169e-d26c-48d5-bcc8-fbfb3022972a" (UID: "57f4169e-d26c-48d5-bcc8-fbfb3022972a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:44.169771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.169571 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tmp-dir\") pod \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\" (UID: \"57f4169e-d26c-48d5-bcc8-fbfb3022972a\") " Oct 30 05:21:44.169856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.169811 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tmp-dir" pod "57f4169e-d26c-48d5-bcc8-fbfb3022972a" (UID: "57f4169e-d26c-48d5-bcc8-fbfb3022972a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:44.170406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.170381 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:44.170406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.170407 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:44.170616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.170420 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:44.172403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.171985 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tls-certs" pod "57f4169e-d26c-48d5-bcc8-fbfb3022972a" (UID: "57f4169e-d26c-48d5-bcc8-fbfb3022972a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:21:44.172403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.172031 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-dshm" pod "57f4169e-d26c-48d5-bcc8-fbfb3022972a" (UID: "57f4169e-d26c-48d5-bcc8-fbfb3022972a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:44.173101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.173077 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f4169e-d26c-48d5-bcc8-fbfb3022972a-kube-api-access-r9zl6" pod "57f4169e-d26c-48d5-bcc8-fbfb3022972a" (UID: "57f4169e-d26c-48d5-bcc8-fbfb3022972a"). InnerVolumeSpecName "kube-api-access-r9zl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:44.274990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.274953 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9zl6\" (UniqueName: \"kubernetes.io/projected/57f4169e-d26c-48d5-bcc8-fbfb3022972a-kube-api-access-r9zl6\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:44.274990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.274992 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f4169e-d26c-48d5-bcc8-fbfb3022972a-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:44.274990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.275005 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f4169e-d26c-48d5-bcc8-fbfb3022972a-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:44.727858 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:21:44.727817 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc232fb_c42a_4876_9aa3_e64af1c1484a.slice/crio-c073912bcd3f599c7880d29b267db56be47755a5af33256b5f194dfae00cb0b3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc232fb_c42a_4876_9aa3_e64af1c1484a.slice/crio-conmon-c073912bcd3f599c7880d29b267db56be47755a5af33256b5f194dfae00cb0b3.scope\": RecentStats: unable to find data in memory cache]" Oct 30 05:21:44.813889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.813801 6587 generic.go:356] "Generic (PLEG): container finished" podID="2dc232fb-c42a-4876-9aa3-e64af1c1484a" containerID="c073912bcd3f599c7880d29b267db56be47755a5af33256b5f194dfae00cb0b3" exitCode=0 Oct 30 05:21:44.814298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.813892 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" event={"ID":"2dc232fb-c42a-4876-9aa3-e64af1c1484a","Type":"ContainerDied","Data":"c073912bcd3f599c7880d29b267db56be47755a5af33256b5f194dfae00cb0b3"} Oct 30 05:21:44.815498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.815470 6587 generic.go:356] "Generic (PLEG): container finished" podID="57f4169e-d26c-48d5-bcc8-fbfb3022972a" containerID="78cf8b4c3cb09fed0be742ef8127d222952d9dc67a4eac78eacafcd346716443" exitCode=0 Oct 30 05:21:44.815622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.815514 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" Oct 30 05:21:44.815622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.815551 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" event={"ID":"57f4169e-d26c-48d5-bcc8-fbfb3022972a","Type":"ContainerDied","Data":"78cf8b4c3cb09fed0be742ef8127d222952d9dc67a4eac78eacafcd346716443"} Oct 30 05:21:44.815622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.815592 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676" event={"ID":"57f4169e-d26c-48d5-bcc8-fbfb3022972a","Type":"ContainerDied","Data":"a03b7b86732f128ff762a49a9ed28f1c838384e0fdc52bee310ffbab0f9f8240"} Oct 30 05:21:44.815622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.815609 6587 scope.go:122] "RemoveContainer" containerID="78cf8b4c3cb09fed0be742ef8127d222952d9dc67a4eac78eacafcd346716443" Oct 30 05:21:44.855222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.855087 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx"] Oct 30 05:21:44.860881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.860694 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-453511-csclx"] Oct 30 05:21:44.863655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.863619 6587 scope.go:122] "RemoveContainer" containerID="78cf8b4c3cb09fed0be742ef8127d222952d9dc67a4eac78eacafcd346716443" Oct 30 05:21:44.864048 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:21:44.864018 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cf8b4c3cb09fed0be742ef8127d222952d9dc67a4eac78eacafcd346716443\": container with ID starting with 78cf8b4c3cb09fed0be742ef8127d222952d9dc67a4eac78eacafcd346716443 not found: ID does not exist" containerID="78cf8b4c3cb09fed0be742ef8127d222952d9dc67a4eac78eacafcd346716443" Oct 30 05:21:44.864180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.864059 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cf8b4c3cb09fed0be742ef8127d222952d9dc67a4eac78eacafcd346716443"} err="failed to get container status \"78cf8b4c3cb09fed0be742ef8127d222952d9dc67a4eac78eacafcd346716443\": rpc error: code = NotFound desc = could not find container \"78cf8b4c3cb09fed0be742ef8127d222952d9dc67a4eac78eacafcd346716443\": container with ID starting with 78cf8b4c3cb09fed0be742ef8127d222952d9dc67a4eac78eacafcd346716443 not found: ID does not exist" Oct 30 05:21:44.905738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.905708 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676"] Oct 30 05:21:44.909886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:44.909860 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2a8f18/test-model-2a8f18-kserve-5876b794f5-94676"] Oct 30 05:21:45.587416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:45.587377 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f4169e-d26c-48d5-bcc8-fbfb3022972a" path="/var/lib/kubelet/pods/57f4169e-d26c-48d5-bcc8-fbfb3022972a/volumes" Oct 30 05:21:45.588094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:45.588069 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb4d9567-0b72-431a-b33a-5d6570482975" path="/var/lib/kubelet/pods/fb4d9567-0b72-431a-b33a-5d6570482975/volumes" Oct 30 05:21:46.013834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:46.013804 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" Oct 30 05:21:46.163663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:46.163573 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc232fb-c42a-4876-9aa3-e64af1c1484a-kube-api-access-dvrs5\" (UniqueName: \"kubernetes.io/projected/2dc232fb-c42a-4876-9aa3-e64af1c1484a-kube-api-access-dvrs5\") pod \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\" (UID: \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\") " Oct 30 05:21:46.163663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:46.163626 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-xks-ca\") pod \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\" (UID: \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\") " Oct 30 05:21:46.163878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:46.163825 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-openshift-ca\") pod \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\" (UID: \"2dc232fb-c42a-4876-9aa3-e64af1c1484a\") " Oct 30 05:21:46.164345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:46.164303 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-openshift-ca" pod "2dc232fb-c42a-4876-9aa3-e64af1c1484a" (UID: "2dc232fb-c42a-4876-9aa3-e64af1c1484a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:46.164656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:46.164632 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:46.165988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:46.165962 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc232fb-c42a-4876-9aa3-e64af1c1484a-kube-api-access-dvrs5" pod "2dc232fb-c42a-4876-9aa3-e64af1c1484a" (UID: "2dc232fb-c42a-4876-9aa3-e64af1c1484a"). InnerVolumeSpecName "kube-api-access-dvrs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:46.166091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:46.166077 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-xks-ca" pod "2dc232fb-c42a-4876-9aa3-e64af1c1484a" (UID: "2dc232fb-c42a-4876-9aa3-e64af1c1484a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:21:46.266633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:46.266592 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvrs5\" (UniqueName: \"kubernetes.io/projected/2dc232fb-c42a-4876-9aa3-e64af1c1484a-kube-api-access-dvrs5\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:46.266633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:46.266631 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc232fb-c42a-4876-9aa3-e64af1c1484a-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:46.835374 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:46.835337 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" Oct 30 05:21:46.835581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:46.835405 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp" event={"ID":"2dc232fb-c42a-4876-9aa3-e64af1c1484a","Type":"ContainerDied","Data":"62bf8fb74e5a68f292ac07e93f966508e499ee422982a5ca6b5b52d0f5af56ea"} Oct 30 05:21:46.835581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:46.835454 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62bf8fb74e5a68f292ac07e93f966508e499ee422982a5ca6b5b52d0f5af56ea" Oct 30 05:21:48.748663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:48.748630 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" Oct 30 05:21:49.404336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:49.404297 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g"] Oct 30 05:21:49.417824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:49.417796 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ccafe0-nnb9g"] Oct 30 05:21:49.584824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:49.584794 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4486d33f-de9c-4380-9a1e-c40afef5f1ec" path="/var/lib/kubelet/pods/4486d33f-de9c-4380-9a1e-c40afef5f1ec/volumes" Oct 30 05:21:51.856769 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:51.856735 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk"] Oct 30 05:21:51.857606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:51.857569 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" podUID="548909cc-6196-4b54-9372-e9f61190ddce" containerName="istio-proxy" containerID="cri-o://1a460147748390515cee934b5a061105f177ccfccf23f29dd450b14cad1356a2" gracePeriod=30 Oct 30 05:21:55.988487 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:21:55.988450 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:21:56.125118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.125083 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9"] Oct 30 05:21:56.125994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.125975 6587 cpu_manager.go:414] "containerMap: removing container" podUID="57f4169e-d26c-48d5-bcc8-fbfb3022972a" containerName="main" Oct 30 05:21:56.125994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.125994 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f4169e-d26c-48d5-bcc8-fbfb3022972a" containerName="main" Oct 30 05:21:56.126119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.126041 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2dc232fb-c42a-4876-9aa3-e64af1c1484a" containerName="revoke-keys" Oct 30 05:21:56.126119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.126046 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc232fb-c42a-4876-9aa3-e64af1c1484a" containerName="revoke-keys" Oct 30 05:21:56.126201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.126166 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc232fb-c42a-4876-9aa3-e64af1c1484a" containerName="revoke-keys" Oct 30 05:21:56.126201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.126182 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f4169e-d26c-48d5-bcc8-fbfb3022972a" containerName="main" Oct 30 05:21:56.131332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.131313 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" Oct 30 05:21:56.158604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.158569 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9"] Oct 30 05:21:56.163509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.163477 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1a39058-bb2d9\" (UID: \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" Oct 30 05:21:56.163642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.163604 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvjqb\" (UniqueName: \"kubernetes.io/projected/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-kube-api-access-bvjqb\") pod \"maas-api-revoke-sub-e1a39058-bb2d9\" (UID: \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" Oct 30 05:21:56.163713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.163700 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1a39058-bb2d9\" (UID: \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" Oct 30 05:21:56.165412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.165387 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:21:56.168552 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.168527 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:21:56.168678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.168609 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:21:56.168678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.168657 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:21:56.265774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.265685 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1a39058-bb2d9\" (UID: \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" Oct 30 05:21:56.265958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.265790 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjqb\" (UniqueName: \"kubernetes.io/projected/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-kube-api-access-bvjqb\") pod \"maas-api-revoke-sub-e1a39058-bb2d9\" (UID: \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" Oct 30 05:21:56.265958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.265900 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1a39058-bb2d9\" (UID: \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" Oct 30 05:21:56.266419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.266392 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1a39058-bb2d9\" (UID: \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" Oct 30 05:21:56.268237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.268215 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1a39058-bb2d9\" (UID: \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" Oct 30 05:21:56.287270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.287244 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvjqb\" (UniqueName: \"kubernetes.io/projected/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-kube-api-access-bvjqb\") pod \"maas-api-revoke-sub-e1a39058-bb2d9\" (UID: \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" Oct 30 05:21:56.447226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:56.447189 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" Oct 30 05:21:57.033757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.033702 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt"] Oct 30 05:21:57.039840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.039816 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" Oct 30 05:21:57.040457 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:21:57.040430 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750ce4e5_54b1_40a0_95e0_c496b89f9b4e.slice/crio-2a47e0bf13ac929ddafa4230931ef4aae1f7ce129a556cf4a2d2ca5316ad730d WatchSource:0}: Error finding container 2a47e0bf13ac929ddafa4230931ef4aae1f7ce129a556cf4a2d2ca5316ad730d: Status 404 returned error can't find the container with id 2a47e0bf13ac929ddafa4230931ef4aae1f7ce129a556cf4a2d2ca5316ad730d Oct 30 05:21:57.042640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.042617 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9"] Oct 30 05:21:57.083715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.065429 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt"] Oct 30 05:21:57.138677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.137801 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzdfr\" (UniqueName: \"kubernetes.io/projected/04975143-7d13-480c-b880-4c317015e96b-kube-api-access-lzdfr\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt\" (UID: \"04975143-7d13-480c-b880-4c317015e96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" Oct 30 05:21:57.138677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.138016 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04975143-7d13-480c-b880-4c317015e96b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt\" (UID: \"04975143-7d13-480c-b880-4c317015e96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" Oct 30 05:21:57.138677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.138067 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04975143-7d13-480c-b880-4c317015e96b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt\" (UID: \"04975143-7d13-480c-b880-4c317015e96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" Oct 30 05:21:57.170182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.170156 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:21:57.241381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.240460 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-data\") pod \"548909cc-6196-4b54-9372-e9f61190ddce\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.242527 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-data" pod "548909cc-6196-4b54-9372-e9f61190ddce" (UID: "548909cc-6196-4b54-9372-e9f61190ddce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.242694 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-credential-socket\") pod \"548909cc-6196-4b54-9372-e9f61190ddce\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.242762 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/548909cc-6196-4b54-9372-e9f61190ddce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548909cc-6196-4b54-9372-e9f61190ddce-istio-podinfo\") pod \"548909cc-6196-4b54-9372-e9f61190ddce\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.243023 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-kube-api-access-dkspw\" (UniqueName: \"kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-kube-api-access-dkspw\") pod \"548909cc-6196-4b54-9372-e9f61190ddce\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.243087 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-socket\") pod \"548909cc-6196-4b54-9372-e9f61190ddce\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.243155 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-envoy\") pod \"548909cc-6196-4b54-9372-e9f61190ddce\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.243228 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-certs\") pod \"548909cc-6196-4b54-9372-e9f61190ddce\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.243324 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/548909cc-6196-4b54-9372-e9f61190ddce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548909cc-6196-4b54-9372-e9f61190ddce-istiod-ca-cert\") pod \"548909cc-6196-4b54-9372-e9f61190ddce\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.243443 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-istio-token\" (UniqueName: \"kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-istio-token\") pod \"548909cc-6196-4b54-9372-e9f61190ddce\" (UID: \"548909cc-6196-4b54-9372-e9f61190ddce\") " Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.243761 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04975143-7d13-480c-b880-4c317015e96b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt\" (UID: \"04975143-7d13-480c-b880-4c317015e96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.243804 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04975143-7d13-480c-b880-4c317015e96b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt\" (UID: \"04975143-7d13-480c-b880-4c317015e96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.244162 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzdfr\" (UniqueName: \"kubernetes.io/projected/04975143-7d13-480c-b880-4c317015e96b-kube-api-access-lzdfr\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt\" (UID: \"04975143-7d13-480c-b880-4c317015e96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.244330 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.244940 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-credential-socket" pod "548909cc-6196-4b54-9372-e9f61190ddce" (UID: "548909cc-6196-4b54-9372-e9f61190ddce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.246694 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-certs" pod "548909cc-6196-4b54-9372-e9f61190ddce" (UID: "548909cc-6196-4b54-9372-e9f61190ddce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.247055 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-socket" pod "548909cc-6196-4b54-9372-e9f61190ddce" (UID: "548909cc-6196-4b54-9372-e9f61190ddce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.249685 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548909cc-6196-4b54-9372-e9f61190ddce-istiod-ca-cert" pod "548909cc-6196-4b54-9372-e9f61190ddce" (UID: "548909cc-6196-4b54-9372-e9f61190ddce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:21:57.256407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.253278 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04975143-7d13-480c-b880-4c317015e96b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt\" (UID: \"04975143-7d13-480c-b880-4c317015e96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" Oct 30 05:21:57.317213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.317173 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/548909cc-6196-4b54-9372-e9f61190ddce-istio-podinfo" pod "548909cc-6196-4b54-9372-e9f61190ddce" (UID: "548909cc-6196-4b54-9372-e9f61190ddce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:21:57.317527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.317502 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04975143-7d13-480c-b880-4c317015e96b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt\" (UID: \"04975143-7d13-480c-b880-4c317015e96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" Oct 30 05:21:57.317727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.317672 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-istio-token" pod "548909cc-6196-4b54-9372-e9f61190ddce" (UID: "548909cc-6196-4b54-9372-e9f61190ddce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:57.317869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.317781 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzdfr\" (UniqueName: \"kubernetes.io/projected/04975143-7d13-480c-b880-4c317015e96b-kube-api-access-lzdfr\") pod \"maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt\" (UID: \"04975143-7d13-480c-b880-4c317015e96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" Oct 30 05:21:57.318250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.318221 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-kube-api-access-dkspw" pod "548909cc-6196-4b54-9372-e9f61190ddce" (UID: "548909cc-6196-4b54-9372-e9f61190ddce"). InnerVolumeSpecName "kube-api-access-dkspw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:21:57.318349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.318264 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-envoy" pod "548909cc-6196-4b54-9372-e9f61190ddce" (UID: "548909cc-6196-4b54-9372-e9f61190ddce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:21:57.345519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.345492 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.345519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.345521 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548909cc-6196-4b54-9372-e9f61190ddce-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.345685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.345530 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkspw\" (UniqueName: \"kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-kube-api-access-dkspw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.345685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.345540 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.345685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.345549 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.345685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.345557 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548909cc-6196-4b54-9372-e9f61190ddce-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.345685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.345564 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548909cc-6196-4b54-9372-e9f61190ddce-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.345685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.345572 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548909cc-6196-4b54-9372-e9f61190ddce-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:21:57.367657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.367634 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" Oct 30 05:21:57.677589 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:21:57.677554 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04975143_7d13_480c_b880_4c317015e96b.slice/crio-7c0c8a43df841e6a68ba2edcb8c354e4fb947a3c5fbd478153623564974d7452 WatchSource:0}: Error finding container 7c0c8a43df841e6a68ba2edcb8c354e4fb947a3c5fbd478153623564974d7452: Status 404 returned error can't find the container with id 7c0c8a43df841e6a68ba2edcb8c354e4fb947a3c5fbd478153623564974d7452 Oct 30 05:21:57.678683 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.678659 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt"] Oct 30 05:21:57.946352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.946322 6587 generic.go:356] "Generic (PLEG): container finished" podID="548909cc-6196-4b54-9372-e9f61190ddce" containerID="1a460147748390515cee934b5a061105f177ccfccf23f29dd450b14cad1356a2" exitCode=0 Oct 30 05:21:57.946352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.946387 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" Oct 30 05:21:57.946672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.946399 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" event={"ID":"548909cc-6196-4b54-9372-e9f61190ddce","Type":"ContainerDied","Data":"1a460147748390515cee934b5a061105f177ccfccf23f29dd450b14cad1356a2"} Oct 30 05:21:57.946672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.946459 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk" event={"ID":"548909cc-6196-4b54-9372-e9f61190ddce","Type":"ContainerDied","Data":"67bb309ef212bcaa1e874e2065d0ca61334e2f64b50f1a7a00ac6d6dff415e37"} Oct 30 05:21:57.946672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.946485 6587 scope.go:122] "RemoveContainer" containerID="1a460147748390515cee934b5a061105f177ccfccf23f29dd450b14cad1356a2" Oct 30 05:21:57.948301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.948278 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" event={"ID":"04975143-7d13-480c-b880-4c317015e96b","Type":"ContainerStarted","Data":"7c0c8a43df841e6a68ba2edcb8c354e4fb947a3c5fbd478153623564974d7452"} Oct 30 05:21:57.950496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.950472 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" event={"ID":"750ce4e5-54b1-40a0-95e0-c496b89f9b4e","Type":"ContainerStarted","Data":"dfcc719f90826824832851b7244774bbf26af809697d03c0e2ec4689e58656b6"} Oct 30 05:21:57.950620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.950522 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" event={"ID":"750ce4e5-54b1-40a0-95e0-c496b89f9b4e","Type":"ContainerStarted","Data":"2a47e0bf13ac929ddafa4230931ef4aae1f7ce129a556cf4a2d2ca5316ad730d"} Oct 30 05:21:57.958954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.958931 6587 scope.go:122] "RemoveContainer" containerID="145cc54dd3354c34a60099025653385ed8c73ed200d9a158cae9c68ebba6580b" Oct 30 05:21:57.969337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.969312 6587 scope.go:122] "RemoveContainer" containerID="1a460147748390515cee934b5a061105f177ccfccf23f29dd450b14cad1356a2" Oct 30 05:21:57.969643 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:21:57.969619 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a460147748390515cee934b5a061105f177ccfccf23f29dd450b14cad1356a2\": container with ID starting with 1a460147748390515cee934b5a061105f177ccfccf23f29dd450b14cad1356a2 not found: ID does not exist" containerID="1a460147748390515cee934b5a061105f177ccfccf23f29dd450b14cad1356a2" Oct 30 05:21:57.969704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.969655 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a460147748390515cee934b5a061105f177ccfccf23f29dd450b14cad1356a2"} err="failed to get container status \"1a460147748390515cee934b5a061105f177ccfccf23f29dd450b14cad1356a2\": rpc error: code = NotFound desc = could not find container \"1a460147748390515cee934b5a061105f177ccfccf23f29dd450b14cad1356a2\": container with ID starting with 1a460147748390515cee934b5a061105f177ccfccf23f29dd450b14cad1356a2 not found: ID does not exist" Oct 30 05:21:57.969704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.969680 6587 scope.go:122] "RemoveContainer" containerID="145cc54dd3354c34a60099025653385ed8c73ed200d9a158cae9c68ebba6580b" Oct 30 05:21:57.969955 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:21:57.969935 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145cc54dd3354c34a60099025653385ed8c73ed200d9a158cae9c68ebba6580b\": container with ID starting with 145cc54dd3354c34a60099025653385ed8c73ed200d9a158cae9c68ebba6580b not found: ID does not exist" containerID="145cc54dd3354c34a60099025653385ed8c73ed200d9a158cae9c68ebba6580b" Oct 30 05:21:57.970010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:57.969960 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145cc54dd3354c34a60099025653385ed8c73ed200d9a158cae9c68ebba6580b"} err="failed to get container status \"145cc54dd3354c34a60099025653385ed8c73ed200d9a158cae9c68ebba6580b\": rpc error: code = NotFound desc = could not find container \"145cc54dd3354c34a60099025653385ed8c73ed200d9a158cae9c68ebba6580b\": container with ID starting with 145cc54dd3354c34a60099025653385ed8c73ed200d9a158cae9c68ebba6580b not found: ID does not exist" Oct 30 05:21:58.007081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:58.007021 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" podStartSLOduration=2.00700852 podStartE2EDuration="2.00700852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:21:58.003739328 +0000 UTC m=+2242.531506114" watchObservedRunningTime="2026-10-30 05:21:58.00700852 +0000 UTC m=+2242.534775306" Oct 30 05:21:58.045528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:58.045492 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk"] Oct 30 05:21:58.049928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:58.049902 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk"] Oct 30 05:21:58.962823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:58.962731 6587 generic.go:356] "Generic (PLEG): container finished" podID="04975143-7d13-480c-b880-4c317015e96b" containerID="64b6bdfad00018cd8eb156e1cc406f9d3332a30a11250a252481bd9c898d0395" exitCode=0 Oct 30 05:21:58.962990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:58.962816 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" event={"ID":"04975143-7d13-480c-b880-4c317015e96b","Type":"ContainerDied","Data":"64b6bdfad00018cd8eb156e1cc406f9d3332a30a11250a252481bd9c898d0395"} Oct 30 05:21:58.964795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:58.964765 6587 generic.go:356] "Generic (PLEG): container finished" podID="750ce4e5-54b1-40a0-95e0-c496b89f9b4e" containerID="dfcc719f90826824832851b7244774bbf26af809697d03c0e2ec4689e58656b6" exitCode=0 Oct 30 05:21:58.964932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:58.964848 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" event={"ID":"750ce4e5-54b1-40a0-95e0-c496b89f9b4e","Type":"ContainerDied","Data":"dfcc719f90826824832851b7244774bbf26af809697d03c0e2ec4689e58656b6"} Oct 30 05:21:59.585539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:21:59.585509 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="548909cc-6196-4b54-9372-e9f61190ddce" path="/var/lib/kubelet/pods/548909cc-6196-4b54-9372-e9f61190ddce/volumes" Oct 30 05:22:00.186091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.186060 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" Oct 30 05:22:00.223814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.223748 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" Oct 30 05:22:00.292405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.292374 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04975143-7d13-480c-b880-4c317015e96b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04975143-7d13-480c-b880-4c317015e96b-maas-api-openshift-ca\") pod \"04975143-7d13-480c-b880-4c317015e96b\" (UID: \"04975143-7d13-480c-b880-4c317015e96b\") " Oct 30 05:22:00.292590 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.292432 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-openshift-ca\") pod \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\" (UID: \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\") " Oct 30 05:22:00.292590 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.292514 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04975143-7d13-480c-b880-4c317015e96b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04975143-7d13-480c-b880-4c317015e96b-maas-api-xks-ca\") pod \"04975143-7d13-480c-b880-4c317015e96b\" (UID: \"04975143-7d13-480c-b880-4c317015e96b\") " Oct 30 05:22:00.292739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.292595 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04975143-7d13-480c-b880-4c317015e96b-kube-api-access-lzdfr\" (UniqueName: \"kubernetes.io/projected/04975143-7d13-480c-b880-4c317015e96b-kube-api-access-lzdfr\") pod \"04975143-7d13-480c-b880-4c317015e96b\" (UID: \"04975143-7d13-480c-b880-4c317015e96b\") " Oct 30 05:22:00.292739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.292673 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-xks-ca\") pod \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\" (UID: \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\") " Oct 30 05:22:00.292858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.292766 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-kube-api-access-bvjqb\" (UniqueName: \"kubernetes.io/projected/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-kube-api-access-bvjqb\") pod \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\" (UID: \"750ce4e5-54b1-40a0-95e0-c496b89f9b4e\") " Oct 30 05:22:00.293454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.293422 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-openshift-ca" pod "750ce4e5-54b1-40a0-95e0-c496b89f9b4e" (UID: "750ce4e5-54b1-40a0-95e0-c496b89f9b4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:00.293678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.293657 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04975143-7d13-480c-b880-4c317015e96b-maas-api-openshift-ca" pod "04975143-7d13-480c-b880-4c317015e96b" (UID: "04975143-7d13-480c-b880-4c317015e96b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:00.293954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.293927 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04975143-7d13-480c-b880-4c317015e96b-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:00.294042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.293957 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:00.296271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.296238 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-kube-api-access-bvjqb" pod "750ce4e5-54b1-40a0-95e0-c496b89f9b4e" (UID: "750ce4e5-54b1-40a0-95e0-c496b89f9b4e"). InnerVolumeSpecName "kube-api-access-bvjqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:00.296529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.296508 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-xks-ca" pod "750ce4e5-54b1-40a0-95e0-c496b89f9b4e" (UID: "750ce4e5-54b1-40a0-95e0-c496b89f9b4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:22:00.297459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.297238 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04975143-7d13-480c-b880-4c317015e96b-maas-api-xks-ca" pod "04975143-7d13-480c-b880-4c317015e96b" (UID: "04975143-7d13-480c-b880-4c317015e96b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:22:00.298102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.298081 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04975143-7d13-480c-b880-4c317015e96b-kube-api-access-lzdfr" pod "04975143-7d13-480c-b880-4c317015e96b" (UID: "04975143-7d13-480c-b880-4c317015e96b"). InnerVolumeSpecName "kube-api-access-lzdfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:00.395447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.395411 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvjqb\" (UniqueName: \"kubernetes.io/projected/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-kube-api-access-bvjqb\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:00.395447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.395442 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04975143-7d13-480c-b880-4c317015e96b-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:00.395447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.395452 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzdfr\" (UniqueName: \"kubernetes.io/projected/04975143-7d13-480c-b880-4c317015e96b-kube-api-access-lzdfr\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:00.395691 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:00.395462 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750ce4e5-54b1-40a0-95e0-c496b89f9b4e-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:01.070983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:01.070742 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" event={"ID":"04975143-7d13-480c-b880-4c317015e96b","Type":"ContainerDied","Data":"7c0c8a43df841e6a68ba2edcb8c354e4fb947a3c5fbd478153623564974d7452"} Oct 30 05:22:01.070983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:01.070840 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c0c8a43df841e6a68ba2edcb8c354e4fb947a3c5fbd478153623564974d7452" Oct 30 05:22:01.070983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:01.070907 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt" Oct 30 05:22:01.074336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:01.074304 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" event={"ID":"750ce4e5-54b1-40a0-95e0-c496b89f9b4e","Type":"ContainerDied","Data":"2a47e0bf13ac929ddafa4230931ef4aae1f7ce129a556cf4a2d2ca5316ad730d"} Oct 30 05:22:01.074430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:01.074377 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a47e0bf13ac929ddafa4230931ef4aae1f7ce129a556cf4a2d2ca5316ad730d" Oct 30 05:22:01.074466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:01.074444 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9" Oct 30 05:22:07.505033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:07.505000 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj"] Oct 30 05:22:07.543745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:07.543705 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk"] Oct 30 05:22:12.086314 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.086270 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr"] Oct 30 05:22:12.087297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.087271 6587 cpu_manager.go:414] "containerMap: removing container" podUID="750ce4e5-54b1-40a0-95e0-c496b89f9b4e" containerName="revoke-keys" Oct 30 05:22:12.087458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.087299 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750ce4e5-54b1-40a0-95e0-c496b89f9b4e" containerName="revoke-keys" Oct 30 05:22:12.087458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.087372 6587 cpu_manager.go:414] "containerMap: removing container" podUID="04975143-7d13-480c-b880-4c317015e96b" containerName="revoke-keys" Oct 30 05:22:12.087458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.087381 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04975143-7d13-480c-b880-4c317015e96b" containerName="revoke-keys" Oct 30 05:22:12.087458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.087432 6587 cpu_manager.go:414] "containerMap: removing container" podUID="548909cc-6196-4b54-9372-e9f61190ddce" containerName="istio-proxy" Oct 30 05:22:12.087458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.087440 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548909cc-6196-4b54-9372-e9f61190ddce" containerName="istio-proxy" Oct 30 05:22:12.087735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.087653 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="548909cc-6196-4b54-9372-e9f61190ddce" containerName="istio-proxy" Oct 30 05:22:12.087735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.087669 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="548909cc-6196-4b54-9372-e9f61190ddce" containerName="istio-proxy" Oct 30 05:22:12.087735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.087680 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="750ce4e5-54b1-40a0-95e0-c496b89f9b4e" containerName="revoke-keys" Oct 30 05:22:12.087735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.087701 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="04975143-7d13-480c-b880-4c317015e96b" containerName="revoke-keys" Oct 30 05:22:12.093004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.092982 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" Oct 30 05:22:12.109509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.109485 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr"] Oct 30 05:22:12.117851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.117624 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:22:12.117851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.117741 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:22:12.117851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.117819 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:22:12.118815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.118796 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:22:12.213371 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.213333 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zmr\" (UniqueName: \"kubernetes.io/projected/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-kube-api-access-z7zmr\") pod \"maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr\" (UID: \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" Oct 30 05:22:12.213591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.213574 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr\" (UID: \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" Oct 30 05:22:12.213642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.213629 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr\" (UID: \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" Oct 30 05:22:12.315442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.315347 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr\" (UID: \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" Oct 30 05:22:12.315647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.315488 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr\" (UID: \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" Oct 30 05:22:12.315729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.315645 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zmr\" (UniqueName: \"kubernetes.io/projected/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-kube-api-access-z7zmr\") pod \"maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr\" (UID: \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" Oct 30 05:22:12.316277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.316246 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr\" (UID: \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" Oct 30 05:22:12.317989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.317962 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr\" (UID: \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" Oct 30 05:22:12.338019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.337957 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zmr\" (UniqueName: \"kubernetes.io/projected/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-kube-api-access-z7zmr\") pod \"maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr\" (UID: \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" Oct 30 05:22:12.409841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.409809 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" Oct 30 05:22:12.506086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.505969 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" podUID="fbe6597b-6a31-4372-a6aa-d6b041b06b56" containerName="payload-processing" containerID="cri-o://5aeb9ba2a48924face9a4612f54190f27f195565cb74f25c5bc7afee4eb7d41d" gracePeriod=25 Oct 30 05:22:12.544334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.544262 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" podUID="dfe25060-9266-482c-8a47-f2f9e317ecb0" containerName="payload-pre-processing" containerID="cri-o://755ac7d6aa832662106632d946eb9eeeac60aaa6b5455aeca2b53f6ca9032980" gracePeriod=25 Oct 30 05:22:12.719635 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:22:12.719604 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48331a9d_5dc6_4711_a162_c22f7ab8c1b2.slice/crio-5dec916d5d1ea2d23624d50d0c8a432d6453e0907923d66646c680ec6f6f63a0 WatchSource:0}: Error finding container 5dec916d5d1ea2d23624d50d0c8a432d6453e0907923d66646c680ec6f6f63a0: Status 404 returned error can't find the container with id 5dec916d5d1ea2d23624d50d0c8a432d6453e0907923d66646c680ec6f6f63a0 Oct 30 05:22:12.722637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:12.722605 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr"] Oct 30 05:22:13.210852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:13.210806 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" event={"ID":"48331a9d-5dc6-4711-a162-c22f7ab8c1b2","Type":"ContainerStarted","Data":"62a35cc4baeff17674b4aa5688b23af6c256cdb99c74525acd30e32d8eedf59c"} Oct 30 05:22:13.211338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:13.210879 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" event={"ID":"48331a9d-5dc6-4711-a162-c22f7ab8c1b2","Type":"ContainerStarted","Data":"5dec916d5d1ea2d23624d50d0c8a432d6453e0907923d66646c680ec6f6f63a0"} Oct 30 05:22:13.252458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:13.252402 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" podStartSLOduration=1.25238599 podStartE2EDuration="1.25238599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:22:13.251978488 +0000 UTC m=+2257.779745275" watchObservedRunningTime="2026-10-30 05:22:13.25238599 +0000 UTC m=+2257.780152772" Oct 30 05:22:14.221430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:14.221394 6587 generic.go:356] "Generic (PLEG): container finished" podID="48331a9d-5dc6-4711-a162-c22f7ab8c1b2" containerID="62a35cc4baeff17674b4aa5688b23af6c256cdb99c74525acd30e32d8eedf59c" exitCode=0 Oct 30 05:22:14.221900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:14.221521 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" event={"ID":"48331a9d-5dc6-4711-a162-c22f7ab8c1b2","Type":"ContainerDied","Data":"62a35cc4baeff17674b4aa5688b23af6c256cdb99c74525acd30e32d8eedf59c"} Oct 30 05:22:15.451508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:15.451482 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" Oct 30 05:22:15.569989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:15.569960 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-openshift-ca\") pod \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\" (UID: \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\") " Oct 30 05:22:15.570191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:15.570020 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-kube-api-access-z7zmr\" (UniqueName: \"kubernetes.io/projected/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-kube-api-access-z7zmr\") pod \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\" (UID: \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\") " Oct 30 05:22:15.570191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:15.570096 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-xks-ca\") pod \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\" (UID: \"48331a9d-5dc6-4711-a162-c22f7ab8c1b2\") " Oct 30 05:22:15.570349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:15.570323 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-openshift-ca" pod "48331a9d-5dc6-4711-a162-c22f7ab8c1b2" (UID: "48331a9d-5dc6-4711-a162-c22f7ab8c1b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:15.570861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:15.570838 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:15.572300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:15.572274 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-xks-ca" pod "48331a9d-5dc6-4711-a162-c22f7ab8c1b2" (UID: "48331a9d-5dc6-4711-a162-c22f7ab8c1b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:22:15.572415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:15.572390 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-kube-api-access-z7zmr" pod "48331a9d-5dc6-4711-a162-c22f7ab8c1b2" (UID: "48331a9d-5dc6-4711-a162-c22f7ab8c1b2"). InnerVolumeSpecName "kube-api-access-z7zmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:15.672109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:15.672070 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7zmr\" (UniqueName: \"kubernetes.io/projected/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-kube-api-access-z7zmr\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:15.672109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:15.672103 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48331a9d-5dc6-4711-a162-c22f7ab8c1b2-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:15.932230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:15.932135 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" podUID="dfe25060-9266-482c-8a47-f2f9e317ecb0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:22:15.932430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:15.932161 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" podUID="fbe6597b-6a31-4372-a6aa-d6b041b06b56" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:22:16.266436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:16.265456 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" event={"ID":"48331a9d-5dc6-4711-a162-c22f7ab8c1b2","Type":"ContainerDied","Data":"5dec916d5d1ea2d23624d50d0c8a432d6453e0907923d66646c680ec6f6f63a0"} Oct 30 05:22:16.266436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:16.265519 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dec916d5d1ea2d23624d50d0c8a432d6453e0907923d66646c680ec6f6f63a0" Oct 30 05:22:16.266436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:16.265586 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr" Oct 30 05:22:20.178765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.178686 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4"] Oct 30 05:22:20.179887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.179866 6587 cpu_manager.go:414] "containerMap: removing container" podUID="48331a9d-5dc6-4711-a162-c22f7ab8c1b2" containerName="revoke-keys" Oct 30 05:22:20.179950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.179890 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48331a9d-5dc6-4711-a162-c22f7ab8c1b2" containerName="revoke-keys" Oct 30 05:22:20.179950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.179935 6587 cpu_manager.go:414] "containerMap: removing container" podUID="548909cc-6196-4b54-9372-e9f61190ddce" containerName="istio-proxy" Oct 30 05:22:20.179950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.179943 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548909cc-6196-4b54-9372-e9f61190ddce" containerName="istio-proxy" Oct 30 05:22:20.180108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.180096 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="48331a9d-5dc6-4711-a162-c22f7ab8c1b2" containerName="revoke-keys" Oct 30 05:22:20.184909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.184870 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" Oct 30 05:22:20.205499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.205459 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4"] Oct 30 05:22:20.212311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.209190 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:22:20.212311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.209296 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:22:20.212311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.209465 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:22:20.214971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.214948 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:22:20.275299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.275266 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53e0a2e-ztcj4\" (UID: \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" Oct 30 05:22:20.275479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.275406 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53e0a2e-ztcj4\" (UID: \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" Oct 30 05:22:20.275479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.275470 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msm59\" (UniqueName: \"kubernetes.io/projected/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-kube-api-access-msm59\") pod \"maas-api-revoke-sub-a53e0a2e-ztcj4\" (UID: \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" Oct 30 05:22:20.376530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.376496 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msm59\" (UniqueName: \"kubernetes.io/projected/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-kube-api-access-msm59\") pod \"maas-api-revoke-sub-a53e0a2e-ztcj4\" (UID: \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" Oct 30 05:22:20.376775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.376759 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53e0a2e-ztcj4\" (UID: \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" Oct 30 05:22:20.376866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.376850 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53e0a2e-ztcj4\" (UID: \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" Oct 30 05:22:20.377464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.377438 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53e0a2e-ztcj4\" (UID: \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" Oct 30 05:22:20.379280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.379249 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53e0a2e-ztcj4\" (UID: \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" Oct 30 05:22:20.396780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.396745 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msm59\" (UniqueName: \"kubernetes.io/projected/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-kube-api-access-msm59\") pod \"maas-api-revoke-sub-a53e0a2e-ztcj4\" (UID: \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" Oct 30 05:22:20.505486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.505400 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" Oct 30 05:22:20.862835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:20.858717 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4"] Oct 30 05:22:20.868010 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:22:20.867977 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b567ca_fe72_4ebe_a47a_4df8699bbc55.slice/crio-7341e2e2c6d273c56fd76846cb2f35e6e60148f021fa3da7ab94a119994b1421 WatchSource:0}: Error finding container 7341e2e2c6d273c56fd76846cb2f35e6e60148f021fa3da7ab94a119994b1421: Status 404 returned error can't find the container with id 7341e2e2c6d273c56fd76846cb2f35e6e60148f021fa3da7ab94a119994b1421 Oct 30 05:22:21.376900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:21.376862 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" event={"ID":"b8b567ca-fe72-4ebe-a47a-4df8699bbc55","Type":"ContainerStarted","Data":"7341e2e2c6d273c56fd76846cb2f35e6e60148f021fa3da7ab94a119994b1421"} Oct 30 05:22:22.391707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:22.391669 6587 generic.go:356] "Generic (PLEG): container finished" podID="b8b567ca-fe72-4ebe-a47a-4df8699bbc55" containerID="6fbbbae98753ad6b376c8df26aae858f0e83e77816b4c168bdef9b9005a0b071" exitCode=0 Oct 30 05:22:22.392091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:22.391762 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" event={"ID":"b8b567ca-fe72-4ebe-a47a-4df8699bbc55","Type":"ContainerDied","Data":"6fbbbae98753ad6b376c8df26aae858f0e83e77816b4c168bdef9b9005a0b071"} Oct 30 05:22:23.589133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:23.589110 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" Oct 30 05:22:23.638938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:23.638900 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-openshift-ca\") pod \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\" (UID: \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\") " Oct 30 05:22:23.639136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:23.639006 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-kube-api-access-msm59\" (UniqueName: \"kubernetes.io/projected/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-kube-api-access-msm59\") pod \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\" (UID: \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\") " Oct 30 05:22:23.639136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:23.639075 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-xks-ca\") pod \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\" (UID: \"b8b567ca-fe72-4ebe-a47a-4df8699bbc55\") " Oct 30 05:22:23.639463 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:23.639333 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-openshift-ca" pod "b8b567ca-fe72-4ebe-a47a-4df8699bbc55" (UID: "b8b567ca-fe72-4ebe-a47a-4df8699bbc55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:23.640085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:23.640058 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:23.642415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:23.641347 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-xks-ca" pod "b8b567ca-fe72-4ebe-a47a-4df8699bbc55" (UID: "b8b567ca-fe72-4ebe-a47a-4df8699bbc55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:22:23.642415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:23.641696 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-kube-api-access-msm59" pod "b8b567ca-fe72-4ebe-a47a-4df8699bbc55" (UID: "b8b567ca-fe72-4ebe-a47a-4df8699bbc55"). InnerVolumeSpecName "kube-api-access-msm59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:23.743204 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:23.743174 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msm59\" (UniqueName: \"kubernetes.io/projected/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-kube-api-access-msm59\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:23.743204 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:23.743205 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b567ca-fe72-4ebe-a47a-4df8699bbc55-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:24.410724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:24.410690 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" Oct 30 05:22:24.410910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:24.410721 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4" event={"ID":"b8b567ca-fe72-4ebe-a47a-4df8699bbc55","Type":"ContainerDied","Data":"7341e2e2c6d273c56fd76846cb2f35e6e60148f021fa3da7ab94a119994b1421"} Oct 30 05:22:24.410910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:24.410782 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7341e2e2c6d273c56fd76846cb2f35e6e60148f021fa3da7ab94a119994b1421" Oct 30 05:22:25.942961 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:25.942911 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" podUID="dfe25060-9266-482c-8a47-f2f9e317ecb0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:22:25.959140 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:25.959084 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" podUID="fbe6597b-6a31-4372-a6aa-d6b041b06b56" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:22:25.992423 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:22:25.992387 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:22:30.583286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.583254 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f"] Oct 30 05:22:30.584058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.584042 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b8b567ca-fe72-4ebe-a47a-4df8699bbc55" containerName="revoke-keys" Oct 30 05:22:30.584107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.584059 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b567ca-fe72-4ebe-a47a-4df8699bbc55" containerName="revoke-keys" Oct 30 05:22:30.584214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.584204 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b567ca-fe72-4ebe-a47a-4df8699bbc55" containerName="revoke-keys" Oct 30 05:22:30.588664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.588639 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" Oct 30 05:22:30.606581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.606555 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f"] Oct 30 05:22:30.614146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.614117 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:22:30.614146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.614134 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:22:30.614474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.614447 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:22:30.618243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.618222 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:22:30.758272 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.758236 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4605208-cb14-449a-b590-132cdb64945e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3b14f40-8zw8f\" (UID: \"b4605208-cb14-449a-b590-132cdb64945e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" Oct 30 05:22:30.758447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.758394 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4605208-cb14-449a-b590-132cdb64945e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3b14f40-8zw8f\" (UID: \"b4605208-cb14-449a-b590-132cdb64945e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" Oct 30 05:22:30.758857 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.758829 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-864mx\" (UniqueName: \"kubernetes.io/projected/b4605208-cb14-449a-b590-132cdb64945e-kube-api-access-864mx\") pod \"maas-api-revoke-sub-d3b14f40-8zw8f\" (UID: \"b4605208-cb14-449a-b590-132cdb64945e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" Oct 30 05:22:30.860750 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.860655 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4605208-cb14-449a-b590-132cdb64945e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3b14f40-8zw8f\" (UID: \"b4605208-cb14-449a-b590-132cdb64945e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" Oct 30 05:22:30.860943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.860924 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-864mx\" (UniqueName: \"kubernetes.io/projected/b4605208-cb14-449a-b590-132cdb64945e-kube-api-access-864mx\") pod \"maas-api-revoke-sub-d3b14f40-8zw8f\" (UID: \"b4605208-cb14-449a-b590-132cdb64945e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" Oct 30 05:22:30.861046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.861031 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4605208-cb14-449a-b590-132cdb64945e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3b14f40-8zw8f\" (UID: \"b4605208-cb14-449a-b590-132cdb64945e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" Oct 30 05:22:30.861381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.861347 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4605208-cb14-449a-b590-132cdb64945e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3b14f40-8zw8f\" (UID: \"b4605208-cb14-449a-b590-132cdb64945e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" Oct 30 05:22:30.863552 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.863529 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4605208-cb14-449a-b590-132cdb64945e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3b14f40-8zw8f\" (UID: \"b4605208-cb14-449a-b590-132cdb64945e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" Oct 30 05:22:30.881759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.881732 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-864mx\" (UniqueName: \"kubernetes.io/projected/b4605208-cb14-449a-b590-132cdb64945e-kube-api-access-864mx\") pod \"maas-api-revoke-sub-d3b14f40-8zw8f\" (UID: \"b4605208-cb14-449a-b590-132cdb64945e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" Oct 30 05:22:30.903368 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:30.903323 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" Oct 30 05:22:31.215455 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:22:31.215418 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4605208_cb14_449a_b590_132cdb64945e.slice/crio-bc17bfdc0ba36b72fedc3bbc8158c439487d7fe588c9e8c0e066df80a191ef87 WatchSource:0}: Error finding container bc17bfdc0ba36b72fedc3bbc8158c439487d7fe588c9e8c0e066df80a191ef87: Status 404 returned error can't find the container with id bc17bfdc0ba36b72fedc3bbc8158c439487d7fe588c9e8c0e066df80a191ef87 Oct 30 05:22:31.217410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:31.217384 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f"] Oct 30 05:22:31.474495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:31.474401 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" event={"ID":"b4605208-cb14-449a-b590-132cdb64945e","Type":"ContainerStarted","Data":"bc17bfdc0ba36b72fedc3bbc8158c439487d7fe588c9e8c0e066df80a191ef87"} Oct 30 05:22:32.485475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:32.485387 6587 generic.go:356] "Generic (PLEG): container finished" podID="b4605208-cb14-449a-b590-132cdb64945e" containerID="f5aee60dad9ac1c7f5c2fdf8e04ba0a83414e25dba60c0ac0aab3f584dabebfa" exitCode=0 Oct 30 05:22:32.485850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:32.485474 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" event={"ID":"b4605208-cb14-449a-b590-132cdb64945e","Type":"ContainerDied","Data":"f5aee60dad9ac1c7f5c2fdf8e04ba0a83414e25dba60c0ac0aab3f584dabebfa"} Oct 30 05:22:32.817554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:32.817521 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" Oct 30 05:22:32.846891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:32.846866 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" Oct 30 05:22:32.911180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:32.911134 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe6597b-6a31-4372-a6aa-d6b041b06b56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe6597b-6a31-4372-a6aa-d6b041b06b56-plugins-config-volume\") pod \"fbe6597b-6a31-4372-a6aa-d6b041b06b56\" (UID: \"fbe6597b-6a31-4372-a6aa-d6b041b06b56\") " Oct 30 05:22:32.911412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:32.911208 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe25060-9266-482c-8a47-f2f9e317ecb0-kube-api-access-d779j\" (UniqueName: \"kubernetes.io/projected/dfe25060-9266-482c-8a47-f2f9e317ecb0-kube-api-access-d779j\") pod \"dfe25060-9266-482c-8a47-f2f9e317ecb0\" (UID: \"dfe25060-9266-482c-8a47-f2f9e317ecb0\") " Oct 30 05:22:32.911412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:32.911250 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe6597b-6a31-4372-a6aa-d6b041b06b56-kube-api-access-qdcwc\" (UniqueName: \"kubernetes.io/projected/fbe6597b-6a31-4372-a6aa-d6b041b06b56-kube-api-access-qdcwc\") pod \"fbe6597b-6a31-4372-a6aa-d6b041b06b56\" (UID: \"fbe6597b-6a31-4372-a6aa-d6b041b06b56\") " Oct 30 05:22:32.911715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:32.911688 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe25060-9266-482c-8a47-f2f9e317ecb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe25060-9266-482c-8a47-f2f9e317ecb0-plugins-config-volume\") pod \"dfe25060-9266-482c-8a47-f2f9e317ecb0\" (UID: \"dfe25060-9266-482c-8a47-f2f9e317ecb0\") " Oct 30 05:22:32.913398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:32.913275 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe25060-9266-482c-8a47-f2f9e317ecb0-plugins-config-volume" pod "dfe25060-9266-482c-8a47-f2f9e317ecb0" (UID: "dfe25060-9266-482c-8a47-f2f9e317ecb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:32.916299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:32.915995 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe6597b-6a31-4372-a6aa-d6b041b06b56-plugins-config-volume" pod "fbe6597b-6a31-4372-a6aa-d6b041b06b56" (UID: "fbe6597b-6a31-4372-a6aa-d6b041b06b56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:32.971164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:32.971101 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe25060-9266-482c-8a47-f2f9e317ecb0-kube-api-access-d779j" pod "dfe25060-9266-482c-8a47-f2f9e317ecb0" (UID: "dfe25060-9266-482c-8a47-f2f9e317ecb0"). InnerVolumeSpecName "kube-api-access-d779j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:32.986732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:32.986655 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe6597b-6a31-4372-a6aa-d6b041b06b56-kube-api-access-qdcwc" pod "fbe6597b-6a31-4372-a6aa-d6b041b06b56" (UID: "fbe6597b-6a31-4372-a6aa-d6b041b06b56"). InnerVolumeSpecName "kube-api-access-qdcwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:33.014928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.014890 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe25060-9266-482c-8a47-f2f9e317ecb0-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:33.014928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.014933 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe6597b-6a31-4372-a6aa-d6b041b06b56-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:33.015174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.014949 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d779j\" (UniqueName: \"kubernetes.io/projected/dfe25060-9266-482c-8a47-f2f9e317ecb0-kube-api-access-d779j\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:33.015174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.014963 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdcwc\" (UniqueName: \"kubernetes.io/projected/fbe6597b-6a31-4372-a6aa-d6b041b06b56-kube-api-access-qdcwc\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:33.495805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.495771 6587 generic.go:356] "Generic (PLEG): container finished" podID="fbe6597b-6a31-4372-a6aa-d6b041b06b56" containerID="5aeb9ba2a48924face9a4612f54190f27f195565cb74f25c5bc7afee4eb7d41d" exitCode=0 Oct 30 05:22:33.496291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.495837 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" Oct 30 05:22:33.496291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.495849 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" event={"ID":"fbe6597b-6a31-4372-a6aa-d6b041b06b56","Type":"ContainerDied","Data":"5aeb9ba2a48924face9a4612f54190f27f195565cb74f25c5bc7afee4eb7d41d"} Oct 30 05:22:33.496291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.495942 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj" event={"ID":"fbe6597b-6a31-4372-a6aa-d6b041b06b56","Type":"ContainerDied","Data":"c2ff2dbfe567eba0e0cb63214b9d16f42db7af23fe82c8fa2b2cc1e51603d43e"} Oct 30 05:22:33.496291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.495971 6587 scope.go:122] "RemoveContainer" containerID="5aeb9ba2a48924face9a4612f54190f27f195565cb74f25c5bc7afee4eb7d41d" Oct 30 05:22:33.497645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.497623 6587 generic.go:356] "Generic (PLEG): container finished" podID="dfe25060-9266-482c-8a47-f2f9e317ecb0" containerID="755ac7d6aa832662106632d946eb9eeeac60aaa6b5455aeca2b53f6ca9032980" exitCode=0 Oct 30 05:22:33.497763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.497651 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" event={"ID":"dfe25060-9266-482c-8a47-f2f9e317ecb0","Type":"ContainerDied","Data":"755ac7d6aa832662106632d946eb9eeeac60aaa6b5455aeca2b53f6ca9032980"} Oct 30 05:22:33.497763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.497691 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" Oct 30 05:22:33.497763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.497700 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk" event={"ID":"dfe25060-9266-482c-8a47-f2f9e317ecb0","Type":"ContainerDied","Data":"50377e4f7d59813e0b9659af34553c3a0ca8e0be953a721e7a61cb735d616923"} Oct 30 05:22:33.511737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.511715 6587 scope.go:122] "RemoveContainer" containerID="5aeb9ba2a48924face9a4612f54190f27f195565cb74f25c5bc7afee4eb7d41d" Oct 30 05:22:33.512068 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:22:33.512041 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aeb9ba2a48924face9a4612f54190f27f195565cb74f25c5bc7afee4eb7d41d\": container with ID starting with 5aeb9ba2a48924face9a4612f54190f27f195565cb74f25c5bc7afee4eb7d41d not found: ID does not exist" containerID="5aeb9ba2a48924face9a4612f54190f27f195565cb74f25c5bc7afee4eb7d41d" Oct 30 05:22:33.512183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.512076 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aeb9ba2a48924face9a4612f54190f27f195565cb74f25c5bc7afee4eb7d41d"} err="failed to get container status \"5aeb9ba2a48924face9a4612f54190f27f195565cb74f25c5bc7afee4eb7d41d\": rpc error: code = NotFound desc = could not find container \"5aeb9ba2a48924face9a4612f54190f27f195565cb74f25c5bc7afee4eb7d41d\": container with ID starting with 5aeb9ba2a48924face9a4612f54190f27f195565cb74f25c5bc7afee4eb7d41d not found: ID does not exist" Oct 30 05:22:33.512183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.512094 6587 scope.go:122] "RemoveContainer" containerID="755ac7d6aa832662106632d946eb9eeeac60aaa6b5455aeca2b53f6ca9032980" Oct 30 05:22:33.525432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.525410 6587 scope.go:122] "RemoveContainer" containerID="755ac7d6aa832662106632d946eb9eeeac60aaa6b5455aeca2b53f6ca9032980" Oct 30 05:22:33.525763 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:22:33.525731 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755ac7d6aa832662106632d946eb9eeeac60aaa6b5455aeca2b53f6ca9032980\": container with ID starting with 755ac7d6aa832662106632d946eb9eeeac60aaa6b5455aeca2b53f6ca9032980 not found: ID does not exist" containerID="755ac7d6aa832662106632d946eb9eeeac60aaa6b5455aeca2b53f6ca9032980" Oct 30 05:22:33.525838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.525761 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755ac7d6aa832662106632d946eb9eeeac60aaa6b5455aeca2b53f6ca9032980"} err="failed to get container status \"755ac7d6aa832662106632d946eb9eeeac60aaa6b5455aeca2b53f6ca9032980\": rpc error: code = NotFound desc = could not find container \"755ac7d6aa832662106632d946eb9eeeac60aaa6b5455aeca2b53f6ca9032980\": container with ID starting with 755ac7d6aa832662106632d946eb9eeeac60aaa6b5455aeca2b53f6ca9032980 not found: ID does not exist" Oct 30 05:22:33.555530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.555502 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk"] Oct 30 05:22:33.559862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.559835 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk"] Oct 30 05:22:33.590926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.590891 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe25060-9266-482c-8a47-f2f9e317ecb0" path="/var/lib/kubelet/pods/dfe25060-9266-482c-8a47-f2f9e317ecb0/volumes" Oct 30 05:22:33.606767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.606734 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj"] Oct 30 05:22:33.611045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.611019 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj"] Oct 30 05:22:33.684644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.684620 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" Oct 30 05:22:33.725351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.725215 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4605208-cb14-449a-b590-132cdb64945e-kube-api-access-864mx\" (UniqueName: \"kubernetes.io/projected/b4605208-cb14-449a-b590-132cdb64945e-kube-api-access-864mx\") pod \"b4605208-cb14-449a-b590-132cdb64945e\" (UID: \"b4605208-cb14-449a-b590-132cdb64945e\") " Oct 30 05:22:33.725995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.725968 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4605208-cb14-449a-b590-132cdb64945e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4605208-cb14-449a-b590-132cdb64945e-maas-api-openshift-ca\") pod \"b4605208-cb14-449a-b590-132cdb64945e\" (UID: \"b4605208-cb14-449a-b590-132cdb64945e\") " Oct 30 05:22:33.726161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.726075 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4605208-cb14-449a-b590-132cdb64945e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4605208-cb14-449a-b590-132cdb64945e-maas-api-xks-ca\") pod \"b4605208-cb14-449a-b590-132cdb64945e\" (UID: \"b4605208-cb14-449a-b590-132cdb64945e\") " Oct 30 05:22:33.726587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.726452 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4605208-cb14-449a-b590-132cdb64945e-maas-api-openshift-ca" pod "b4605208-cb14-449a-b590-132cdb64945e" (UID: "b4605208-cb14-449a-b590-132cdb64945e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:33.727687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.727627 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4605208-cb14-449a-b590-132cdb64945e-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:33.729291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.729110 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4605208-cb14-449a-b590-132cdb64945e-kube-api-access-864mx" pod "b4605208-cb14-449a-b590-132cdb64945e" (UID: "b4605208-cb14-449a-b590-132cdb64945e"). InnerVolumeSpecName "kube-api-access-864mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:33.732857 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.732829 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4605208-cb14-449a-b590-132cdb64945e-maas-api-xks-ca" pod "b4605208-cb14-449a-b590-132cdb64945e" (UID: "b4605208-cb14-449a-b590-132cdb64945e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:22:33.829071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.829029 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4605208-cb14-449a-b590-132cdb64945e-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:33.829071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:33.829062 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-864mx\" (UniqueName: \"kubernetes.io/projected/b4605208-cb14-449a-b590-132cdb64945e-kube-api-access-864mx\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:34.511093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:34.511063 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" Oct 30 05:22:34.511093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:34.511072 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f" event={"ID":"b4605208-cb14-449a-b590-132cdb64945e","Type":"ContainerDied","Data":"bc17bfdc0ba36b72fedc3bbc8158c439487d7fe588c9e8c0e066df80a191ef87"} Oct 30 05:22:34.511809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:34.511130 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc17bfdc0ba36b72fedc3bbc8158c439487d7fe588c9e8c0e066df80a191ef87" Oct 30 05:22:35.585915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:35.585881 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe6597b-6a31-4372-a6aa-d6b041b06b56" path="/var/lib/kubelet/pods/fbe6597b-6a31-4372-a6aa-d6b041b06b56/volumes" Oct 30 05:22:36.689965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:36.689926 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt"] Oct 30 05:22:36.706631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:36.706591 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a8f18-jr2tt"] Oct 30 05:22:37.584701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:37.584665 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04975143-7d13-480c-b880-4c317015e96b" path="/var/lib/kubelet/pods/04975143-7d13-480c-b880-4c317015e96b/volumes" Oct 30 05:22:43.653640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:43.653607 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj"] Oct 30 05:22:43.654158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:43.653864 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" podUID="fd2824ca-4137-4f91-90c8-21e086a336d9" containerName="istio-proxy" containerID="cri-o://b78ec7da36f9f4bd659b1e82422a10945314bd5a50d6937a3a9d6704e929716c" gracePeriod=30 Oct 30 05:22:45.750640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:45.749625 6587 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:22:45.750640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:45.749704 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" podUID="fd2824ca-4137-4f91-90c8-21e086a336d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:22:47.985372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:47.985327 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr"] Oct 30 05:22:47.986277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:47.986254 6587 cpu_manager.go:414] "containerMap: removing container" podUID="dfe25060-9266-482c-8a47-f2f9e317ecb0" containerName="payload-pre-processing" Oct 30 05:22:47.986277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:47.986276 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe25060-9266-482c-8a47-f2f9e317ecb0" containerName="payload-pre-processing" Oct 30 05:22:47.986483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:47.986293 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fbe6597b-6a31-4372-a6aa-d6b041b06b56" containerName="payload-processing" Oct 30 05:22:47.986483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:47.986298 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe6597b-6a31-4372-a6aa-d6b041b06b56" containerName="payload-processing" Oct 30 05:22:47.986483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:47.986316 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b4605208-cb14-449a-b590-132cdb64945e" containerName="revoke-keys" Oct 30 05:22:47.986483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:47.986321 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4605208-cb14-449a-b590-132cdb64945e" containerName="revoke-keys" Oct 30 05:22:47.986483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:47.986466 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe25060-9266-482c-8a47-f2f9e317ecb0" containerName="payload-pre-processing" Oct 30 05:22:47.986483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:47.986478 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe6597b-6a31-4372-a6aa-d6b041b06b56" containerName="payload-processing" Oct 30 05:22:47.986483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:47.986486 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4605208-cb14-449a-b590-132cdb64945e" containerName="revoke-keys" Oct 30 05:22:47.992810 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:47.992787 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.016431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.016400 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr"] Oct 30 05:22:48.022490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.022466 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-674082-openshift-default-dockercfg-g6ls2\"" Oct 30 05:22:48.100648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.100612 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-podinfo\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.101088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.101041 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-data\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.101261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.101113 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-token\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.101338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.101285 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-credential-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.101476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.101420 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgnxh\" (UniqueName: \"kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-kube-api-access-sgnxh\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.101544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.101491 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-envoy\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.101603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.101539 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-certs\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.101603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.101577 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istiod-ca-cert\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.101732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.101604 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.111754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.111727 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27"] Oct 30 05:22:48.203855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.203818 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-data\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.204042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.203897 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-token\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.204093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.204054 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-credential-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.204146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.204100 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgnxh\" (UniqueName: \"kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-kube-api-access-sgnxh\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.204206 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.204158 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-envoy\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.204265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.204206 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-certs\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.204265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.204216 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-data\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.204265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.204243 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istiod-ca-cert\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.204487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.204273 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.204551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.204480 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-podinfo\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.204551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.204509 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-credential-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.204657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.204622 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-certs\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.204908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.204887 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.205067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.205047 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istiod-ca-cert\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.206540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.206519 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-envoy\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.206947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.206928 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-podinfo\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.276435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.273979 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgnxh\" (UniqueName: \"kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-kube-api-access-sgnxh\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.306745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.301422 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-token\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.329381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.316224 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:48.500992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.500918 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr"] Oct 30 05:22:48.589376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.562430 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-356cae-bxtrr"] Oct 30 05:22:48.745016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.744989 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr"] Oct 30 05:22:48.771411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.771379 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:22:48.771515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.771449 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:22:48.771515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.771477 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:22:48.956106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:48.956078 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:22:49.042273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.042213 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-kube-api-access-f8ttt\" (UniqueName: \"kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-kube-api-access-f8ttt\") pod \"fd2824ca-4137-4f91-90c8-21e086a336d9\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " Oct 30 05:22:49.042784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.042751 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-data\") pod \"fd2824ca-4137-4f91-90c8-21e086a336d9\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " Oct 30 05:22:49.042864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.042813 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-envoy\") pod \"fd2824ca-4137-4f91-90c8-21e086a336d9\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " Oct 30 05:22:49.043029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.043003 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-token\" (UniqueName: \"kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-token\") pod \"fd2824ca-4137-4f91-90c8-21e086a336d9\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " Oct 30 05:22:49.043136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.043112 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-socket\") pod \"fd2824ca-4137-4f91-90c8-21e086a336d9\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " Oct 30 05:22:49.043200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.043186 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2824ca-4137-4f91-90c8-21e086a336d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2824ca-4137-4f91-90c8-21e086a336d9-istiod-ca-cert\") pod \"fd2824ca-4137-4f91-90c8-21e086a336d9\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " Oct 30 05:22:49.043269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.043251 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-credential-socket\") pod \"fd2824ca-4137-4f91-90c8-21e086a336d9\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " Oct 30 05:22:49.043384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.043304 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-podinfo\") pod \"fd2824ca-4137-4f91-90c8-21e086a336d9\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " Oct 30 05:22:49.043461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.043407 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-certs\") pod \"fd2824ca-4137-4f91-90c8-21e086a336d9\" (UID: \"fd2824ca-4137-4f91-90c8-21e086a336d9\") " Oct 30 05:22:49.044695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.044663 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-credential-socket" pod "fd2824ca-4137-4f91-90c8-21e086a336d9" (UID: "fd2824ca-4137-4f91-90c8-21e086a336d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:49.044885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.044863 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-data" pod "fd2824ca-4137-4f91-90c8-21e086a336d9" (UID: "fd2824ca-4137-4f91-90c8-21e086a336d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:49.045193 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.045082 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-socket" pod "fd2824ca-4137-4f91-90c8-21e086a336d9" (UID: "fd2824ca-4137-4f91-90c8-21e086a336d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:49.045294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.045271 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-certs" pod "fd2824ca-4137-4f91-90c8-21e086a336d9" (UID: "fd2824ca-4137-4f91-90c8-21e086a336d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:49.045426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.045397 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2824ca-4137-4f91-90c8-21e086a336d9-istiod-ca-cert" pod "fd2824ca-4137-4f91-90c8-21e086a336d9" (UID: "fd2824ca-4137-4f91-90c8-21e086a336d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:22:49.047995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.047968 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-token" pod "fd2824ca-4137-4f91-90c8-21e086a336d9" (UID: "fd2824ca-4137-4f91-90c8-21e086a336d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:49.060856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.060830 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-kube-api-access-f8ttt" pod "fd2824ca-4137-4f91-90c8-21e086a336d9" (UID: "fd2824ca-4137-4f91-90c8-21e086a336d9"). InnerVolumeSpecName "kube-api-access-f8ttt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:49.061282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.061251 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-envoy" pod "fd2824ca-4137-4f91-90c8-21e086a336d9" (UID: "fd2824ca-4137-4f91-90c8-21e086a336d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:49.061417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.061342 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-podinfo" pod "fd2824ca-4137-4f91-90c8-21e086a336d9" (UID: "fd2824ca-4137-4f91-90c8-21e086a336d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:22:49.145750 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.145671 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.145964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.145936 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.145964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.145965 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2824ca-4137-4f91-90c8-21e086a336d9-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.148476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.145979 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.148476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.145993 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.148476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.146006 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.148476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.146018 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8ttt\" (UniqueName: \"kubernetes.io/projected/fd2824ca-4137-4f91-90c8-21e086a336d9-kube-api-access-f8ttt\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.148476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.146031 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.148476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.146044 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2824ca-4137-4f91-90c8-21e086a336d9-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:22:49.608473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.608439 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48331a9d-5dc6-4711-a162-c22f7ab8c1b2" path="/var/lib/kubelet/pods/48331a9d-5dc6-4711-a162-c22f7ab8c1b2/volumes" Oct 30 05:22:49.742461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.742409 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" event={"ID":"cfc65f69-cd06-4575-9e52-29f042c2bdc7","Type":"ContainerStarted","Data":"052301a58b96cf1fd15e0abddb5a44657e019d60019881734c819567f07d7bd6"} Oct 30 05:22:49.742666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.742484 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" event={"ID":"cfc65f69-cd06-4575-9e52-29f042c2bdc7","Type":"ContainerStarted","Data":"e856e9426e0f6b5fc2405d4df27eca5a4fe59c71417d9f20a3fbe30e0ae11a5c"} Oct 30 05:22:49.744508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.744475 6587 generic.go:356] "Generic (PLEG): container finished" podID="fd2824ca-4137-4f91-90c8-21e086a336d9" containerID="b78ec7da36f9f4bd659b1e82422a10945314bd5a50d6937a3a9d6704e929716c" exitCode=0 Oct 30 05:22:49.744767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.744737 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" event={"ID":"fd2824ca-4137-4f91-90c8-21e086a336d9","Type":"ContainerDied","Data":"b78ec7da36f9f4bd659b1e82422a10945314bd5a50d6937a3a9d6704e929716c"} Oct 30 05:22:49.744925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.744793 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" event={"ID":"fd2824ca-4137-4f91-90c8-21e086a336d9","Type":"ContainerDied","Data":"fed4d4d692cb255bf191ef87e24ac81232e9ee3b987e2599f4c81f76fab1b098"} Oct 30 05:22:49.744925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.744821 6587 scope.go:122] "RemoveContainer" containerID="b78ec7da36f9f4bd659b1e82422a10945314bd5a50d6937a3a9d6704e929716c" Oct 30 05:22:49.745052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.744973 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj" Oct 30 05:22:49.761426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.761399 6587 scope.go:122] "RemoveContainer" containerID="b78ec7da36f9f4bd659b1e82422a10945314bd5a50d6937a3a9d6704e929716c" Oct 30 05:22:49.761787 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:22:49.761765 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78ec7da36f9f4bd659b1e82422a10945314bd5a50d6937a3a9d6704e929716c\": container with ID starting with b78ec7da36f9f4bd659b1e82422a10945314bd5a50d6937a3a9d6704e929716c not found: ID does not exist" containerID="b78ec7da36f9f4bd659b1e82422a10945314bd5a50d6937a3a9d6704e929716c" Oct 30 05:22:49.761855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.761796 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78ec7da36f9f4bd659b1e82422a10945314bd5a50d6937a3a9d6704e929716c"} err="failed to get container status \"b78ec7da36f9f4bd659b1e82422a10945314bd5a50d6937a3a9d6704e929716c\": rpc error: code = NotFound desc = could not find container \"b78ec7da36f9f4bd659b1e82422a10945314bd5a50d6937a3a9d6704e929716c\": container with ID starting with b78ec7da36f9f4bd659b1e82422a10945314bd5a50d6937a3a9d6704e929716c not found: ID does not exist" Oct 30 05:22:49.788408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.788344 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" podStartSLOduration=2.788331851 podStartE2EDuration="2.788331851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:22:49.786852461 +0000 UTC m=+2294.314619247" watchObservedRunningTime="2026-10-30 05:22:49.788331851 +0000 UTC m=+2294.316098636" Oct 30 05:22:49.831267 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.831231 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj"] Oct 30 05:22:49.837809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:49.837776 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj"] Oct 30 05:22:50.316732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:50.316702 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:50.324473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:50.324447 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:50.755194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:50.755105 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:50.756219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:50.756198 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:22:51.585524 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:51.585491 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2824ca-4137-4f91-90c8-21e086a336d9" path="/var/lib/kubelet/pods/fd2824ca-4137-4f91-90c8-21e086a336d9/volumes" Oct 30 05:22:55.593984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:55.593946 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9"] Oct 30 05:22:55.594471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:55.594199 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" podUID="dd91a815-98b4-496a-885d-7a0b1d3e4369" containerName="istio-proxy" containerID="cri-o://ab98be8848449e092352989b59c94bea44699b414ba51761caf6b1fb3685801a" gracePeriod=30 Oct 30 05:22:55.859989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:55.859800 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz"] Oct 30 05:22:55.861143 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:55.861115 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fd2824ca-4137-4f91-90c8-21e086a336d9" containerName="istio-proxy" Oct 30 05:22:55.861143 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:55.861142 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2824ca-4137-4f91-90c8-21e086a336d9" containerName="istio-proxy" Oct 30 05:22:55.861433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:55.861417 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2824ca-4137-4f91-90c8-21e086a336d9" containerName="istio-proxy" Oct 30 05:22:55.868345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:55.868325 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" Oct 30 05:22:55.882413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:55.882004 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz"] Oct 30 05:22:55.949380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:55.948698 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6b3855-dockercfg-h5gxz\"" Oct 30 05:22:55.949380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:55.948809 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6b3855\"" Oct 30 05:22:56.023519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.023477 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktm7n\" (UniqueName: \"kubernetes.io/projected/65bfb1ee-74e7-4fb4-8738-607e42dec925-kube-api-access-ktm7n\") pod \"payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz\" (UID: \"65bfb1ee-74e7-4fb4-8738-607e42dec925\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" Oct 30 05:22:56.060382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.034716 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bfb1ee-74e7-4fb4-8738-607e42dec925-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz\" (UID: \"65bfb1ee-74e7-4fb4-8738-607e42dec925\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" Oct 30 05:22:56.164931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.164842 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktm7n\" (UniqueName: \"kubernetes.io/projected/65bfb1ee-74e7-4fb4-8738-607e42dec925-kube-api-access-ktm7n\") pod \"payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz\" (UID: \"65bfb1ee-74e7-4fb4-8738-607e42dec925\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" Oct 30 05:22:56.165116 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.165036 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bfb1ee-74e7-4fb4-8738-607e42dec925-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz\" (UID: \"65bfb1ee-74e7-4fb4-8738-607e42dec925\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" Oct 30 05:22:56.165927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.165771 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bfb1ee-74e7-4fb4-8738-607e42dec925-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz\" (UID: \"65bfb1ee-74e7-4fb4-8738-607e42dec925\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" Oct 30 05:22:56.170908 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:22:56.170872 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:22:56.171064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.171044 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v"] Oct 30 05:22:56.175965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.175947 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" Oct 30 05:22:56.195047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.192056 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:22:56.195047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.192168 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:22:56.195047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.192246 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:22:56.195047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.193458 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktm7n\" (UniqueName: \"kubernetes.io/projected/65bfb1ee-74e7-4fb4-8738-607e42dec925-kube-api-access-ktm7n\") pod \"payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz\" (UID: \"65bfb1ee-74e7-4fb4-8738-607e42dec925\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" Oct 30 05:22:56.195047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.193742 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:22:56.199379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.199341 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v"] Oct 30 05:22:56.266519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.266487 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6slsh\" (UniqueName: \"kubernetes.io/projected/3eb62f15-762d-4625-b471-a07aad8c8aa7-kube-api-access-6slsh\") pod \"maas-api-revoke-keys-e2e-inf-b-e16297-zr26v\" (UID: \"3eb62f15-762d-4625-b471-a07aad8c8aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" Oct 30 05:22:56.266701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.266638 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e16297-zr26v\" (UID: \"3eb62f15-762d-4625-b471-a07aad8c8aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" Oct 30 05:22:56.266701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.266690 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e16297-zr26v\" (UID: \"3eb62f15-762d-4625-b471-a07aad8c8aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" Oct 30 05:22:56.368038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.368003 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6slsh\" (UniqueName: \"kubernetes.io/projected/3eb62f15-762d-4625-b471-a07aad8c8aa7-kube-api-access-6slsh\") pod \"maas-api-revoke-keys-e2e-inf-b-e16297-zr26v\" (UID: \"3eb62f15-762d-4625-b471-a07aad8c8aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" Oct 30 05:22:56.368241 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.368178 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e16297-zr26v\" (UID: \"3eb62f15-762d-4625-b471-a07aad8c8aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" Oct 30 05:22:56.368326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.368256 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e16297-zr26v\" (UID: \"3eb62f15-762d-4625-b471-a07aad8c8aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" Oct 30 05:22:56.368949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.368924 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e16297-zr26v\" (UID: \"3eb62f15-762d-4625-b471-a07aad8c8aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" Oct 30 05:22:56.370745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.370725 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e16297-zr26v\" (UID: \"3eb62f15-762d-4625-b471-a07aad8c8aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" Oct 30 05:22:56.388311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.388286 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6slsh\" (UniqueName: \"kubernetes.io/projected/3eb62f15-762d-4625-b471-a07aad8c8aa7-kube-api-access-6slsh\") pod \"maas-api-revoke-keys-e2e-inf-b-e16297-zr26v\" (UID: \"3eb62f15-762d-4625-b471-a07aad8c8aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" Oct 30 05:22:56.492047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.491967 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" Oct 30 05:22:56.532284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.530942 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" Oct 30 05:22:56.879865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.879775 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz"] Oct 30 05:22:56.881825 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:22:56.881794 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bfb1ee_74e7_4fb4_8738_607e42dec925.slice/crio-da0f13e32507c9ae449e83f6e102644bdee266f9d5de76d73e7881e97c3d3f40 WatchSource:0}: Error finding container da0f13e32507c9ae449e83f6e102644bdee266f9d5de76d73e7881e97c3d3f40: Status 404 returned error can't find the container with id da0f13e32507c9ae449e83f6e102644bdee266f9d5de76d73e7881e97c3d3f40 Oct 30 05:22:56.886672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:56.886640 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v"] Oct 30 05:22:57.821724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:57.821679 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" event={"ID":"3eb62f15-762d-4625-b471-a07aad8c8aa7","Type":"ContainerStarted","Data":"24d520ed0e9677ce6902df1b5ff6087f8ce6dd67305be264f45159cff5678581"} Oct 30 05:22:57.821912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:57.821751 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" event={"ID":"3eb62f15-762d-4625-b471-a07aad8c8aa7","Type":"ContainerStarted","Data":"da051cfa5b7ba28143986c7c935260d396e369fbefb12763867fe834f3493db5"} Oct 30 05:22:57.824108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:57.824080 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" event={"ID":"65bfb1ee-74e7-4fb4-8738-607e42dec925","Type":"ContainerStarted","Data":"03e7ddc4d643ad06d18252a96f89e0a293621c0856ccf4d78d25f365a511018b"} Oct 30 05:22:57.824272 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:57.824132 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" event={"ID":"65bfb1ee-74e7-4fb4-8738-607e42dec925","Type":"ContainerStarted","Data":"da0f13e32507c9ae449e83f6e102644bdee266f9d5de76d73e7881e97c3d3f40"} Oct 30 05:22:57.824393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:57.824372 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" Oct 30 05:22:57.867284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:57.867235 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" podStartSLOduration=1.8672227860000001 podStartE2EDuration="1.867222786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:22:57.864694163 +0000 UTC m=+2302.392460950" watchObservedRunningTime="2026-10-30 05:22:57.867222786 +0000 UTC m=+2302.394989571" Oct 30 05:22:57.909027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:57.908976 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" podStartSLOduration=2.908963564 podStartE2EDuration="2.908963564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:22:57.90586127 +0000 UTC m=+2302.433628059" watchObservedRunningTime="2026-10-30 05:22:57.908963564 +0000 UTC m=+2302.436730350" Oct 30 05:22:58.837549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:58.837514 6587 generic.go:356] "Generic (PLEG): container finished" podID="3eb62f15-762d-4625-b471-a07aad8c8aa7" containerID="24d520ed0e9677ce6902df1b5ff6087f8ce6dd67305be264f45159cff5678581" exitCode=0 Oct 30 05:22:58.837754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:58.837595 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" event={"ID":"3eb62f15-762d-4625-b471-a07aad8c8aa7","Type":"ContainerDied","Data":"24d520ed0e9677ce6902df1b5ff6087f8ce6dd67305be264f45159cff5678581"} Oct 30 05:22:59.030057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:59.030019 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b"] Oct 30 05:22:59.034193 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:59.034166 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8efdc2-tw82b"] Oct 30 05:22:59.584934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:22:59.584899 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9368d63-2c7a-4b6e-a818-e52d05407199" path="/var/lib/kubelet/pods/e9368d63-2c7a-4b6e-a818-e52d05407199/volumes" Oct 30 05:23:00.204064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.204039 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" Oct 30 05:23:00.262380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.260716 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-openshift-ca\") pod \"3eb62f15-762d-4625-b471-a07aad8c8aa7\" (UID: \"3eb62f15-762d-4625-b471-a07aad8c8aa7\") " Oct 30 05:23:00.262380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.260903 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-xks-ca\") pod \"3eb62f15-762d-4625-b471-a07aad8c8aa7\" (UID: \"3eb62f15-762d-4625-b471-a07aad8c8aa7\") " Oct 30 05:23:00.262380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.260949 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb62f15-762d-4625-b471-a07aad8c8aa7-kube-api-access-6slsh\" (UniqueName: \"kubernetes.io/projected/3eb62f15-762d-4625-b471-a07aad8c8aa7-kube-api-access-6slsh\") pod \"3eb62f15-762d-4625-b471-a07aad8c8aa7\" (UID: \"3eb62f15-762d-4625-b471-a07aad8c8aa7\") " Oct 30 05:23:00.262765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.262716 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-openshift-ca" pod "3eb62f15-762d-4625-b471-a07aad8c8aa7" (UID: "3eb62f15-762d-4625-b471-a07aad8c8aa7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:00.277558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.274151 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb62f15-762d-4625-b471-a07aad8c8aa7-kube-api-access-6slsh" pod "3eb62f15-762d-4625-b471-a07aad8c8aa7" (UID: "3eb62f15-762d-4625-b471-a07aad8c8aa7"). InnerVolumeSpecName "kube-api-access-6slsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:00.277558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.275829 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-xks-ca" pod "3eb62f15-762d-4625-b471-a07aad8c8aa7" (UID: "3eb62f15-762d-4625-b471-a07aad8c8aa7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:23:00.364329 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.364292 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:00.364329 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.364326 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eb62f15-762d-4625-b471-a07aad8c8aa7-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:00.364556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.364339 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6slsh\" (UniqueName: \"kubernetes.io/projected/3eb62f15-762d-4625-b471-a07aad8c8aa7-kube-api-access-6slsh\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:00.879620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.879594 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" Oct 30 05:23:00.879762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.879630 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v" event={"ID":"3eb62f15-762d-4625-b471-a07aad8c8aa7","Type":"ContainerDied","Data":"da051cfa5b7ba28143986c7c935260d396e369fbefb12763867fe834f3493db5"} Oct 30 05:23:00.879762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.879692 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da051cfa5b7ba28143986c7c935260d396e369fbefb12763867fe834f3493db5" Oct 30 05:23:00.881263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.881240 6587 generic.go:356] "Generic (PLEG): container finished" podID="dd91a815-98b4-496a-885d-7a0b1d3e4369" containerID="ab98be8848449e092352989b59c94bea44699b414ba51761caf6b1fb3685801a" exitCode=0 Oct 30 05:23:00.881463 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.881320 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" event={"ID":"dd91a815-98b4-496a-885d-7a0b1d3e4369","Type":"ContainerDied","Data":"ab98be8848449e092352989b59c94bea44699b414ba51761caf6b1fb3685801a"} Oct 30 05:23:00.885713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.885694 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:23:00.972764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.972731 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-socket\") pod \"dd91a815-98b4-496a-885d-7a0b1d3e4369\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " Oct 30 05:23:00.972919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.972775 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-data\") pod \"dd91a815-98b4-496a-885d-7a0b1d3e4369\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " Oct 30 05:23:00.972919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.972813 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-certs\") pod \"dd91a815-98b4-496a-885d-7a0b1d3e4369\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " Oct 30 05:23:00.973023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.972961 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-kube-api-access-t2ltn\" (UniqueName: \"kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-kube-api-access-t2ltn\") pod \"dd91a815-98b4-496a-885d-7a0b1d3e4369\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " Oct 30 05:23:00.973066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.973021 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-socket" pod "dd91a815-98b4-496a-885d-7a0b1d3e4369" (UID: "dd91a815-98b4-496a-885d-7a0b1d3e4369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:00.973106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.973048 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-podinfo\") pod \"dd91a815-98b4-496a-885d-7a0b1d3e4369\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " Oct 30 05:23:00.973146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.973102 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-data" pod "dd91a815-98b4-496a-885d-7a0b1d3e4369" (UID: "dd91a815-98b4-496a-885d-7a0b1d3e4369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:00.973146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.973125 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-certs" pod "dd91a815-98b4-496a-885d-7a0b1d3e4369" (UID: "dd91a815-98b4-496a-885d-7a0b1d3e4369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:00.973243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.973144 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-token\" (UniqueName: \"kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-token\") pod \"dd91a815-98b4-496a-885d-7a0b1d3e4369\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " Oct 30 05:23:00.973243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.973211 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-credential-socket\") pod \"dd91a815-98b4-496a-885d-7a0b1d3e4369\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " Oct 30 05:23:00.973375 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.973264 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-envoy\") pod \"dd91a815-98b4-496a-885d-7a0b1d3e4369\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " Oct 30 05:23:00.973375 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.973300 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd91a815-98b4-496a-885d-7a0b1d3e4369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd91a815-98b4-496a-885d-7a0b1d3e4369-istiod-ca-cert\") pod \"dd91a815-98b4-496a-885d-7a0b1d3e4369\" (UID: \"dd91a815-98b4-496a-885d-7a0b1d3e4369\") " Oct 30 05:23:00.973697 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.973673 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd91a815-98b4-496a-885d-7a0b1d3e4369-istiod-ca-cert" pod "dd91a815-98b4-496a-885d-7a0b1d3e4369" (UID: "dd91a815-98b4-496a-885d-7a0b1d3e4369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:00.974095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.974076 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd91a815-98b4-496a-885d-7a0b1d3e4369-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:00.974207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.974099 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:00.974207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.974115 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:00.974207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.974127 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:00.974492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.974410 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-credential-socket" pod "dd91a815-98b4-496a-885d-7a0b1d3e4369" (UID: "dd91a815-98b4-496a-885d-7a0b1d3e4369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:00.975446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.975425 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-podinfo" pod "dd91a815-98b4-496a-885d-7a0b1d3e4369" (UID: "dd91a815-98b4-496a-885d-7a0b1d3e4369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:23:00.975770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.975750 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-kube-api-access-t2ltn" pod "dd91a815-98b4-496a-885d-7a0b1d3e4369" (UID: "dd91a815-98b4-496a-885d-7a0b1d3e4369"). InnerVolumeSpecName "kube-api-access-t2ltn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:00.975848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.975756 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-envoy" pod "dd91a815-98b4-496a-885d-7a0b1d3e4369" (UID: "dd91a815-98b4-496a-885d-7a0b1d3e4369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:00.975848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:00.975788 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-token" pod "dd91a815-98b4-496a-885d-7a0b1d3e4369" (UID: "dd91a815-98b4-496a-885d-7a0b1d3e4369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:01.075826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:01.075797 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:01.075826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:01.075823 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:01.075826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:01.075832 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:01.076058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:01.075842 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2ltn\" (UniqueName: \"kubernetes.io/projected/dd91a815-98b4-496a-885d-7a0b1d3e4369-kube-api-access-t2ltn\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:01.076058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:01.075852 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd91a815-98b4-496a-885d-7a0b1d3e4369-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:01.892757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:01.892681 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" Oct 30 05:23:01.892757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:01.892696 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9" event={"ID":"dd91a815-98b4-496a-885d-7a0b1d3e4369","Type":"ContainerDied","Data":"9b7f02897e8137bfd0c0919827468f72dc9dd49e83a7c79f6453d9bf14638d78"} Oct 30 05:23:01.893281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:01.892761 6587 scope.go:122] "RemoveContainer" containerID="ab98be8848449e092352989b59c94bea44699b414ba51761caf6b1fb3685801a" Oct 30 05:23:01.966238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:01.966195 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9"] Oct 30 05:23:01.978268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:01.978239 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9"] Oct 30 05:23:03.141164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.141095 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" podUID="e9325b4e-ca66-4cff-9104-ab1959ef8e8c" containerName="main" containerID="cri-o://51a64992d018b04be588426e3f21c5039eefc4df31337060f5e24a5cee38c8f0" gracePeriod=45 Oct 30 05:23:03.452192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.452160 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:23:03.531099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.531051 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tmp-dir\") pod \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " Oct 30 05:23:03.531292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.531276 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-kube-api-access-t5szm\" (UniqueName: \"kubernetes.io/projected/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-kube-api-access-t5szm\") pod \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " Oct 30 05:23:03.531402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.531376 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tmp-dir" pod "e9325b4e-ca66-4cff-9104-ab1959ef8e8c" (UID: "e9325b4e-ca66-4cff-9104-ab1959ef8e8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:03.531484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.531415 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-dshm\") pod \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " Oct 30 05:23:03.531484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.531463 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-model-cache\") pod \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " Oct 30 05:23:03.531609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.531585 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tls-certs\") pod \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " Oct 30 05:23:03.531705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.531657 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-home\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-home\") pod \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\" (UID: \"e9325b4e-ca66-4cff-9104-ab1959ef8e8c\") " Oct 30 05:23:03.535932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.532398 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:03.535932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.532596 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-home" pod "e9325b4e-ca66-4cff-9104-ab1959ef8e8c" (UID: "e9325b4e-ca66-4cff-9104-ab1959ef8e8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:03.535932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.534279 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-model-cache" pod "e9325b4e-ca66-4cff-9104-ab1959ef8e8c" (UID: "e9325b4e-ca66-4cff-9104-ab1959ef8e8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:03.537028 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.536989 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tls-certs" pod "e9325b4e-ca66-4cff-9104-ab1959ef8e8c" (UID: "e9325b4e-ca66-4cff-9104-ab1959ef8e8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:23:03.537156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.537028 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-kube-api-access-t5szm" pod "e9325b4e-ca66-4cff-9104-ab1959ef8e8c" (UID: "e9325b4e-ca66-4cff-9104-ab1959ef8e8c"). InnerVolumeSpecName "kube-api-access-t5szm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:03.538428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.538398 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-dshm" pod "e9325b4e-ca66-4cff-9104-ab1959ef8e8c" (UID: "e9325b4e-ca66-4cff-9104-ab1959ef8e8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:03.590529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.590477 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd91a815-98b4-496a-885d-7a0b1d3e4369" path="/var/lib/kubelet/pods/dd91a815-98b4-496a-885d-7a0b1d3e4369/volumes" Oct 30 05:23:03.634160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.634132 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:03.634160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.634158 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:03.634160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.634169 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5szm\" (UniqueName: \"kubernetes.io/projected/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-kube-api-access-t5szm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:03.634724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.634179 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:03.634724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.634188 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9325b4e-ca66-4cff-9104-ab1959ef8e8c-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:03.974480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.971095 6587 generic.go:356] "Generic (PLEG): container finished" podID="e9325b4e-ca66-4cff-9104-ab1959ef8e8c" containerID="51a64992d018b04be588426e3f21c5039eefc4df31337060f5e24a5cee38c8f0" exitCode=0 Oct 30 05:23:03.974480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.971184 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" event={"ID":"e9325b4e-ca66-4cff-9104-ab1959ef8e8c","Type":"ContainerDied","Data":"51a64992d018b04be588426e3f21c5039eefc4df31337060f5e24a5cee38c8f0"} Oct 30 05:23:03.974480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.971219 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" event={"ID":"e9325b4e-ca66-4cff-9104-ab1959ef8e8c","Type":"ContainerDied","Data":"705c153798536bee07da554a627db2a624116744a64ebf389e707288bb667d89"} Oct 30 05:23:03.974480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.971241 6587 scope.go:122] "RemoveContainer" containerID="51a64992d018b04be588426e3f21c5039eefc4df31337060f5e24a5cee38c8f0" Oct 30 05:23:03.974480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:03.971423 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27" Oct 30 05:23:04.020276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:04.019577 6587 scope.go:122] "RemoveContainer" containerID="51a64992d018b04be588426e3f21c5039eefc4df31337060f5e24a5cee38c8f0" Oct 30 05:23:04.020276 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:23:04.020000 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a64992d018b04be588426e3f21c5039eefc4df31337060f5e24a5cee38c8f0\": container with ID starting with 51a64992d018b04be588426e3f21c5039eefc4df31337060f5e24a5cee38c8f0 not found: ID does not exist" containerID="51a64992d018b04be588426e3f21c5039eefc4df31337060f5e24a5cee38c8f0" Oct 30 05:23:04.020276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:04.020039 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a64992d018b04be588426e3f21c5039eefc4df31337060f5e24a5cee38c8f0"} err="failed to get container status \"51a64992d018b04be588426e3f21c5039eefc4df31337060f5e24a5cee38c8f0\": rpc error: code = NotFound desc = could not find container \"51a64992d018b04be588426e3f21c5039eefc4df31337060f5e24a5cee38c8f0\": container with ID starting with 51a64992d018b04be588426e3f21c5039eefc4df31337060f5e24a5cee38c8f0 not found: ID does not exist" Oct 30 05:23:04.089471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:04.068421 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27"] Oct 30 05:23:04.183383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:04.182507 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-a914e9/auth-test-model-a914e9-kserve-c5c66466f-zcj27"] Oct 30 05:23:05.493383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:05.487497 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:23:05.528380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:05.523439 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:23:05.669407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:05.668033 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9325b4e-ca66-4cff-9104-ab1959ef8e8c" path="/var/lib/kubelet/pods/e9325b4e-ca66-4cff-9104-ab1959ef8e8c/volumes" Oct 30 05:23:06.766419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:06.762424 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q"] Oct 30 05:23:06.801826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:06.801791 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv"] Oct 30 05:23:08.887617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:08.887580 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" Oct 30 05:23:11.762760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:11.762705 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" podUID="02a6a95b-ab1a-441a-8052-ca050858b70a" containerName="payload-processing" containerID="cri-o://6d100dd31096fae724a21d4bfd034164535476d58941a8560e04efc8732c60c9" gracePeriod=25 Oct 30 05:23:11.804319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:11.803340 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" podUID="8fc270df-6323-4559-9598-3a0fa672d13d" containerName="payload-pre-processing" containerID="cri-o://fe6808189a05eb7f455ed4d21ab61a36c87188d178637c28421e20a02c194385" gracePeriod=25 Oct 30 05:23:12.079620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.079596 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" Oct 30 05:23:12.182531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.182486 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a6a95b-ab1a-441a-8052-ca050858b70a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a6a95b-ab1a-441a-8052-ca050858b70a-plugins-config-volume\") pod \"02a6a95b-ab1a-441a-8052-ca050858b70a\" (UID: \"02a6a95b-ab1a-441a-8052-ca050858b70a\") " Oct 30 05:23:12.182802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.182781 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a6a95b-ab1a-441a-8052-ca050858b70a-kube-api-access-gcndb\" (UniqueName: \"kubernetes.io/projected/02a6a95b-ab1a-441a-8052-ca050858b70a-kube-api-access-gcndb\") pod \"02a6a95b-ab1a-441a-8052-ca050858b70a\" (UID: \"02a6a95b-ab1a-441a-8052-ca050858b70a\") " Oct 30 05:23:12.182942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.182903 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a6a95b-ab1a-441a-8052-ca050858b70a-plugins-config-volume" pod "02a6a95b-ab1a-441a-8052-ca050858b70a" (UID: "02a6a95b-ab1a-441a-8052-ca050858b70a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:12.183405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.183376 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a6a95b-ab1a-441a-8052-ca050858b70a-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:12.188024 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.187941 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a6a95b-ab1a-441a-8052-ca050858b70a-kube-api-access-gcndb" pod "02a6a95b-ab1a-441a-8052-ca050858b70a" (UID: "02a6a95b-ab1a-441a-8052-ca050858b70a"). InnerVolumeSpecName "kube-api-access-gcndb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:12.285333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.285254 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcndb\" (UniqueName: \"kubernetes.io/projected/02a6a95b-ab1a-441a-8052-ca050858b70a-kube-api-access-gcndb\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:12.307979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.307945 6587 generic.go:356] "Generic (PLEG): container finished" podID="02a6a95b-ab1a-441a-8052-ca050858b70a" containerID="6d100dd31096fae724a21d4bfd034164535476d58941a8560e04efc8732c60c9" exitCode=0 Oct 30 05:23:12.308156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.307993 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" Oct 30 05:23:12.308156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.308047 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" event={"ID":"02a6a95b-ab1a-441a-8052-ca050858b70a","Type":"ContainerDied","Data":"6d100dd31096fae724a21d4bfd034164535476d58941a8560e04efc8732c60c9"} Oct 30 05:23:12.308156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.308111 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q" event={"ID":"02a6a95b-ab1a-441a-8052-ca050858b70a","Type":"ContainerDied","Data":"88802efed95ef2b475d99e822a34392d26d128e5b3418d2cf6cff59efd02b2c8"} Oct 30 05:23:12.308156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.308144 6587 scope.go:122] "RemoveContainer" containerID="6d100dd31096fae724a21d4bfd034164535476d58941a8560e04efc8732c60c9" Oct 30 05:23:12.323871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.323847 6587 scope.go:122] "RemoveContainer" containerID="6d100dd31096fae724a21d4bfd034164535476d58941a8560e04efc8732c60c9" Oct 30 05:23:12.324260 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:23:12.324233 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d100dd31096fae724a21d4bfd034164535476d58941a8560e04efc8732c60c9\": container with ID starting with 6d100dd31096fae724a21d4bfd034164535476d58941a8560e04efc8732c60c9 not found: ID does not exist" containerID="6d100dd31096fae724a21d4bfd034164535476d58941a8560e04efc8732c60c9" Oct 30 05:23:12.324429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.324273 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d100dd31096fae724a21d4bfd034164535476d58941a8560e04efc8732c60c9"} err="failed to get container status \"6d100dd31096fae724a21d4bfd034164535476d58941a8560e04efc8732c60c9\": rpc error: code = NotFound desc = could not find container \"6d100dd31096fae724a21d4bfd034164535476d58941a8560e04efc8732c60c9\": container with ID starting with 6d100dd31096fae724a21d4bfd034164535476d58941a8560e04efc8732c60c9 not found: ID does not exist" Oct 30 05:23:12.367642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.367603 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q"] Oct 30 05:23:12.372165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:12.372139 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q"] Oct 30 05:23:13.585481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:13.585440 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a6a95b-ab1a-441a-8052-ca050858b70a" path="/var/lib/kubelet/pods/02a6a95b-ab1a-441a-8052-ca050858b70a/volumes" Oct 30 05:23:14.919041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:14.918992 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv"] Oct 30 05:23:14.920862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:14.920837 6587 cpu_manager.go:414] "containerMap: removing container" podUID="dd91a815-98b4-496a-885d-7a0b1d3e4369" containerName="istio-proxy" Oct 30 05:23:14.921045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:14.921031 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd91a815-98b4-496a-885d-7a0b1d3e4369" containerName="istio-proxy" Oct 30 05:23:14.921177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:14.921167 6587 cpu_manager.go:414] "containerMap: removing container" podUID="02a6a95b-ab1a-441a-8052-ca050858b70a" containerName="payload-processing" Oct 30 05:23:14.921274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:14.921261 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a6a95b-ab1a-441a-8052-ca050858b70a" containerName="payload-processing" Oct 30 05:23:14.921419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:14.921399 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e9325b4e-ca66-4cff-9104-ab1959ef8e8c" containerName="main" Oct 30 05:23:14.921419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:14.921413 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9325b4e-ca66-4cff-9104-ab1959ef8e8c" containerName="main" Oct 30 05:23:14.921419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:14.921426 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3eb62f15-762d-4625-b471-a07aad8c8aa7" containerName="revoke-keys" Oct 30 05:23:14.921699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:14.921445 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb62f15-762d-4625-b471-a07aad8c8aa7" containerName="revoke-keys" Oct 30 05:23:14.921699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:14.921671 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a6a95b-ab1a-441a-8052-ca050858b70a" containerName="payload-processing" Oct 30 05:23:14.921699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:14.921689 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9325b4e-ca66-4cff-9104-ab1959ef8e8c" containerName="main" Oct 30 05:23:14.921699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:14.921699 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb62f15-762d-4625-b471-a07aad8c8aa7" containerName="revoke-keys" Oct 30 05:23:14.921935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:14.921718 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd91a815-98b4-496a-885d-7a0b1d3e4369" containerName="istio-proxy" Oct 30 05:23:15.032495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.032458 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv"] Oct 30 05:23:15.032706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.032627 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.062722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.062585 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-097d90\"/\"default-dockercfg-j2lf2\"" Oct 30 05:23:15.062722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.062642 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-097d90\"/\"openshift-service-ca.crt\"" Oct 30 05:23:15.062722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.062697 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-097d90\"/\"kube-root-ca.crt\"" Oct 30 05:23:15.062722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.062697 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-097d90\"/\"auth-test-model-097d90-kserve-self-signed-certs\"" Oct 30 05:23:15.129780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.129744 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxcp\" (UniqueName: \"kubernetes.io/projected/3de242ad-68be-4de3-8d17-7d3d7c96ab32-kube-api-access-smxcp\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.129977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.129815 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tls-certs\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.129977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.129958 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-model-cache\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.130248 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.130227 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-home\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.130307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.130275 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-dshm\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.130435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.130418 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tmp-dir\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.239678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.236845 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-home\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.239678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.236908 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-dshm\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.239678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.236991 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tmp-dir\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.239678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.237039 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxcp\" (UniqueName: \"kubernetes.io/projected/3de242ad-68be-4de3-8d17-7d3d7c96ab32-kube-api-access-smxcp\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.239678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.237119 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tls-certs\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.239678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.237192 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-model-cache\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.239678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.237769 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-model-cache\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.239678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.238097 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-home\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.261404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.261132 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tmp-dir\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.285776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.282993 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tls-certs\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.285776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.283238 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-dshm\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.295184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.295119 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxcp\" (UniqueName: \"kubernetes.io/projected/3de242ad-68be-4de3-8d17-7d3d7c96ab32-kube-api-access-smxcp\") pod \"auth-test-model-097d90-kserve-67988d4b89-v48fv\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:15.413253 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.413209 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:16.689161 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:23:15.791996 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de242ad_68be_4de3_8d17_7d3d7c96ab32.slice/crio-97a649d6031a684ea5f35bade84ac9334a53ca4cc1b0898ccbd632a2ad06735a WatchSource:0}: Error finding container 97a649d6031a684ea5f35bade84ac9334a53ca4cc1b0898ccbd632a2ad06735a: Status 404 returned error can't find the container with id 97a649d6031a684ea5f35bade84ac9334a53ca4cc1b0898ccbd632a2ad06735a Oct 30 05:23:16.689161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:15.794338 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv"] Oct 30 05:23:16.689161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:16.035434 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh"] Oct 30 05:23:16.689161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:16.039989 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9405336d-bqfdh"] Oct 30 05:23:16.689161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:16.263105 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" podUID="8fc270df-6323-4559-9598-3a0fa672d13d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:23:16.689161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:16.355490 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" event={"ID":"3de242ad-68be-4de3-8d17-7d3d7c96ab32","Type":"ContainerStarted","Data":"d54c4d61678130895b044418218ef5cdadc3c3cd960d365cf199dcf97537e9ab"} Oct 30 05:23:16.689161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:16.355540 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" event={"ID":"3de242ad-68be-4de3-8d17-7d3d7c96ab32","Type":"ContainerStarted","Data":"97a649d6031a684ea5f35bade84ac9334a53ca4cc1b0898ccbd632a2ad06735a"} Oct 30 05:23:16.689161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:16.400888 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" podStartSLOduration=2.400872647 podStartE2EDuration="2.400872647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:23:16.396584416 +0000 UTC m=+2320.924351201" watchObservedRunningTime="2026-10-30 05:23:16.400872647 +0000 UTC m=+2320.928639433" Oct 30 05:23:16.689161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:16.413403 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:17.370606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:17.370574 6587 generic.go:356] "Generic (PLEG): container finished" podID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerID="e48d684c1e26930843b6b216d57b811792f6e1958c5fb2919f8a0756adcc34e0" exitCode=0 Oct 30 05:23:17.370812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:17.370637 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" event={"ID":"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c","Type":"ContainerDied","Data":"e48d684c1e26930843b6b216d57b811792f6e1958c5fb2919f8a0756adcc34e0"} Oct 30 05:23:17.371294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:17.371260 6587 scope.go:122] "RemoveContainer" containerID="e48d684c1e26930843b6b216d57b811792f6e1958c5fb2919f8a0756adcc34e0" Oct 30 05:23:17.373476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:17.373441 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:23:17.373586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:17.373531 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:23:17.373586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:17.373579 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:23:17.591572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:17.591540 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993542b9-d84d-466b-8cbf-e43c3bee49cf" path="/var/lib/kubelet/pods/993542b9-d84d-466b-8cbf-e43c3bee49cf/volumes" Oct 30 05:23:18.381044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:18.381001 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" event={"ID":"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c","Type":"ContainerStarted","Data":"14d8d9568466e6c3a15430b748ea3a4672d3d7c52ec41810dcbab4b811341bc9"} Oct 30 05:23:18.594243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:18.594212 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:23:18.864208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:18.595696 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:23:18.864208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:18.595742 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 05:23:18.988964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:18.988925 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": read tcp 10.134.0.2:39490->10.134.0.254:15021: read: connection reset by peer" start-of-body= Oct 30 05:23:18.989095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:18.989002 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": read tcp 10.134.0.2:39490->10.134.0.254:15021: read: connection reset by peer" Oct 30 05:23:19.361650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:19.361601 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:23:19.391178 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:19.391147 6587 generic.go:356] "Generic (PLEG): container finished" podID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerID="250dc0b7b4d0e922ebc9cd1e5dac86cf9d68e5cc75841625e541237809aca257" exitCode=0 Oct 30 05:23:19.391607 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:19.391227 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" event={"ID":"2e7930f4-1e78-4bd5-be93-010fe7e02d20","Type":"ContainerDied","Data":"250dc0b7b4d0e922ebc9cd1e5dac86cf9d68e5cc75841625e541237809aca257"} Oct 30 05:23:19.391761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:19.391745 6587 scope.go:122] "RemoveContainer" containerID="250dc0b7b4d0e922ebc9cd1e5dac86cf9d68e5cc75841625e541237809aca257" Oct 30 05:23:19.393559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:19.393533 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:23:19.393633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:19.393599 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:23:19.393670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:19.393631 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:23:19.594664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:19.594632 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:23:19.594863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:19.594689 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 05:23:20.405680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:20.405638 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" event={"ID":"2e7930f4-1e78-4bd5-be93-010fe7e02d20","Type":"ContainerStarted","Data":"f977a47e488d84b54b06f1ad51501c5fe4252a314e0f3f720660defa7389b114"} Oct 30 05:23:20.602381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:20.601545 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:23:20.602381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:20.601632 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 05:23:20.992795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:20.992757 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:23:20.992985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:20.992960 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 05:23:20.993038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:20.993006 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 05:23:21.593985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:21.593926 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:23:21.594530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:21.593996 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 05:23:21.992965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:21.992884 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 05:23:21.993185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:21.993161 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 05:23:22.309871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.309697 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6"] Oct 30 05:23:22.314973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.314943 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.354382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.350841 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-34722e\"/\"kube-root-ca.crt\"" Oct 30 05:23:22.354382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.351035 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-34722e\"/\"auth-test-model-34722e-kserve-self-signed-certs\"" Oct 30 05:23:22.354382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.351714 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-34722e\"/\"default-dockercfg-4fbl7\"" Oct 30 05:23:22.354382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.351783 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-34722e\"/\"openshift-service-ca.crt\"" Oct 30 05:23:22.591426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.589600 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tmp-dir\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.591426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.589796 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd6dm\" (UniqueName: \"kubernetes.io/projected/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-kube-api-access-cd6dm\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.591426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.589876 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-home\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.591426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.589949 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tls-certs\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.591426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.589988 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-model-cache\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.591426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.590039 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-dshm\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.595533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.595507 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:23:22.598796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.598765 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 05:23:22.599757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.599737 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6"] Oct 30 05:23:22.691394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.691344 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd6dm\" (UniqueName: \"kubernetes.io/projected/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-kube-api-access-cd6dm\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.691619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.691446 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-home\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.691619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.691517 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tls-certs\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.691619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.691578 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-model-cache\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.691800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.691690 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-dshm\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.691946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.691926 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tmp-dir\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.692255 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.692222 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-home\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.692386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.692346 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tmp-dir\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.692529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.692509 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-model-cache\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.694677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.694654 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-dshm\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.695628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.695609 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tls-certs\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.714129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.714106 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd6dm\" (UniqueName: \"kubernetes.io/projected/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-kube-api-access-cd6dm\") pod \"auth-test-model-34722e-kserve-764b8744bf-qn5q6\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.902802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.902767 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:22.993118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.993000 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 05:23:22.993118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:22.993090 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 05:23:23.595213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:23.594162 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:23:23.595213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:23.594220 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 05:23:23.606492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:23.605824 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6"] Oct 30 05:23:23.992982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:23.992938 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 05:23:23.993190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:23.992991 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 05:23:24.599434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:24.599396 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:23:24.600665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:24.600641 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:23:24.612227 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:24.612189 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" event={"ID":"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e","Type":"ContainerStarted","Data":"f288c5770f5a090499c2eeb8efb6b895b432a1b459b158167772100e4d55ae78"} Oct 30 05:23:24.612615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:24.612233 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" event={"ID":"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e","Type":"ContainerStarted","Data":"bf1ceefed9032fa43624ee07b1b652d4de907b5529cdd272c1250405a286b1cb"} Oct 30 05:23:24.700752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:24.700702 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" podStartSLOduration=2.700687326 podStartE2EDuration="2.700687326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:23:24.699390163 +0000 UTC m=+2329.227156950" watchObservedRunningTime="2026-10-30 05:23:24.700687326 +0000 UTC m=+2329.228454111" Oct 30 05:23:24.903862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:24.903776 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:24.992928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:24.992894 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 05:23:24.993100 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:24.992954 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 05:23:25.414106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:25.414073 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:25.455807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:25.455753 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:25.658199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:25.658166 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:23:25.991169 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:23:25.991126 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:23:25.993575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:25.993555 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 05:23:25.993675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:25.993601 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 05:23:26.261600 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:26.261515 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" podUID="8fc270df-6323-4559-9598-3a0fa672d13d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:23:26.998471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:26.998438 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:23:26.998955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:26.998930 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:23:26.999657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:26.999638 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:23:28.686042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.686007 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn"] Oct 30 05:23:28.689730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.689710 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" Oct 30 05:23:28.709301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.709275 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn"] Oct 30 05:23:28.717632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.717579 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:23:28.717632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.717584 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:23:28.717839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.717709 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:23:28.718288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.718266 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:23:28.732349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.732317 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn6nv\" (UniqueName: \"kubernetes.io/projected/b0805112-162f-41d2-82fc-544db635b227-kube-api-access-hn6nv\") pod \"maas-api-revoke-sub-da31fc8b-htszn\" (UID: \"b0805112-162f-41d2-82fc-544db635b227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" Oct 30 05:23:28.732621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.732601 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0805112-162f-41d2-82fc-544db635b227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da31fc8b-htszn\" (UID: \"b0805112-162f-41d2-82fc-544db635b227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" Oct 30 05:23:28.732876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.732859 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0805112-162f-41d2-82fc-544db635b227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da31fc8b-htszn\" (UID: \"b0805112-162f-41d2-82fc-544db635b227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" Oct 30 05:23:28.835277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.835235 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0805112-162f-41d2-82fc-544db635b227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da31fc8b-htszn\" (UID: \"b0805112-162f-41d2-82fc-544db635b227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" Oct 30 05:23:28.835598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.835576 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn6nv\" (UniqueName: \"kubernetes.io/projected/b0805112-162f-41d2-82fc-544db635b227-kube-api-access-hn6nv\") pod \"maas-api-revoke-sub-da31fc8b-htszn\" (UID: \"b0805112-162f-41d2-82fc-544db635b227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" Oct 30 05:23:28.835655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.835628 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0805112-162f-41d2-82fc-544db635b227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da31fc8b-htszn\" (UID: \"b0805112-162f-41d2-82fc-544db635b227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" Oct 30 05:23:28.835918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.835892 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0805112-162f-41d2-82fc-544db635b227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da31fc8b-htszn\" (UID: \"b0805112-162f-41d2-82fc-544db635b227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" Oct 30 05:23:28.838016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.837992 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0805112-162f-41d2-82fc-544db635b227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da31fc8b-htszn\" (UID: \"b0805112-162f-41d2-82fc-544db635b227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" Oct 30 05:23:28.856675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:28.856648 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn6nv\" (UniqueName: \"kubernetes.io/projected/b0805112-162f-41d2-82fc-544db635b227-kube-api-access-hn6nv\") pod \"maas-api-revoke-sub-da31fc8b-htszn\" (UID: \"b0805112-162f-41d2-82fc-544db635b227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" Oct 30 05:23:29.005312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:29.005217 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" Oct 30 05:23:29.318445 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:23:29.318412 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0805112_162f_41d2_82fc_544db635b227.slice/crio-d90a1ceb20415d9036dc84393fe0958e6b31871d88ef46c390b990326db20baf WatchSource:0}: Error finding container d90a1ceb20415d9036dc84393fe0958e6b31871d88ef46c390b990326db20baf: Status 404 returned error can't find the container with id d90a1ceb20415d9036dc84393fe0958e6b31871d88ef46c390b990326db20baf Oct 30 05:23:29.322617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:29.322589 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn"] Oct 30 05:23:29.685879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:29.685786 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" event={"ID":"b0805112-162f-41d2-82fc-544db635b227","Type":"ContainerStarted","Data":"d90a1ceb20415d9036dc84393fe0958e6b31871d88ef46c390b990326db20baf"} Oct 30 05:23:30.698001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:30.697911 6587 generic.go:356] "Generic (PLEG): container finished" podID="b0805112-162f-41d2-82fc-544db635b227" containerID="e161730fb007f0e91f8b3a6b3155db32200818a715c195d494d8c6d41417544e" exitCode=0 Oct 30 05:23:30.698582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:30.698196 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" event={"ID":"b0805112-162f-41d2-82fc-544db635b227","Type":"ContainerDied","Data":"e161730fb007f0e91f8b3a6b3155db32200818a715c195d494d8c6d41417544e"} Oct 30 05:23:31.943957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:31.943917 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" Oct 30 05:23:32.093847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.093808 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0805112-162f-41d2-82fc-544db635b227-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0805112-162f-41d2-82fc-544db635b227-maas-api-openshift-ca\") pod \"b0805112-162f-41d2-82fc-544db635b227\" (UID: \"b0805112-162f-41d2-82fc-544db635b227\") " Oct 30 05:23:32.094287 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.094036 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0805112-162f-41d2-82fc-544db635b227-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0805112-162f-41d2-82fc-544db635b227-maas-api-xks-ca\") pod \"b0805112-162f-41d2-82fc-544db635b227\" (UID: \"b0805112-162f-41d2-82fc-544db635b227\") " Oct 30 05:23:32.094287 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.094127 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0805112-162f-41d2-82fc-544db635b227-kube-api-access-hn6nv\" (UniqueName: \"kubernetes.io/projected/b0805112-162f-41d2-82fc-544db635b227-kube-api-access-hn6nv\") pod \"b0805112-162f-41d2-82fc-544db635b227\" (UID: \"b0805112-162f-41d2-82fc-544db635b227\") " Oct 30 05:23:32.096936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.095326 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0805112-162f-41d2-82fc-544db635b227-maas-api-openshift-ca" pod "b0805112-162f-41d2-82fc-544db635b227" (UID: "b0805112-162f-41d2-82fc-544db635b227"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:32.097520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.097463 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0805112-162f-41d2-82fc-544db635b227-maas-api-xks-ca" pod "b0805112-162f-41d2-82fc-544db635b227" (UID: "b0805112-162f-41d2-82fc-544db635b227"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:23:32.097723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.097694 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0805112-162f-41d2-82fc-544db635b227-kube-api-access-hn6nv" pod "b0805112-162f-41d2-82fc-544db635b227" (UID: "b0805112-162f-41d2-82fc-544db635b227"). InnerVolumeSpecName "kube-api-access-hn6nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:32.101189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.101166 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" Oct 30 05:23:32.196448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.196411 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc270df-6323-4559-9598-3a0fa672d13d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc270df-6323-4559-9598-3a0fa672d13d-plugins-config-volume\") pod \"8fc270df-6323-4559-9598-3a0fa672d13d\" (UID: \"8fc270df-6323-4559-9598-3a0fa672d13d\") " Oct 30 05:23:32.196778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.196757 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc270df-6323-4559-9598-3a0fa672d13d-kube-api-access-x2blz\" (UniqueName: \"kubernetes.io/projected/8fc270df-6323-4559-9598-3a0fa672d13d-kube-api-access-x2blz\") pod \"8fc270df-6323-4559-9598-3a0fa672d13d\" (UID: \"8fc270df-6323-4559-9598-3a0fa672d13d\") " Oct 30 05:23:32.196869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.196772 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc270df-6323-4559-9598-3a0fa672d13d-plugins-config-volume" pod "8fc270df-6323-4559-9598-3a0fa672d13d" (UID: "8fc270df-6323-4559-9598-3a0fa672d13d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:32.197526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.197490 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0805112-162f-41d2-82fc-544db635b227-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:32.197526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.197518 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0805112-162f-41d2-82fc-544db635b227-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:32.197712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.197533 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn6nv\" (UniqueName: \"kubernetes.io/projected/b0805112-162f-41d2-82fc-544db635b227-kube-api-access-hn6nv\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:32.197712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.197549 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc270df-6323-4559-9598-3a0fa672d13d-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:32.199075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.199052 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc270df-6323-4559-9598-3a0fa672d13d-kube-api-access-x2blz" pod "8fc270df-6323-4559-9598-3a0fa672d13d" (UID: "8fc270df-6323-4559-9598-3a0fa672d13d"). InnerVolumeSpecName "kube-api-access-x2blz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:32.299153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.299125 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2blz\" (UniqueName: \"kubernetes.io/projected/8fc270df-6323-4559-9598-3a0fa672d13d-kube-api-access-x2blz\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:32.717184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.717101 6587 generic.go:356] "Generic (PLEG): container finished" podID="8fc270df-6323-4559-9598-3a0fa672d13d" containerID="fe6808189a05eb7f455ed4d21ab61a36c87188d178637c28421e20a02c194385" exitCode=0 Oct 30 05:23:32.717184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.717151 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" Oct 30 05:23:32.717453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.717179 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" event={"ID":"8fc270df-6323-4559-9598-3a0fa672d13d","Type":"ContainerDied","Data":"fe6808189a05eb7f455ed4d21ab61a36c87188d178637c28421e20a02c194385"} Oct 30 05:23:32.717453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.717247 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv" event={"ID":"8fc270df-6323-4559-9598-3a0fa672d13d","Type":"ContainerDied","Data":"a88f78e77a02190feeba48281a3a48073fa8ca7fd295b2e936761526743de858"} Oct 30 05:23:32.717453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.717282 6587 scope.go:122] "RemoveContainer" containerID="fe6808189a05eb7f455ed4d21ab61a36c87188d178637c28421e20a02c194385" Oct 30 05:23:32.719234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.719215 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" Oct 30 05:23:32.719372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.719245 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn" event={"ID":"b0805112-162f-41d2-82fc-544db635b227","Type":"ContainerDied","Data":"d90a1ceb20415d9036dc84393fe0958e6b31871d88ef46c390b990326db20baf"} Oct 30 05:23:32.719372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.719293 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d90a1ceb20415d9036dc84393fe0958e6b31871d88ef46c390b990326db20baf" Oct 30 05:23:32.731243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.731220 6587 scope.go:122] "RemoveContainer" containerID="fe6808189a05eb7f455ed4d21ab61a36c87188d178637c28421e20a02c194385" Oct 30 05:23:32.731788 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:23:32.731758 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6808189a05eb7f455ed4d21ab61a36c87188d178637c28421e20a02c194385\": container with ID starting with fe6808189a05eb7f455ed4d21ab61a36c87188d178637c28421e20a02c194385 not found: ID does not exist" containerID="fe6808189a05eb7f455ed4d21ab61a36c87188d178637c28421e20a02c194385" Oct 30 05:23:32.731890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.731797 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6808189a05eb7f455ed4d21ab61a36c87188d178637c28421e20a02c194385"} err="failed to get container status \"fe6808189a05eb7f455ed4d21ab61a36c87188d178637c28421e20a02c194385\": rpc error: code = NotFound desc = could not find container \"fe6808189a05eb7f455ed4d21ab61a36c87188d178637c28421e20a02c194385\": container with ID starting with fe6808189a05eb7f455ed4d21ab61a36c87188d178637c28421e20a02c194385 not found: ID does not exist" Oct 30 05:23:32.774939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.774906 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv"] Oct 30 05:23:32.788340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.788310 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv"] Oct 30 05:23:32.903206 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.903171 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:32.916872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:32.916844 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:33.595428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:33.595216 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc270df-6323-4559-9598-3a0fa672d13d" path="/var/lib/kubelet/pods/8fc270df-6323-4559-9598-3a0fa672d13d/volumes" Oct 30 05:23:33.748051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:33.748016 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:23:41.024856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.024764 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s"] Oct 30 05:23:41.604133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.604103 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf"] Oct 30 05:23:41.604923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.604905 6587 cpu_manager.go:414] "containerMap: removing container" podUID="8fc270df-6323-4559-9598-3a0fa672d13d" containerName="payload-pre-processing" Oct 30 05:23:41.604980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.604925 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc270df-6323-4559-9598-3a0fa672d13d" containerName="payload-pre-processing" Oct 30 05:23:41.604980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.604942 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b0805112-162f-41d2-82fc-544db635b227" containerName="revoke-keys" Oct 30 05:23:41.604980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.604948 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0805112-162f-41d2-82fc-544db635b227" containerName="revoke-keys" Oct 30 05:23:41.605103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.605094 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc270df-6323-4559-9598-3a0fa672d13d" containerName="payload-pre-processing" Oct 30 05:23:41.605139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.605106 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0805112-162f-41d2-82fc-544db635b227" containerName="revoke-keys" Oct 30 05:23:41.609058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.609040 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" Oct 30 05:23:41.627207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.627180 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf"] Oct 30 05:23:41.635948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.635419 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:23:41.635948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.635419 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:23:41.635948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.635458 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:23:41.635948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.635886 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:23:41.686659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.686622 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85608484-9ggnf\" (UID: \"fc0a3b92-9af2-4868-8c56-6822eed68522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" Oct 30 05:23:41.686925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.686900 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqgw5\" (UniqueName: \"kubernetes.io/projected/fc0a3b92-9af2-4868-8c56-6822eed68522-kube-api-access-xqgw5\") pod \"maas-api-revoke-sub-85608484-9ggnf\" (UID: \"fc0a3b92-9af2-4868-8c56-6822eed68522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" Oct 30 05:23:41.687033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.687003 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85608484-9ggnf\" (UID: \"fc0a3b92-9af2-4868-8c56-6822eed68522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" Oct 30 05:23:41.788393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.788316 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqgw5\" (UniqueName: \"kubernetes.io/projected/fc0a3b92-9af2-4868-8c56-6822eed68522-kube-api-access-xqgw5\") pod \"maas-api-revoke-sub-85608484-9ggnf\" (UID: \"fc0a3b92-9af2-4868-8c56-6822eed68522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" Oct 30 05:23:41.788627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.788434 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85608484-9ggnf\" (UID: \"fc0a3b92-9af2-4868-8c56-6822eed68522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" Oct 30 05:23:41.788627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.788586 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85608484-9ggnf\" (UID: \"fc0a3b92-9af2-4868-8c56-6822eed68522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" Oct 30 05:23:41.789297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.789269 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85608484-9ggnf\" (UID: \"fc0a3b92-9af2-4868-8c56-6822eed68522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" Oct 30 05:23:41.790911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.790890 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85608484-9ggnf\" (UID: \"fc0a3b92-9af2-4868-8c56-6822eed68522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" Oct 30 05:23:41.809075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.809049 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqgw5\" (UniqueName: \"kubernetes.io/projected/fc0a3b92-9af2-4868-8c56-6822eed68522-kube-api-access-xqgw5\") pod \"maas-api-revoke-sub-85608484-9ggnf\" (UID: \"fc0a3b92-9af2-4868-8c56-6822eed68522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" Oct 30 05:23:41.879794 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.879709 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8"] Oct 30 05:23:41.879961 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.879927 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" podUID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" containerName="istio-proxy" containerID="cri-o://6ae04dede44f0081ae741ddd111e38a6477660c193f8c1fcbe147e516d2a7866" gracePeriod=30 Oct 30 05:23:41.931711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:41.931674 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" Oct 30 05:23:42.245430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:42.245304 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf"] Oct 30 05:23:42.820232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:42.820194 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" event={"ID":"fc0a3b92-9af2-4868-8c56-6822eed68522","Type":"ContainerStarted","Data":"450460be9adfb5d1cd8ac66ddefc0e2c49e50e3827cb6375a649bdfd1576a506"} Oct 30 05:23:42.820481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:42.820260 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" event={"ID":"fc0a3b92-9af2-4868-8c56-6822eed68522","Type":"ContainerStarted","Data":"87041111b31ab51f2a495bd2afa3aafe7040615c20f13d2ccf42f6375092cae2"} Oct 30 05:23:42.866317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:42.866258 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" podStartSLOduration=1.86624246 podStartE2EDuration="1.86624246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:23:42.861997251 +0000 UTC m=+2347.389764038" watchObservedRunningTime="2026-10-30 05:23:42.86624246 +0000 UTC m=+2347.394009268" Oct 30 05:23:43.833461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:43.833425 6587 generic.go:356] "Generic (PLEG): container finished" podID="fc0a3b92-9af2-4868-8c56-6822eed68522" containerID="450460be9adfb5d1cd8ac66ddefc0e2c49e50e3827cb6375a649bdfd1576a506" exitCode=0 Oct 30 05:23:43.833897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:43.833477 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" event={"ID":"fc0a3b92-9af2-4868-8c56-6822eed68522","Type":"ContainerDied","Data":"450460be9adfb5d1cd8ac66ddefc0e2c49e50e3827cb6375a649bdfd1576a506"} Oct 30 05:23:45.026351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:45.026327 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" Oct 30 05:23:45.137556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:45.137461 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0a3b92-9af2-4868-8c56-6822eed68522-kube-api-access-xqgw5\" (UniqueName: \"kubernetes.io/projected/fc0a3b92-9af2-4868-8c56-6822eed68522-kube-api-access-xqgw5\") pod \"fc0a3b92-9af2-4868-8c56-6822eed68522\" (UID: \"fc0a3b92-9af2-4868-8c56-6822eed68522\") " Oct 30 05:23:45.138167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:45.137831 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-xks-ca\") pod \"fc0a3b92-9af2-4868-8c56-6822eed68522\" (UID: \"fc0a3b92-9af2-4868-8c56-6822eed68522\") " Oct 30 05:23:45.138389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:45.138164 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-openshift-ca\") pod \"fc0a3b92-9af2-4868-8c56-6822eed68522\" (UID: \"fc0a3b92-9af2-4868-8c56-6822eed68522\") " Oct 30 05:23:45.138471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:45.138424 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-openshift-ca" pod "fc0a3b92-9af2-4868-8c56-6822eed68522" (UID: "fc0a3b92-9af2-4868-8c56-6822eed68522"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:45.139117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:45.139091 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:45.140480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:45.140452 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0a3b92-9af2-4868-8c56-6822eed68522-kube-api-access-xqgw5" pod "fc0a3b92-9af2-4868-8c56-6822eed68522" (UID: "fc0a3b92-9af2-4868-8c56-6822eed68522"). InnerVolumeSpecName "kube-api-access-xqgw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:45.140589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:45.140552 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-xks-ca" pod "fc0a3b92-9af2-4868-8c56-6822eed68522" (UID: "fc0a3b92-9af2-4868-8c56-6822eed68522"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:23:45.241196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:45.241161 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc0a3b92-9af2-4868-8c56-6822eed68522-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:45.241196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:45.241192 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqgw5\" (UniqueName: \"kubernetes.io/projected/fc0a3b92-9af2-4868-8c56-6822eed68522-kube-api-access-xqgw5\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:45.916437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:45.910647 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" Oct 30 05:23:45.916437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:45.910672 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf" event={"ID":"fc0a3b92-9af2-4868-8c56-6822eed68522","Type":"ContainerDied","Data":"87041111b31ab51f2a495bd2afa3aafe7040615c20f13d2ccf42f6375092cae2"} Oct 30 05:23:45.916437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:45.910730 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87041111b31ab51f2a495bd2afa3aafe7040615c20f13d2ccf42f6375092cae2" Oct 30 05:23:46.160307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:46.160264 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v"] Oct 30 05:23:46.164974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:46.164949 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e16297-zr26v"] Oct 30 05:23:46.826062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:46.825985 6587 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:23:46.826062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:46.826038 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" podUID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:23:47.176232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.176206 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:23:47.311705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.311666 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-certs\") pod \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " Oct 30 05:23:47.311885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.311724 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istiod-ca-cert\") pod \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " Oct 30 05:23:47.311885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.311833 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-credential-socket\") pod \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " Oct 30 05:23:47.312033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.311976 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-certs" pod "e298688e-c592-47ed-b2e0-2f06a53e6b9a" (UID: "e298688e-c592-47ed-b2e0-2f06a53e6b9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:47.312033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.312012 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-token\" (UniqueName: \"kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-token\") pod \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " Oct 30 05:23:47.312143 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.312080 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-credential-socket" pod "e298688e-c592-47ed-b2e0-2f06a53e6b9a" (UID: "e298688e-c592-47ed-b2e0-2f06a53e6b9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:47.312226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.312193 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-podinfo\") pod \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " Oct 30 05:23:47.312226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.312198 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istiod-ca-cert" pod "e298688e-c592-47ed-b2e0-2f06a53e6b9a" (UID: "e298688e-c592-47ed-b2e0-2f06a53e6b9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:47.312400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.312258 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-socket\") pod \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " Oct 30 05:23:47.312400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.312335 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-data\") pod \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " Oct 30 05:23:47.312531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.312438 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-kube-api-access-f7z4h\" (UniqueName: \"kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-kube-api-access-f7z4h\") pod \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " Oct 30 05:23:47.312531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.312515 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-envoy\") pod \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\" (UID: \"e298688e-c592-47ed-b2e0-2f06a53e6b9a\") " Oct 30 05:23:47.312662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.312637 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-data" pod "e298688e-c592-47ed-b2e0-2f06a53e6b9a" (UID: "e298688e-c592-47ed-b2e0-2f06a53e6b9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:47.312725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.312651 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-socket" pod "e298688e-c592-47ed-b2e0-2f06a53e6b9a" (UID: "e298688e-c592-47ed-b2e0-2f06a53e6b9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:47.313310 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.313283 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:47.313455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.313314 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:47.313455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.313328 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:47.313455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.313341 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:47.313455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.313404 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:47.316908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.315432 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-token" pod "e298688e-c592-47ed-b2e0-2f06a53e6b9a" (UID: "e298688e-c592-47ed-b2e0-2f06a53e6b9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:47.317175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.317149 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-envoy" pod "e298688e-c592-47ed-b2e0-2f06a53e6b9a" (UID: "e298688e-c592-47ed-b2e0-2f06a53e6b9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:47.317282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.317216 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-kube-api-access-f7z4h" pod "e298688e-c592-47ed-b2e0-2f06a53e6b9a" (UID: "e298688e-c592-47ed-b2e0-2f06a53e6b9a"). InnerVolumeSpecName "kube-api-access-f7z4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:47.317740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.317710 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-podinfo" pod "e298688e-c592-47ed-b2e0-2f06a53e6b9a" (UID: "e298688e-c592-47ed-b2e0-2f06a53e6b9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:23:47.415546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.415455 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:47.415546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.415494 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:47.415546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.415511 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7z4h\" (UniqueName: \"kubernetes.io/projected/e298688e-c592-47ed-b2e0-2f06a53e6b9a-kube-api-access-f7z4h\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:47.415546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.415524 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e298688e-c592-47ed-b2e0-2f06a53e6b9a-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:47.586129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.586089 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb62f15-762d-4625-b471-a07aad8c8aa7" path="/var/lib/kubelet/pods/3eb62f15-762d-4625-b471-a07aad8c8aa7/volumes" Oct 30 05:23:47.936229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.936195 6587 generic.go:356] "Generic (PLEG): container finished" podID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" containerID="6ae04dede44f0081ae741ddd111e38a6477660c193f8c1fcbe147e516d2a7866" exitCode=0 Oct 30 05:23:47.936435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.936277 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" Oct 30 05:23:47.936435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.936279 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" event={"ID":"e298688e-c592-47ed-b2e0-2f06a53e6b9a","Type":"ContainerDied","Data":"6ae04dede44f0081ae741ddd111e38a6477660c193f8c1fcbe147e516d2a7866"} Oct 30 05:23:47.936435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.936397 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8" event={"ID":"e298688e-c592-47ed-b2e0-2f06a53e6b9a","Type":"ContainerDied","Data":"4aa9236dbf2a892e8edb97f9baa38e7ce231de37d649824030503c74861d0387"} Oct 30 05:23:47.936435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.936431 6587 scope.go:122] "RemoveContainer" containerID="6ae04dede44f0081ae741ddd111e38a6477660c193f8c1fcbe147e516d2a7866" Oct 30 05:23:47.951327 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.951296 6587 scope.go:122] "RemoveContainer" containerID="6ae04dede44f0081ae741ddd111e38a6477660c193f8c1fcbe147e516d2a7866" Oct 30 05:23:47.951708 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:23:47.951681 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae04dede44f0081ae741ddd111e38a6477660c193f8c1fcbe147e516d2a7866\": container with ID starting with 6ae04dede44f0081ae741ddd111e38a6477660c193f8c1fcbe147e516d2a7866 not found: ID does not exist" containerID="6ae04dede44f0081ae741ddd111e38a6477660c193f8c1fcbe147e516d2a7866" Oct 30 05:23:47.951787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.951720 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae04dede44f0081ae741ddd111e38a6477660c193f8c1fcbe147e516d2a7866"} err="failed to get container status \"6ae04dede44f0081ae741ddd111e38a6477660c193f8c1fcbe147e516d2a7866\": rpc error: code = NotFound desc = could not find container \"6ae04dede44f0081ae741ddd111e38a6477660c193f8c1fcbe147e516d2a7866\": container with ID starting with 6ae04dede44f0081ae741ddd111e38a6477660c193f8c1fcbe147e516d2a7866 not found: ID does not exist" Oct 30 05:23:47.996861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:47.996822 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8"] Oct 30 05:23:48.001586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:48.001557 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8"] Oct 30 05:23:49.585316 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:49.585279 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" path="/var/lib/kubelet/pods/e298688e-c592-47ed-b2e0-2f06a53e6b9a/volumes" Oct 30 05:23:53.361338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:53.361304 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5"] Oct 30 05:23:53.361832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:53.361613 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" podUID="abdd1519-522f-4302-844d-947822be0960" containerName="istio-proxy" containerID="cri-o://366a66da5be1ff07fa5ca3138ada6e1e84affcf9d31e4e9c8fc0627be2d5aa4e" gracePeriod=30 Oct 30 05:23:55.079456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:55.077423 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5"] Oct 30 05:23:55.085871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:55.085834 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b6b062e-7s6l5"] Oct 30 05:23:55.586006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:55.585964 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835cb573-de48-4f9a-a0d9-a429c76dffdb" path="/var/lib/kubelet/pods/835cb573-de48-4f9a-a0d9-a429c76dffdb/volumes" Oct 30 05:23:55.990344 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:23:55.990260 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:23:56.054346 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.054298 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" podUID="2719613c-cb60-4cc8-a36d-b5cdf835aca3" containerName="main" containerID="cri-o://51495dc25b4cab73630d934cba4b4c28dad0d482d6e948eecb56cfecbfd812ff" gracePeriod=45 Oct 30 05:23:56.392021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.391982 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:23:56.473068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.473032 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-home\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-home\") pod \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " Oct 30 05:23:56.473256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.473144 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2719613c-cb60-4cc8-a36d-b5cdf835aca3-kube-api-access-kvndc\" (UniqueName: \"kubernetes.io/projected/2719613c-cb60-4cc8-a36d-b5cdf835aca3-kube-api-access-kvndc\") pod \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " Oct 30 05:23:56.473256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.473236 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tls-certs\" (UniqueName: \"kubernetes.io/secret/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tls-certs\") pod \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " Oct 30 05:23:56.473491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.473408 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-home" pod "2719613c-cb60-4cc8-a36d-b5cdf835aca3" (UID: "2719613c-cb60-4cc8-a36d-b5cdf835aca3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:56.473491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.473437 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-model-cache\") pod \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " Oct 30 05:23:56.473621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.473578 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-dshm\") pod \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " Oct 30 05:23:56.473679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.473650 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tmp-dir\") pod \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\" (UID: \"2719613c-cb60-4cc8-a36d-b5cdf835aca3\") " Oct 30 05:23:56.473737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.473669 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-model-cache" pod "2719613c-cb60-4cc8-a36d-b5cdf835aca3" (UID: "2719613c-cb60-4cc8-a36d-b5cdf835aca3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:56.475652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.473932 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tmp-dir" pod "2719613c-cb60-4cc8-a36d-b5cdf835aca3" (UID: "2719613c-cb60-4cc8-a36d-b5cdf835aca3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:56.475652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.474483 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:56.475652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.474506 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:56.475652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.474519 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:56.475886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.475868 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2719613c-cb60-4cc8-a36d-b5cdf835aca3-kube-api-access-kvndc" pod "2719613c-cb60-4cc8-a36d-b5cdf835aca3" (UID: "2719613c-cb60-4cc8-a36d-b5cdf835aca3"). InnerVolumeSpecName "kube-api-access-kvndc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:56.476680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.476639 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-dshm" pod "2719613c-cb60-4cc8-a36d-b5cdf835aca3" (UID: "2719613c-cb60-4cc8-a36d-b5cdf835aca3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:56.476806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.476727 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tls-certs" pod "2719613c-cb60-4cc8-a36d-b5cdf835aca3" (UID: "2719613c-cb60-4cc8-a36d-b5cdf835aca3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:23:56.581380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.576714 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2719613c-cb60-4cc8-a36d-b5cdf835aca3-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:56.581380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.576760 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvndc\" (UniqueName: \"kubernetes.io/projected/2719613c-cb60-4cc8-a36d-b5cdf835aca3-kube-api-access-kvndc\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:56.581380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:56.576777 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2719613c-cb60-4cc8-a36d-b5cdf835aca3-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:57.065744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:57.065703 6587 generic.go:356] "Generic (PLEG): container finished" podID="2719613c-cb60-4cc8-a36d-b5cdf835aca3" containerID="51495dc25b4cab73630d934cba4b4c28dad0d482d6e948eecb56cfecbfd812ff" exitCode=0 Oct 30 05:23:57.066257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:57.066226 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" event={"ID":"2719613c-cb60-4cc8-a36d-b5cdf835aca3","Type":"ContainerDied","Data":"51495dc25b4cab73630d934cba4b4c28dad0d482d6e948eecb56cfecbfd812ff"} Oct 30 05:23:57.068629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:57.066941 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" Oct 30 05:23:57.068836 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:57.068814 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s" event={"ID":"2719613c-cb60-4cc8-a36d-b5cdf835aca3","Type":"ContainerDied","Data":"b990aa82a75aab689e2666fffb2901c12f6eac0b39e29c4fee71053d68a88e2d"} Oct 30 05:23:57.069031 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:57.068968 6587 scope.go:122] "RemoveContainer" containerID="51495dc25b4cab73630d934cba4b4c28dad0d482d6e948eecb56cfecbfd812ff" Oct 30 05:23:57.094413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:57.093583 6587 scope.go:122] "RemoveContainer" containerID="51495dc25b4cab73630d934cba4b4c28dad0d482d6e948eecb56cfecbfd812ff" Oct 30 05:23:57.094413 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:23:57.094053 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51495dc25b4cab73630d934cba4b4c28dad0d482d6e948eecb56cfecbfd812ff\": container with ID starting with 51495dc25b4cab73630d934cba4b4c28dad0d482d6e948eecb56cfecbfd812ff not found: ID does not exist" containerID="51495dc25b4cab73630d934cba4b4c28dad0d482d6e948eecb56cfecbfd812ff" Oct 30 05:23:57.094413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:57.094096 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51495dc25b4cab73630d934cba4b4c28dad0d482d6e948eecb56cfecbfd812ff"} err="failed to get container status \"51495dc25b4cab73630d934cba4b4c28dad0d482d6e948eecb56cfecbfd812ff\": rpc error: code = NotFound desc = could not find container \"51495dc25b4cab73630d934cba4b4c28dad0d482d6e948eecb56cfecbfd812ff\": container with ID starting with 51495dc25b4cab73630d934cba4b4c28dad0d482d6e948eecb56cfecbfd812ff not found: ID does not exist" Oct 30 05:23:57.126844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:57.126798 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s"] Oct 30 05:23:57.141737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:57.141699 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-802b8a/test-model-802b8a-kserve-5dcdbbcdf-hhm4s"] Oct 30 05:23:57.585045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:57.585004 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2719613c-cb60-4cc8-a36d-b5cdf835aca3" path="/var/lib/kubelet/pods/2719613c-cb60-4cc8-a36d-b5cdf835aca3/volumes" Oct 30 05:23:58.763232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.762927 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:23:58.856951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.856913 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abdd1519-522f-4302-844d-947822be0960-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abdd1519-522f-4302-844d-947822be0960-istio-podinfo\") pod \"abdd1519-522f-4302-844d-947822be0960\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " Oct 30 05:23:58.857281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.857251 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-kube-api-access-sc8zp\" (UniqueName: \"kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-kube-api-access-sc8zp\") pod \"abdd1519-522f-4302-844d-947822be0960\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " Oct 30 05:23:58.858075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.858048 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-envoy\") pod \"abdd1519-522f-4302-844d-947822be0960\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " Oct 30 05:23:58.859897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.859760 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abdd1519-522f-4302-844d-947822be0960-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abdd1519-522f-4302-844d-947822be0960-istiod-ca-cert\") pod \"abdd1519-522f-4302-844d-947822be0960\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " Oct 30 05:23:58.860414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.860340 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abdd1519-522f-4302-844d-947822be0960-istiod-ca-cert" pod "abdd1519-522f-4302-844d-947822be0960" (UID: "abdd1519-522f-4302-844d-947822be0960"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:58.860934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.860566 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-istio-token\" (UniqueName: \"kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-istio-token\") pod \"abdd1519-522f-4302-844d-947822be0960\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " Oct 30 05:23:58.861526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.861147 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-socket\") pod \"abdd1519-522f-4302-844d-947822be0960\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " Oct 30 05:23:58.861785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.861764 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-credential-socket\") pod \"abdd1519-522f-4302-844d-947822be0960\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " Oct 30 05:23:58.862729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.861501 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-socket" pod "abdd1519-522f-4302-844d-947822be0960" (UID: "abdd1519-522f-4302-844d-947822be0960"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:58.862729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.862093 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-credential-socket" pod "abdd1519-522f-4302-844d-947822be0960" (UID: "abdd1519-522f-4302-844d-947822be0960"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:58.862729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.862604 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-istio-token" pod "abdd1519-522f-4302-844d-947822be0960" (UID: "abdd1519-522f-4302-844d-947822be0960"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:58.863102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.863083 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-certs\") pod \"abdd1519-522f-4302-844d-947822be0960\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " Oct 30 05:23:58.863609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.863589 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-data\") pod \"abdd1519-522f-4302-844d-947822be0960\" (UID: \"abdd1519-522f-4302-844d-947822be0960\") " Oct 30 05:23:58.865446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.863477 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-certs" pod "abdd1519-522f-4302-844d-947822be0960" (UID: "abdd1519-522f-4302-844d-947822be0960"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:58.865568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.863915 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-data" pod "abdd1519-522f-4302-844d-947822be0960" (UID: "abdd1519-522f-4302-844d-947822be0960"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:58.866204 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.866180 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.866337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.866323 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.866473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.866458 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.866623 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.866610 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.866721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.866710 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abdd1519-522f-4302-844d-947822be0960-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.866807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.866796 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.871008 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.870958 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abdd1519-522f-4302-844d-947822be0960-istio-podinfo" pod "abdd1519-522f-4302-844d-947822be0960" (UID: "abdd1519-522f-4302-844d-947822be0960"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:23:58.872260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.872235 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-kube-api-access-sc8zp" pod "abdd1519-522f-4302-844d-947822be0960" (UID: "abdd1519-522f-4302-844d-947822be0960"). InnerVolumeSpecName "kube-api-access-sc8zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:58.874509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.874487 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-envoy" pod "abdd1519-522f-4302-844d-947822be0960" (UID: "abdd1519-522f-4302-844d-947822be0960"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:23:58.968634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.968556 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abdd1519-522f-4302-844d-947822be0960-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.968634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.968586 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc8zp\" (UniqueName: \"kubernetes.io/projected/abdd1519-522f-4302-844d-947822be0960-kube-api-access-sc8zp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:58.968634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:58.968597 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abdd1519-522f-4302-844d-947822be0960-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:23:59.091705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:59.091671 6587 generic.go:356] "Generic (PLEG): container finished" podID="abdd1519-522f-4302-844d-947822be0960" containerID="366a66da5be1ff07fa5ca3138ada6e1e84affcf9d31e4e9c8fc0627be2d5aa4e" exitCode=0 Oct 30 05:23:59.091882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:59.091723 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" Oct 30 05:23:59.091882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:59.091731 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" event={"ID":"abdd1519-522f-4302-844d-947822be0960","Type":"ContainerDied","Data":"366a66da5be1ff07fa5ca3138ada6e1e84affcf9d31e4e9c8fc0627be2d5aa4e"} Oct 30 05:23:59.091882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:59.091783 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5" event={"ID":"abdd1519-522f-4302-844d-947822be0960","Type":"ContainerDied","Data":"24986f0358dbeab631d7909392f918624117816e29cdc682eb48a9cdb27bd19d"} Oct 30 05:23:59.091882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:59.091806 6587 scope.go:122] "RemoveContainer" containerID="366a66da5be1ff07fa5ca3138ada6e1e84affcf9d31e4e9c8fc0627be2d5aa4e" Oct 30 05:23:59.104099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:59.104075 6587 scope.go:122] "RemoveContainer" containerID="35f1f9dff0904c02d65598be5658e7005270f90d83036415c62557860378e5cc" Oct 30 05:23:59.119529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:59.115874 6587 scope.go:122] "RemoveContainer" containerID="366a66da5be1ff07fa5ca3138ada6e1e84affcf9d31e4e9c8fc0627be2d5aa4e" Oct 30 05:23:59.119529 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:23:59.116163 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366a66da5be1ff07fa5ca3138ada6e1e84affcf9d31e4e9c8fc0627be2d5aa4e\": container with ID starting with 366a66da5be1ff07fa5ca3138ada6e1e84affcf9d31e4e9c8fc0627be2d5aa4e not found: ID does not exist" containerID="366a66da5be1ff07fa5ca3138ada6e1e84affcf9d31e4e9c8fc0627be2d5aa4e" Oct 30 05:23:59.119529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:59.116197 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366a66da5be1ff07fa5ca3138ada6e1e84affcf9d31e4e9c8fc0627be2d5aa4e"} err="failed to get container status \"366a66da5be1ff07fa5ca3138ada6e1e84affcf9d31e4e9c8fc0627be2d5aa4e\": rpc error: code = NotFound desc = could not find container \"366a66da5be1ff07fa5ca3138ada6e1e84affcf9d31e4e9c8fc0627be2d5aa4e\": container with ID starting with 366a66da5be1ff07fa5ca3138ada6e1e84affcf9d31e4e9c8fc0627be2d5aa4e not found: ID does not exist" Oct 30 05:23:59.119529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:59.116221 6587 scope.go:122] "RemoveContainer" containerID="35f1f9dff0904c02d65598be5658e7005270f90d83036415c62557860378e5cc" Oct 30 05:23:59.137404 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:23:59.133614 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f1f9dff0904c02d65598be5658e7005270f90d83036415c62557860378e5cc\": container with ID starting with 35f1f9dff0904c02d65598be5658e7005270f90d83036415c62557860378e5cc not found: ID does not exist" containerID="35f1f9dff0904c02d65598be5658e7005270f90d83036415c62557860378e5cc" Oct 30 05:23:59.137404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:59.133659 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f1f9dff0904c02d65598be5658e7005270f90d83036415c62557860378e5cc"} err="failed to get container status \"35f1f9dff0904c02d65598be5658e7005270f90d83036415c62557860378e5cc\": rpc error: code = NotFound desc = could not find container \"35f1f9dff0904c02d65598be5658e7005270f90d83036415c62557860378e5cc\": container with ID starting with 35f1f9dff0904c02d65598be5658e7005270f90d83036415c62557860378e5cc not found: ID does not exist" Oct 30 05:23:59.160648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:59.160616 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5"] Oct 30 05:23:59.173475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:59.173441 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5"] Oct 30 05:23:59.586389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:23:59.586341 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abdd1519-522f-4302-844d-947822be0960" path="/var/lib/kubelet/pods/abdd1519-522f-4302-844d-947822be0960/volumes" Oct 30 05:24:04.538023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.537983 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph"] Oct 30 05:24:04.539120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539091 6587 cpu_manager.go:414] "containerMap: removing container" podUID="abdd1519-522f-4302-844d-947822be0960" containerName="istio-proxy" Oct 30 05:24:04.539120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539120 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abdd1519-522f-4302-844d-947822be0960" containerName="istio-proxy" Oct 30 05:24:04.539278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539140 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" containerName="istio-proxy" Oct 30 05:24:04.539278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539147 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" containerName="istio-proxy" Oct 30 05:24:04.539278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539190 6587 cpu_manager.go:414] "containerMap: removing container" podUID="abdd1519-522f-4302-844d-947822be0960" containerName="istio-proxy" Oct 30 05:24:04.539278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539199 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abdd1519-522f-4302-844d-947822be0960" containerName="istio-proxy" Oct 30 05:24:04.539278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539219 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2719613c-cb60-4cc8-a36d-b5cdf835aca3" containerName="main" Oct 30 05:24:04.539278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539224 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2719613c-cb60-4cc8-a36d-b5cdf835aca3" containerName="main" Oct 30 05:24:04.539278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539278 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fc0a3b92-9af2-4868-8c56-6822eed68522" containerName="revoke-keys" Oct 30 05:24:04.539705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539284 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0a3b92-9af2-4868-8c56-6822eed68522" containerName="revoke-keys" Oct 30 05:24:04.539705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539465 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0a3b92-9af2-4868-8c56-6822eed68522" containerName="revoke-keys" Oct 30 05:24:04.539705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539477 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="abdd1519-522f-4302-844d-947822be0960" containerName="istio-proxy" Oct 30 05:24:04.539705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539484 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e298688e-c592-47ed-b2e0-2f06a53e6b9a" containerName="istio-proxy" Oct 30 05:24:04.539705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.539496 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2719613c-cb60-4cc8-a36d-b5cdf835aca3" containerName="main" Oct 30 05:24:04.542340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.542320 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.551566 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.551538 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l"] Oct 30 05:24:04.552800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.552461 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="abdd1519-522f-4302-844d-947822be0960" containerName="istio-proxy" Oct 30 05:24:04.555313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.555291 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.561457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.561431 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6b3855-openshift-default-dockercfg-vbtxb\"" Oct 30 05:24:04.586867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.578251 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph"] Oct 30 05:24:04.587114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.587087 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h284r\" (UniqueName: \"kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-kube-api-access-h284r\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.587217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.587141 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.587283 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.587264 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.587540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.587391 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.587711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.587581 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.587784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.587711 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.587839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.587812 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.587896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.587883 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.589463 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.588027 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.598286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.598252 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l"] Oct 30 05:24:04.699903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.699855 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.700068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.699951 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.700068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700029 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h284r\" (UniqueName: \"kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-kube-api-access-h284r\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.700068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700059 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.700251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700122 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5pzd\" (UniqueName: \"kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-kube-api-access-h5pzd\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.700251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700164 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.700354 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700273 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.700354 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700307 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.700354 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700322 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.700525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700398 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.700525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700487 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.700638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700616 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.700706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700683 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.700706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700697 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.700819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700739 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.700819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700748 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.700924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700830 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.700924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.700865 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.701064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.701041 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.701113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.701093 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.701167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.701153 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c027be-26de-4cd2-bfcf-b353ccc68924-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.701232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.701218 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.701350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.701330 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.703638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.703609 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.703770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.703715 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.721602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.721570 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h284r\" (UniqueName: \"kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-kube-api-access-h284r\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.722034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.722009 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.802388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.802316 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.802595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.802439 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.802595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.802472 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.802595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.802560 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.802780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.802613 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c027be-26de-4cd2-bfcf-b353ccc68924-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.802852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.802789 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5pzd\" (UniqueName: \"kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-kube-api-access-h5pzd\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.802852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.802831 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.802958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.802926 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.802958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.802932 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.802958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.802943 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.803124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.803015 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.803124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.803087 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.803549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.803524 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c027be-26de-4cd2-bfcf-b353ccc68924-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.803694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.803549 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.805288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.805268 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.805705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.805680 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.826168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.826129 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.827123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.827094 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5pzd\" (UniqueName: \"kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-kube-api-access-h5pzd\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:04.863839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.863805 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:04.927408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:04.924865 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:05.203262 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:24:05.203233 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62bcdd5c_8075_434d_89d4_071ea3f6b9d6.slice/crio-900f6435277f7bb327b729635f9bb3d6466bcff587433e3ff882baf14f444aec WatchSource:0}: Error finding container 900f6435277f7bb327b729635f9bb3d6466bcff587433e3ff882baf14f444aec: Status 404 returned error can't find the container with id 900f6435277f7bb327b729635f9bb3d6466bcff587433e3ff882baf14f444aec Oct 30 05:24:05.205001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:05.204980 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph"] Oct 30 05:24:05.206312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:05.206281 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:05.206451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:05.206346 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:05.206451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:05.206403 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:05.265183 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:24:05.265136 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c027be_26de_4cd2_bfcf_b353ccc68924.slice/crio-78c3dc316c1d0dc707024e206239f3bae3f22855d2bc12256ab20542ecb858ee WatchSource:0}: Error finding container 78c3dc316c1d0dc707024e206239f3bae3f22855d2bc12256ab20542ecb858ee: Status 404 returned error can't find the container with id 78c3dc316c1d0dc707024e206239f3bae3f22855d2bc12256ab20542ecb858ee Oct 30 05:24:05.268343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:05.268319 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l"] Oct 30 05:24:05.268853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:05.268806 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:05.268945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:05.268887 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:05.268945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:05.268916 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:06.198347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.198305 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" event={"ID":"62bcdd5c-8075-434d-89d4-071ea3f6b9d6","Type":"ContainerStarted","Data":"bc33126225acad9d5475c694a8382f69f54db764a8fe5183d3af40d2c89693d9"} Oct 30 05:24:06.198828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.198396 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" event={"ID":"62bcdd5c-8075-434d-89d4-071ea3f6b9d6","Type":"ContainerStarted","Data":"900f6435277f7bb327b729635f9bb3d6466bcff587433e3ff882baf14f444aec"} Oct 30 05:24:06.203330 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.203291 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" event={"ID":"18c027be-26de-4cd2-bfcf-b353ccc68924","Type":"ContainerStarted","Data":"ea59fe5a92dd7b1a945e041e81920c594f8f0cc3a480310069e921ea44967a76"} Oct 30 05:24:06.203497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.203405 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" event={"ID":"18c027be-26de-4cd2-bfcf-b353ccc68924","Type":"ContainerStarted","Data":"78c3dc316c1d0dc707024e206239f3bae3f22855d2bc12256ab20542ecb858ee"} Oct 30 05:24:06.332310 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.332236 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" podStartSLOduration=2.332213092 podStartE2EDuration="2.332213092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:24:06.270856535 +0000 UTC m=+2370.798623323" watchObservedRunningTime="2026-10-30 05:24:06.332213092 +0000 UTC m=+2370.859979879" Oct 30 05:24:06.349676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.349604 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" podStartSLOduration=2.349580951 podStartE2EDuration="2.349580951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:24:06.314980222 +0000 UTC m=+2370.842747007" watchObservedRunningTime="2026-10-30 05:24:06.349580951 +0000 UTC m=+2370.877347739" Oct 30 05:24:06.836383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.823551 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:24:06.836383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.823621 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:24:06.880396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.869721 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:06.880396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.879489 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 05:24:06.880396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.879545 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" podUID="62bcdd5c-8075-434d-89d4-071ea3f6b9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 05:24:06.948407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.948371 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:06.981377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.980570 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 05:24:06.981377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:06.980635 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" podUID="18c027be-26de-4cd2-bfcf-b353ccc68924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 05:24:07.868380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:07.867742 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 05:24:07.868380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:07.867807 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" podUID="62bcdd5c-8075-434d-89d4-071ea3f6b9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 05:24:07.932382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:07.931889 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 05:24:07.932382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:07.931958 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" podUID="18c027be-26de-4cd2-bfcf-b353ccc68924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 05:24:08.064153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:08.064105 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk"] Oct 30 05:24:08.096840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:08.096768 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6eb4b85-b8ttk"] Oct 30 05:24:08.864786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:08.864743 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 05:24:08.865910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:08.865873 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" podUID="62bcdd5c-8075-434d-89d4-071ea3f6b9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 05:24:08.938380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:08.937833 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 05:24:08.938380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:08.937893 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" podUID="18c027be-26de-4cd2-bfcf-b353ccc68924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 05:24:09.286915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:09.286821 6587 generic.go:356] "Generic (PLEG): container finished" podID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerID="052301a58b96cf1fd15e0abddb5a44657e019d60019881734c819567f07d7bd6" exitCode=0 Oct 30 05:24:09.287112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:09.287087 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" event={"ID":"cfc65f69-cd06-4575-9e52-29f042c2bdc7","Type":"ContainerDied","Data":"052301a58b96cf1fd15e0abddb5a44657e019d60019881734c819567f07d7bd6"} Oct 30 05:24:09.288030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:09.288012 6587 scope.go:122] "RemoveContainer" containerID="052301a58b96cf1fd15e0abddb5a44657e019d60019881734c819567f07d7bd6" Oct 30 05:24:09.290111 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:09.290085 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:09.290178 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:09.290153 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:09.290216 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:09.290186 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:09.608463 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:09.608422 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa0d385-5478-4563-99c2-a027e159bf06" path="/var/lib/kubelet/pods/4aa0d385-5478-4563-99c2-a027e159bf06/volumes" Oct 30 05:24:09.867984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:09.867883 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 05:24:09.868281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:09.868232 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" podUID="62bcdd5c-8075-434d-89d4-071ea3f6b9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 05:24:09.939383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:09.930836 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 05:24:09.939383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:09.930892 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" podUID="18c027be-26de-4cd2-bfcf-b353ccc68924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 05:24:10.305727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.305696 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" event={"ID":"cfc65f69-cd06-4575-9e52-29f042c2bdc7","Type":"ContainerStarted","Data":"175e1dd6f95e7859066bdaa6ba59e6861cbe393b2efb0bc8534d533ff34d9b44"} Oct 30 05:24:10.324382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.319693 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:24:10.324382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.319806 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 05:24:10.324382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.319844 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 05:24:10.751433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.740419 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t"] Oct 30 05:24:10.821380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.814525 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:10.920384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.855760 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e15c99\"/\"kube-root-ca.crt\"" Oct 30 05:24:10.950383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.856771 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e15c99\"/\"default-dockercfg-8h65p\"" Oct 30 05:24:10.950383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.856847 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e15c99\"/\"openshift-service-ca.crt\"" Oct 30 05:24:10.950383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.856960 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e15c99\"/\"auth-test-model-e15c99-kserve-self-signed-certs\"" Oct 30 05:24:10.950383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.901614 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 05:24:10.950383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.942904 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" podUID="62bcdd5c-8075-434d-89d4-071ea3f6b9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 05:24:10.950383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.936696 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 05:24:10.950383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:10.943280 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" podUID="18c027be-26de-4cd2-bfcf-b353ccc68924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 05:24:11.063454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.051498 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-dshm\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.063454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.051564 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnkr\" (UniqueName: \"kubernetes.io/projected/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-kube-api-access-bvnkr\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.063454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.051738 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-home\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.063454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.051772 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tmp-dir\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.063454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.051799 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tls-certs\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.063454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.051856 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-model-cache\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.179412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.178081 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t"] Oct 30 05:24:11.208381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.187824 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-home\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.208381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.187873 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tmp-dir\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.208381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.187903 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tls-certs\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.208381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.187962 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-model-cache\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.208381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.188128 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-dshm\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.208381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.188175 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnkr\" (UniqueName: \"kubernetes.io/projected/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-kube-api-access-bvnkr\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.260389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.232426 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-model-cache\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.284394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.283058 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tls-certs\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.365479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.290581 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-dshm\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.365479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.295654 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnkr\" (UniqueName: \"kubernetes.io/projected/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-kube-api-access-bvnkr\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.365479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.296112 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-home\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.365479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.296635 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tmp-dir\") pod \"auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.404423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.380507 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 05:24:11.404423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.381020 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 05:24:11.570394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.568568 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:11.751787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.751731 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:24:11.819576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.786720 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" podUID="d0fd7250-35d9-482a-a71b-2be4c8c3ad6e" containerName="main" probeResult="failure" output="Get \"https://10.134.1.10:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:24:11.829508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.825523 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:24:11.901482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.899385 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 05:24:11.901482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.899459 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" podUID="62bcdd5c-8075-434d-89d4-071ea3f6b9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 05:24:11.932305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.932167 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Oct 30 05:24:11.932305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:11.932235 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" podUID="18c027be-26de-4cd2-bfcf-b353ccc68924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Oct 30 05:24:12.325391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:12.324750 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 05:24:12.325391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:12.324827 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 05:24:12.499405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:12.499270 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t"] Oct 30 05:24:12.499813 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:24:12.499786 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04dcecfd_1ed5_4221_a527_3aeb9adb49f3.slice/crio-feb5268a3d6c2d7d9affc6ed00ae9c2c79e184e3ac9722350e6666abd340bc07 WatchSource:0}: Error finding container feb5268a3d6c2d7d9affc6ed00ae9c2c79e184e3ac9722350e6666abd340bc07: Status 404 returned error can't find the container with id feb5268a3d6c2d7d9affc6ed00ae9c2c79e184e3ac9722350e6666abd340bc07 Oct 30 05:24:12.869713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:12.869682 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:12.870036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:12.870004 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:12.870726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:12.870705 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:24:12.951688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:12.951653 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:12.955348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:12.955131 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:12.955937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:12.955914 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:24:12.998790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:12.998747 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" event={"ID":"04dcecfd-1ed5-4221-a527-3aeb9adb49f3","Type":"ContainerStarted","Data":"b1b039a76f4fc0c989754869e9e605e311360b6f42cde6d4c6d881777c971d15"} Oct 30 05:24:12.999039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:12.999021 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" event={"ID":"04dcecfd-1ed5-4221-a527-3aeb9adb49f3","Type":"ContainerStarted","Data":"feb5268a3d6c2d7d9affc6ed00ae9c2c79e184e3ac9722350e6666abd340bc07"} Oct 30 05:24:13.086725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:13.083175 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" podStartSLOduration=3.08315455 podStartE2EDuration="3.08315455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:24:13.077820699 +0000 UTC m=+2377.605587486" watchObservedRunningTime="2026-10-30 05:24:13.08315455 +0000 UTC m=+2377.610921339" Oct 30 05:24:13.335382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:13.332462 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 05:24:13.335382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:13.332538 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 05:24:13.604312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:13.604228 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:14.325772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.325733 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 05:24:14.326101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.326062 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 05:24:14.558281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.557937 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r"] Oct 30 05:24:14.570393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.566664 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.576980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.576906 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r"] Oct 30 05:24:14.585572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.584673 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp9rf\" (UniqueName: \"kubernetes.io/projected/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-kube-api-access-pp9rf\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.585572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.584785 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-model-cache\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.585572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.584839 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-dshm\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.585572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.584869 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tls-certs\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.585572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.584940 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-home\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.585572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.584980 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tmp-dir\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.585572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.585520 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-0dba31\"/\"openshift-service-ca.crt\"" Oct 30 05:24:14.586048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.585620 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-0dba31\"/\"default-dockercfg-pqhsj\"" Oct 30 05:24:14.602547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.599279 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-0dba31\"/\"kube-root-ca.crt\"" Oct 30 05:24:14.602547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.599321 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-0dba31\"/\"auth-test-model-0dba31-kserve-self-signed-certs\"" Oct 30 05:24:14.685947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.685915 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-home\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.686142 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.685964 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tmp-dir\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.686142 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.686120 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp9rf\" (UniqueName: \"kubernetes.io/projected/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-kube-api-access-pp9rf\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.686317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.686295 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-model-cache\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.686425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.686315 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-home\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.686425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.686353 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-dshm\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.686425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.686405 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tls-certs\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.686714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.686448 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tmp-dir\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.686714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.686593 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-model-cache\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.689005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.688980 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-dshm\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.689266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.689238 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tls-certs\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.710182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.710152 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp9rf\" (UniqueName: \"kubernetes.io/projected/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-kube-api-access-pp9rf\") pod \"auth-test-model-0dba31-kserve-bf4884c8f-9ng6r\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:14.905448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:14.899928 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:15.351970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:15.351928 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 05:24:15.352178 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:15.351990 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 05:24:15.354050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:15.354029 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r"] Oct 30 05:24:15.357429 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:24:15.357396 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde134eba_c93e_4e36_b6bc_4a7dbe2c3685.slice/crio-a4ace8868c1ec32d6355cf39b2b0f01177a1747fb61fe6627dc0babcbdd9d295 WatchSource:0}: Error finding container a4ace8868c1ec32d6355cf39b2b0f01177a1747fb61fe6627dc0babcbdd9d295: Status 404 returned error can't find the container with id a4ace8868c1ec32d6355cf39b2b0f01177a1747fb61fe6627dc0babcbdd9d295 Oct 30 05:24:16.072219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:16.072176 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" event={"ID":"de134eba-c93e-4e36-b6bc-4a7dbe2c3685","Type":"ContainerStarted","Data":"d68a92d32fb5afd2703261a5402f4b61b5ff478fdd0f8ec583c19ee5476b2369"} Oct 30 05:24:16.072675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:16.072235 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" event={"ID":"de134eba-c93e-4e36-b6bc-4a7dbe2c3685","Type":"ContainerStarted","Data":"a4ace8868c1ec32d6355cf39b2b0f01177a1747fb61fe6627dc0babcbdd9d295"} Oct 30 05:24:16.123479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:16.115433 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" podStartSLOduration=2.115416042 podStartE2EDuration="2.115416042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:24:16.112286986 +0000 UTC m=+2380.640053773" watchObservedRunningTime="2026-10-30 05:24:16.115416042 +0000 UTC m=+2380.643182829" Oct 30 05:24:16.336478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:16.327748 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 05:24:16.336478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:16.327811 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 05:24:16.903382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:16.902445 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:17.322877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:17.322840 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:24:17.323410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:17.323388 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:24:17.324166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:17.324139 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:24:19.061381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.058178 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg"] Oct 30 05:24:19.220513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.220481 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7"] Oct 30 05:24:19.224193 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.224166 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" Oct 30 05:24:19.242685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.241306 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:24:19.242685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.241417 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:24:19.247568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.247329 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7"] Oct 30 05:24:19.303384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.302929 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:24:19.303384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.303025 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:24:19.331441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.321545 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7\" (UID: \"edf1b9de-bb3d-4749-ab81-68f466b23e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" Oct 30 05:24:19.331441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.321615 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7\" (UID: \"edf1b9de-bb3d-4749-ab81-68f466b23e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" Oct 30 05:24:19.331441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.321827 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxd74\" (UniqueName: \"kubernetes.io/projected/edf1b9de-bb3d-4749-ab81-68f466b23e65-kube-api-access-zxd74\") pod \"maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7\" (UID: \"edf1b9de-bb3d-4749-ab81-68f466b23e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" Oct 30 05:24:19.423478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.423444 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7\" (UID: \"edf1b9de-bb3d-4749-ab81-68f466b23e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" Oct 30 05:24:19.423673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.423498 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7\" (UID: \"edf1b9de-bb3d-4749-ab81-68f466b23e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" Oct 30 05:24:19.423673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.423655 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxd74\" (UniqueName: \"kubernetes.io/projected/edf1b9de-bb3d-4749-ab81-68f466b23e65-kube-api-access-zxd74\") pod \"maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7\" (UID: \"edf1b9de-bb3d-4749-ab81-68f466b23e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" Oct 30 05:24:19.424281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.424256 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7\" (UID: \"edf1b9de-bb3d-4749-ab81-68f466b23e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" Oct 30 05:24:19.426118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.426090 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7\" (UID: \"edf1b9de-bb3d-4749-ab81-68f466b23e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" Oct 30 05:24:19.447032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.447009 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxd74\" (UniqueName: \"kubernetes.io/projected/edf1b9de-bb3d-4749-ab81-68f466b23e65-kube-api-access-zxd74\") pod \"maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7\" (UID: \"edf1b9de-bb3d-4749-ab81-68f466b23e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" Oct 30 05:24:19.541833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.541800 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" Oct 30 05:24:19.853067 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:24:19.852990 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf1b9de_bb3d_4749_ab81_68f466b23e65.slice/crio-ac7206407267c7b43e20c684ae0b8777caaa552c116b10becc2a383f2da101f6 WatchSource:0}: Error finding container ac7206407267c7b43e20c684ae0b8777caaa552c116b10becc2a383f2da101f6: Status 404 returned error can't find the container with id ac7206407267c7b43e20c684ae0b8777caaa552c116b10becc2a383f2da101f6 Oct 30 05:24:19.856147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:19.856118 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7"] Oct 30 05:24:20.122952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:20.122876 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" event={"ID":"edf1b9de-bb3d-4749-ab81-68f466b23e65","Type":"ContainerStarted","Data":"ac7206407267c7b43e20c684ae0b8777caaa552c116b10becc2a383f2da101f6"} Oct 30 05:24:20.656496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:20.656461 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz"] Oct 30 05:24:21.134931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.134893 6587 generic.go:356] "Generic (PLEG): container finished" podID="edf1b9de-bb3d-4749-ab81-68f466b23e65" containerID="e600cabd6e21e178102acac701ac2f617442d9c0f2fb005cf85455e7f9a5047b" exitCode=0 Oct 30 05:24:21.135402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.134983 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" event={"ID":"edf1b9de-bb3d-4749-ab81-68f466b23e65","Type":"ContainerDied","Data":"e600cabd6e21e178102acac701ac2f617442d9c0f2fb005cf85455e7f9a5047b"} Oct 30 05:24:21.179885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.179854 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8"] Oct 30 05:24:21.184763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.184737 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.205234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.205201 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ff4760\"/\"default-dockercfg-4xj5z\"" Oct 30 05:24:21.205443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.205319 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ff4760\"/\"openshift-service-ca.crt\"" Oct 30 05:24:21.205543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.205473 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ff4760\"/\"auth-test-model-ff4760-kserve-self-signed-certs\"" Oct 30 05:24:21.205611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.205545 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ff4760\"/\"kube-root-ca.crt\"" Oct 30 05:24:21.206174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.206001 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8"] Oct 30 05:24:21.274969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.274895 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-model-cache\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.274969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.274933 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-home\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.275155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.275110 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tls-certs\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.275234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.275220 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-dshm\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.275280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.275262 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tmp-dir\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.275510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.275492 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8l26\" (UniqueName: \"kubernetes.io/projected/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-kube-api-access-j8l26\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.376893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.376856 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8l26\" (UniqueName: \"kubernetes.io/projected/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-kube-api-access-j8l26\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.377079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.376922 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-model-cache\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.377079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.376943 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-home\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.377079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.377010 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tls-certs\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.377269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.377084 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-dshm\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.377269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.377134 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tmp-dir\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.377460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.377433 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-model-cache\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.377553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.377519 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-home\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.377621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.377576 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tmp-dir\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.379504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.379480 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-dshm\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.379732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.379712 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tls-certs\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.397870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.397842 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8l26\" (UniqueName: \"kubernetes.io/projected/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-kube-api-access-j8l26\") pod \"auth-test-model-ff4760-kserve-6d4dd4797-fldz8\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.527434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.526661 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:21.667381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.649324 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:21.667381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.651749 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:21.946810 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:24:21.946773 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1732cb93_3b7c_4eea_a360_8c4cf9e6943d.slice/crio-f6fe1bd2fd8c45e8d9e4295c6fb1f135d874e74068ef660e62ff6636ebf88736 WatchSource:0}: Error finding container f6fe1bd2fd8c45e8d9e4295c6fb1f135d874e74068ef660e62ff6636ebf88736: Status 404 returned error can't find the container with id f6fe1bd2fd8c45e8d9e4295c6fb1f135d874e74068ef660e62ff6636ebf88736 Oct 30 05:24:21.949613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:21.949581 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8"] Oct 30 05:24:22.148691 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.148656 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" event={"ID":"1732cb93-3b7c-4eea-a360-8c4cf9e6943d","Type":"ContainerStarted","Data":"6c12bcafa8459adab42be854b0459096ef0c01938899301258ef1e97d26fa356"} Oct 30 05:24:22.149330 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.148703 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" event={"ID":"1732cb93-3b7c-4eea-a360-8c4cf9e6943d","Type":"ContainerStarted","Data":"f6fe1bd2fd8c45e8d9e4295c6fb1f135d874e74068ef660e62ff6636ebf88736"} Oct 30 05:24:22.163229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.163181 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:24:22.193034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.192948 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" podStartSLOduration=1.192929694 podStartE2EDuration="1.192929694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:24:22.190862648 +0000 UTC m=+2386.718629433" watchObservedRunningTime="2026-10-30 05:24:22.192929694 +0000 UTC m=+2386.720696480" Oct 30 05:24:22.329296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.329266 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" Oct 30 05:24:22.425596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.425528 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-xks-ca\") pod \"edf1b9de-bb3d-4749-ab81-68f466b23e65\" (UID: \"edf1b9de-bb3d-4749-ab81-68f466b23e65\") " Oct 30 05:24:22.425815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.425736 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf1b9de-bb3d-4749-ab81-68f466b23e65-kube-api-access-zxd74\" (UniqueName: \"kubernetes.io/projected/edf1b9de-bb3d-4749-ab81-68f466b23e65-kube-api-access-zxd74\") pod \"edf1b9de-bb3d-4749-ab81-68f466b23e65\" (UID: \"edf1b9de-bb3d-4749-ab81-68f466b23e65\") " Oct 30 05:24:22.426019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.425996 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-openshift-ca\") pod \"edf1b9de-bb3d-4749-ab81-68f466b23e65\" (UID: \"edf1b9de-bb3d-4749-ab81-68f466b23e65\") " Oct 30 05:24:22.427013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.426984 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-openshift-ca" pod "edf1b9de-bb3d-4749-ab81-68f466b23e65" (UID: "edf1b9de-bb3d-4749-ab81-68f466b23e65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:24:22.430409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.429183 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-xks-ca" pod "edf1b9de-bb3d-4749-ab81-68f466b23e65" (UID: "edf1b9de-bb3d-4749-ab81-68f466b23e65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:24:22.430409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.429583 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf1b9de-bb3d-4749-ab81-68f466b23e65-kube-api-access-zxd74" pod "edf1b9de-bb3d-4749-ab81-68f466b23e65" (UID: "edf1b9de-bb3d-4749-ab81-68f466b23e65"). InnerVolumeSpecName "kube-api-access-zxd74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:24:22.528994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.528909 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:22.528994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.528939 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:22.528994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.528963 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxd74\" (UniqueName: \"kubernetes.io/projected/edf1b9de-bb3d-4749-ab81-68f466b23e65-kube-api-access-zxd74\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:22.528994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:22.528973 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf1b9de-bb3d-4749-ab81-68f466b23e65-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:23.161171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:23.161081 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" Oct 30 05:24:23.161171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:23.161117 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7" event={"ID":"edf1b9de-bb3d-4749-ab81-68f466b23e65","Type":"ContainerDied","Data":"ac7206407267c7b43e20c684ae0b8777caaa552c116b10becc2a383f2da101f6"} Oct 30 05:24:23.161720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:23.161171 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac7206407267c7b43e20c684ae0b8777caaa552c116b10becc2a383f2da101f6" Oct 30 05:24:24.900617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:24.900575 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:24.913710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:24.913686 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:25.190078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:25.189995 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:24:25.657747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:25.657704 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" podUID="65bfb1ee-74e7-4fb4-8738-607e42dec925" containerName="payload-pre-processing" containerID="cri-o://03e7ddc4d643ad06d18252a96f89e0a293621c0856ccf4d78d25f365a511018b" gracePeriod=25 Oct 30 05:24:25.987912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:25.987881 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" Oct 30 05:24:26.006343 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:24:26.006299 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:24:26.085219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.085180 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bfb1ee-74e7-4fb4-8738-607e42dec925-kube-api-access-ktm7n\" (UniqueName: \"kubernetes.io/projected/65bfb1ee-74e7-4fb4-8738-607e42dec925-kube-api-access-ktm7n\") pod \"65bfb1ee-74e7-4fb4-8738-607e42dec925\" (UID: \"65bfb1ee-74e7-4fb4-8738-607e42dec925\") " Oct 30 05:24:26.085441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.085315 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bfb1ee-74e7-4fb4-8738-607e42dec925-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bfb1ee-74e7-4fb4-8738-607e42dec925-plugins-config-volume\") pod \"65bfb1ee-74e7-4fb4-8738-607e42dec925\" (UID: \"65bfb1ee-74e7-4fb4-8738-607e42dec925\") " Oct 30 05:24:26.085843 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.085812 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bfb1ee-74e7-4fb4-8738-607e42dec925-plugins-config-volume" pod "65bfb1ee-74e7-4fb4-8738-607e42dec925" (UID: "65bfb1ee-74e7-4fb4-8738-607e42dec925"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:24:26.086141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.086120 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bfb1ee-74e7-4fb4-8738-607e42dec925-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:26.087754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.087728 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bfb1ee-74e7-4fb4-8738-607e42dec925-kube-api-access-ktm7n" pod "65bfb1ee-74e7-4fb4-8738-607e42dec925" (UID: "65bfb1ee-74e7-4fb4-8738-607e42dec925"). InnerVolumeSpecName "kube-api-access-ktm7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:24:26.187847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.187809 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktm7n\" (UniqueName: \"kubernetes.io/projected/65bfb1ee-74e7-4fb4-8738-607e42dec925-kube-api-access-ktm7n\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:26.190803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.190736 6587 generic.go:356] "Generic (PLEG): container finished" podID="65bfb1ee-74e7-4fb4-8738-607e42dec925" containerID="03e7ddc4d643ad06d18252a96f89e0a293621c0856ccf4d78d25f365a511018b" exitCode=0 Oct 30 05:24:26.190803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.190786 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" Oct 30 05:24:26.190967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.190813 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" event={"ID":"65bfb1ee-74e7-4fb4-8738-607e42dec925","Type":"ContainerDied","Data":"03e7ddc4d643ad06d18252a96f89e0a293621c0856ccf4d78d25f365a511018b"} Oct 30 05:24:26.190967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.190858 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz" event={"ID":"65bfb1ee-74e7-4fb4-8738-607e42dec925","Type":"ContainerDied","Data":"da0f13e32507c9ae449e83f6e102644bdee266f9d5de76d73e7881e97c3d3f40"} Oct 30 05:24:26.190967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.190878 6587 scope.go:122] "RemoveContainer" containerID="03e7ddc4d643ad06d18252a96f89e0a293621c0856ccf4d78d25f365a511018b" Oct 30 05:24:26.203504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.203485 6587 scope.go:122] "RemoveContainer" containerID="03e7ddc4d643ad06d18252a96f89e0a293621c0856ccf4d78d25f365a511018b" Oct 30 05:24:26.203781 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:24:26.203760 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e7ddc4d643ad06d18252a96f89e0a293621c0856ccf4d78d25f365a511018b\": container with ID starting with 03e7ddc4d643ad06d18252a96f89e0a293621c0856ccf4d78d25f365a511018b not found: ID does not exist" containerID="03e7ddc4d643ad06d18252a96f89e0a293621c0856ccf4d78d25f365a511018b" Oct 30 05:24:26.203818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.203790 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e7ddc4d643ad06d18252a96f89e0a293621c0856ccf4d78d25f365a511018b"} err="failed to get container status \"03e7ddc4d643ad06d18252a96f89e0a293621c0856ccf4d78d25f365a511018b\": rpc error: code = NotFound desc = could not find container \"03e7ddc4d643ad06d18252a96f89e0a293621c0856ccf4d78d25f365a511018b\": container with ID starting with 03e7ddc4d643ad06d18252a96f89e0a293621c0856ccf4d78d25f365a511018b not found: ID does not exist" Oct 30 05:24:26.253897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.253858 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz"] Oct 30 05:24:26.262007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:26.261977 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz"] Oct 30 05:24:27.586966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:27.586929 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65bfb1ee-74e7-4fb4-8738-607e42dec925" path="/var/lib/kubelet/pods/65bfb1ee-74e7-4fb4-8738-607e42dec925/volumes" Oct 30 05:24:29.701544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:29.701502 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7"] Oct 30 05:24:29.739472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:29.739431 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc"] Oct 30 05:24:31.528962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:31.528926 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:31.542143 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:31.542116 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:32.259446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:32.259411 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:24:34.087617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.087554 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" podUID="427c21bb-df54-419c-a7da-25e015c0dec4" containerName="main" containerID="cri-o://b1588dd7e1bf85401a28516303c0f13cfcad3ca4da56d6026a7e645653a51b00" gracePeriod=45 Oct 30 05:24:34.272728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.272696 6587 generic.go:356] "Generic (PLEG): container finished" podID="427c21bb-df54-419c-a7da-25e015c0dec4" containerID="b1588dd7e1bf85401a28516303c0f13cfcad3ca4da56d6026a7e645653a51b00" exitCode=0 Oct 30 05:24:34.272898 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.272878 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" event={"ID":"427c21bb-df54-419c-a7da-25e015c0dec4","Type":"ContainerDied","Data":"b1588dd7e1bf85401a28516303c0f13cfcad3ca4da56d6026a7e645653a51b00"} Oct 30 05:24:34.394429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.394404 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:24:34.540126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.540094 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427c21bb-df54-419c-a7da-25e015c0dec4-kube-api-access-xwq82\" (UniqueName: \"kubernetes.io/projected/427c21bb-df54-419c-a7da-25e015c0dec4-kube-api-access-xwq82\") pod \"427c21bb-df54-419c-a7da-25e015c0dec4\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " Oct 30 05:24:34.540349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.540294 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-home\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-home\") pod \"427c21bb-df54-419c-a7da-25e015c0dec4\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " Oct 30 05:24:34.540444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.540370 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/427c21bb-df54-419c-a7da-25e015c0dec4-tls-certs\" (UniqueName: \"kubernetes.io/secret/427c21bb-df54-419c-a7da-25e015c0dec4-tls-certs\") pod \"427c21bb-df54-419c-a7da-25e015c0dec4\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " Oct 30 05:24:34.540516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.540463 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-model-cache\") pod \"427c21bb-df54-419c-a7da-25e015c0dec4\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " Oct 30 05:24:34.540577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.540553 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-tmp-dir\") pod \"427c21bb-df54-419c-a7da-25e015c0dec4\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " Oct 30 05:24:34.540713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.540685 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-dshm\") pod \"427c21bb-df54-419c-a7da-25e015c0dec4\" (UID: \"427c21bb-df54-419c-a7da-25e015c0dec4\") " Oct 30 05:24:34.540849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.540669 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-home" pod "427c21bb-df54-419c-a7da-25e015c0dec4" (UID: "427c21bb-df54-419c-a7da-25e015c0dec4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:24:34.540939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.540758 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-model-cache" pod "427c21bb-df54-419c-a7da-25e015c0dec4" (UID: "427c21bb-df54-419c-a7da-25e015c0dec4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:24:34.541232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.541203 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-tmp-dir" pod "427c21bb-df54-419c-a7da-25e015c0dec4" (UID: "427c21bb-df54-419c-a7da-25e015c0dec4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:24:34.541572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.541543 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:34.541572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.541574 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:34.541754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.541587 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:34.542855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.542828 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427c21bb-df54-419c-a7da-25e015c0dec4-kube-api-access-xwq82" pod "427c21bb-df54-419c-a7da-25e015c0dec4" (UID: "427c21bb-df54-419c-a7da-25e015c0dec4"). InnerVolumeSpecName "kube-api-access-xwq82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:24:34.543001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.542980 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/427c21bb-df54-419c-a7da-25e015c0dec4-tls-certs" pod "427c21bb-df54-419c-a7da-25e015c0dec4" (UID: "427c21bb-df54-419c-a7da-25e015c0dec4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:24:34.543063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.542987 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-dshm" pod "427c21bb-df54-419c-a7da-25e015c0dec4" (UID: "427c21bb-df54-419c-a7da-25e015c0dec4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:24:34.643006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.642926 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwq82\" (UniqueName: \"kubernetes.io/projected/427c21bb-df54-419c-a7da-25e015c0dec4-kube-api-access-xwq82\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:34.643006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.642957 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427c21bb-df54-419c-a7da-25e015c0dec4-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:34.643006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.642968 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427c21bb-df54-419c-a7da-25e015c0dec4-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:34.702460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.702403 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" podUID="aeea73b7-9493-4e07-9ede-23123f96f897" containerName="payload-processing" containerID="cri-o://ad8656fd05d3ab92806c38deb7628a8d68a27afcec8198923c2907c098de6b21" gracePeriod=25 Oct 30 05:24:34.740731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:34.740691 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" podUID="b10ce723-0350-4b2f-8180-32bea097068c" containerName="payload-pre-processing" containerID="cri-o://9e66fa0ba50ae834f4cfaf302e00d302c7a296e71aa872a94464b64152d95187" gracePeriod=25 Oct 30 05:24:35.283302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:35.283273 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" Oct 30 05:24:35.283784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:35.283307 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg" event={"ID":"427c21bb-df54-419c-a7da-25e015c0dec4","Type":"ContainerDied","Data":"d4a99db857c73f22b1eb8731f7a21dc67384068f8b6a3b0a56aea630ed14dd0b"} Oct 30 05:24:35.283784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:35.283374 6587 scope.go:122] "RemoveContainer" containerID="b1588dd7e1bf85401a28516303c0f13cfcad3ca4da56d6026a7e645653a51b00" Oct 30 05:24:35.343969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:35.343931 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg"] Oct 30 05:24:35.357605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:35.357570 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-68b893/test-model-68b893-kserve-55f4f9cbc9-cxqdg"] Oct 30 05:24:35.461722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:35.461690 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv"] Oct 30 05:24:35.638818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:35.638787 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427c21bb-df54-419c-a7da-25e015c0dec4" path="/var/lib/kubelet/pods/427c21bb-df54-419c-a7da-25e015c0dec4/volumes" Oct 30 05:24:36.110193 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:36.110163 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:24:36.111247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:36.111225 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:24:39.902635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:39.902597 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" podUID="aeea73b7-9493-4e07-9ede-23123f96f897" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:24:39.903184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:39.902608 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" podUID="b10ce723-0350-4b2f-8180-32bea097068c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:24:40.703461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:40.703427 6587 scope.go:122] "RemoveContainer" containerID="5211f742eb578720d118ba007c8c411b01c155a6e212bb7b7dd66527a008b955" Oct 30 05:24:40.714604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:40.714581 6587 scope.go:122] "RemoveContainer" containerID="37e18287c32eec62814c0a663916f955ab3fdb9c9a3c8ce6374965813f8052f5" Oct 30 05:24:40.729533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:40.729510 6587 scope.go:122] "RemoveContainer" containerID="d2d8512dd194ce8e4b45208430faec5e9c1642a38a92d65594db63aeb2d2ad4c" Oct 30 05:24:47.460205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:47.460143 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7"] Oct 30 05:24:47.470994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:47.470962 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-802b8a-b2kq7"] Oct 30 05:24:47.588512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:47.588476 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf1b9de-bb3d-4749-ab81-68f466b23e65" path="/var/lib/kubelet/pods/edf1b9de-bb3d-4749-ab81-68f466b23e65/volumes" Oct 30 05:24:48.031195 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:48.031156 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5"] Oct 30 05:24:48.047280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:48.047243 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dbccfc3-c2nk5"] Oct 30 05:24:49.584718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:49.584678 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d639fb54-565e-4146-b979-6e52acaed8d3" path="/var/lib/kubelet/pods/d639fb54-565e-4146-b979-6e52acaed8d3/volumes" Oct 30 05:24:49.903389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:49.903278 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" podUID="b10ce723-0350-4b2f-8180-32bea097068c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:24:49.903389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:49.903278 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" podUID="aeea73b7-9493-4e07-9ede-23123f96f897" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:24:50.500383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:50.495787 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" podUID="3de242ad-68be-4de3-8d17-7d3d7c96ab32" containerName="main" containerID="cri-o://d54c4d61678130895b044418218ef5cdadc3c3cd960d365cf199dcf97537e9ab" gracePeriod=45 Oct 30 05:24:50.653080 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:50.652763 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" podUID="3de242ad-68be-4de3-8d17-7d3d7c96ab32" containerName="main" probeResult="failure" output="Get \"https://10.134.1.9:8000/ready\": dial tcp 10.134.1.9:8000: connect: connection refused" Oct 30 05:24:51.531291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:51.531168 6587 generic.go:356] "Generic (PLEG): container finished" podID="3de242ad-68be-4de3-8d17-7d3d7c96ab32" containerID="d54c4d61678130895b044418218ef5cdadc3c3cd960d365cf199dcf97537e9ab" exitCode=0 Oct 30 05:24:51.531899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:51.531578 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" event={"ID":"3de242ad-68be-4de3-8d17-7d3d7c96ab32","Type":"ContainerDied","Data":"d54c4d61678130895b044418218ef5cdadc3c3cd960d365cf199dcf97537e9ab"} Oct 30 05:24:51.647989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:51.647938 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" podUID="3de242ad-68be-4de3-8d17-7d3d7c96ab32" containerName="main" probeResult="failure" output="Get \"https://10.134.1.9:8000/ready\": dial tcp 10.134.1.9:8000: connect: connection refused" Oct 30 05:24:51.648326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:51.648307 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:24:52.644492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.644164 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:24:52.747409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.742752 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tmp-dir\") pod \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " Oct 30 05:24:52.747409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.742819 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de242ad-68be-4de3-8d17-7d3d7c96ab32-kube-api-access-smxcp\" (UniqueName: \"kubernetes.io/projected/3de242ad-68be-4de3-8d17-7d3d7c96ab32-kube-api-access-smxcp\") pod \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " Oct 30 05:24:52.747409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.742886 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-home\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-home\") pod \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " Oct 30 05:24:52.747409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.742917 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-dshm\") pod \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " Oct 30 05:24:52.747409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.743155 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tls-certs\" (UniqueName: \"kubernetes.io/secret/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tls-certs\") pod \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " Oct 30 05:24:52.747409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.743220 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-model-cache\") pod \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\" (UID: \"3de242ad-68be-4de3-8d17-7d3d7c96ab32\") " Oct 30 05:24:52.747409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.744282 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-model-cache" pod "3de242ad-68be-4de3-8d17-7d3d7c96ab32" (UID: "3de242ad-68be-4de3-8d17-7d3d7c96ab32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:24:52.747409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.744527 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tmp-dir" pod "3de242ad-68be-4de3-8d17-7d3d7c96ab32" (UID: "3de242ad-68be-4de3-8d17-7d3d7c96ab32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:24:52.747409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.747313 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-dshm" pod "3de242ad-68be-4de3-8d17-7d3d7c96ab32" (UID: "3de242ad-68be-4de3-8d17-7d3d7c96ab32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:24:52.748073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.747499 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de242ad-68be-4de3-8d17-7d3d7c96ab32-kube-api-access-smxcp" pod "3de242ad-68be-4de3-8d17-7d3d7c96ab32" (UID: "3de242ad-68be-4de3-8d17-7d3d7c96ab32"). InnerVolumeSpecName "kube-api-access-smxcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:24:52.748073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.747604 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-home" pod "3de242ad-68be-4de3-8d17-7d3d7c96ab32" (UID: "3de242ad-68be-4de3-8d17-7d3d7c96ab32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:24:52.750833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.750803 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tls-certs" pod "3de242ad-68be-4de3-8d17-7d3d7c96ab32" (UID: "3de242ad-68be-4de3-8d17-7d3d7c96ab32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:24:52.845973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.845940 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:52.845973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.845967 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:52.845973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.845976 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:52.846223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.845985 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smxcp\" (UniqueName: \"kubernetes.io/projected/3de242ad-68be-4de3-8d17-7d3d7c96ab32-kube-api-access-smxcp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:52.846223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.845995 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:52.846223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:52.846005 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de242ad-68be-4de3-8d17-7d3d7c96ab32-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:53.241309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.241265 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l"] Oct 30 05:24:53.243036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.243000 6587 cpu_manager.go:414] "containerMap: removing container" podUID="edf1b9de-bb3d-4749-ab81-68f466b23e65" containerName="revoke-keys" Oct 30 05:24:53.243036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.243031 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf1b9de-bb3d-4749-ab81-68f466b23e65" containerName="revoke-keys" Oct 30 05:24:53.243251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.243054 6587 cpu_manager.go:414] "containerMap: removing container" podUID="427c21bb-df54-419c-a7da-25e015c0dec4" containerName="main" Oct 30 05:24:53.243251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.243064 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427c21bb-df54-419c-a7da-25e015c0dec4" containerName="main" Oct 30 05:24:53.243251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.243083 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3de242ad-68be-4de3-8d17-7d3d7c96ab32" containerName="main" Oct 30 05:24:53.243251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.243093 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de242ad-68be-4de3-8d17-7d3d7c96ab32" containerName="main" Oct 30 05:24:53.243251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.243145 6587 cpu_manager.go:414] "containerMap: removing container" podUID="65bfb1ee-74e7-4fb4-8738-607e42dec925" containerName="payload-pre-processing" Oct 30 05:24:53.243251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.243154 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bfb1ee-74e7-4fb4-8738-607e42dec925" containerName="payload-pre-processing" Oct 30 05:24:53.243560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.243401 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf1b9de-bb3d-4749-ab81-68f466b23e65" containerName="revoke-keys" Oct 30 05:24:53.243560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.243427 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="427c21bb-df54-419c-a7da-25e015c0dec4" containerName="main" Oct 30 05:24:53.243560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.243436 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de242ad-68be-4de3-8d17-7d3d7c96ab32" containerName="main" Oct 30 05:24:53.243560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.243455 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bfb1ee-74e7-4fb4-8738-607e42dec925" containerName="payload-pre-processing" Oct 30 05:24:53.247091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.247069 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.274333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.274227 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-262c10\"/\"kube-root-ca.crt\"" Oct 30 05:24:53.276342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.276236 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-262c10\"/\"openshift-service-ca.crt\"" Oct 30 05:24:53.279504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.279478 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-262c10\"/\"default-dockercfg-ssckh\"" Oct 30 05:24:53.279678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.279656 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-262c10\"/\"auth-test-model-262c10-kserve-self-signed-certs\"" Oct 30 05:24:53.281121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.281092 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l"] Oct 30 05:24:53.355215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.355172 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwgkt\" (UniqueName: \"kubernetes.io/projected/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-kube-api-access-zwgkt\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.355215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.355216 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tls-certs\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.355504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.355299 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-dshm\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.355504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.355381 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-model-cache\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.355504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.355454 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tmp-dir\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.355682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.355541 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-home\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.456881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.456843 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tmp-dir\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.457144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.456970 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-home\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.457144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.457105 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwgkt\" (UniqueName: \"kubernetes.io/projected/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-kube-api-access-zwgkt\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.457144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.457139 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tls-certs\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.457378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.457191 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-dshm\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.457378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.457294 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-model-cache\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.457378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.457303 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tmp-dir\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.457553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.457396 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-home\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.457645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.457628 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-model-cache\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.459665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.459641 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-dshm\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.459974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.459915 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tls-certs\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.479793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.479767 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwgkt\" (UniqueName: \"kubernetes.io/projected/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-kube-api-access-zwgkt\") pod \"auth-test-model-262c10-kserve-76f9465fcc-gt65l\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.567939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.567905 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:53.621842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.621306 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" event={"ID":"3de242ad-68be-4de3-8d17-7d3d7c96ab32","Type":"ContainerDied","Data":"97a649d6031a684ea5f35bade84ac9334a53ca4cc1b0898ccbd632a2ad06735a"} Oct 30 05:24:53.621842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.621396 6587 scope.go:122] "RemoveContainer" containerID="d54c4d61678130895b044418218ef5cdadc3c3cd960d365cf199dcf97537e9ab" Oct 30 05:24:53.621842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.621595 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv" Oct 30 05:24:53.849792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.849762 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv"] Oct 30 05:24:53.863750 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:53.863708 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-097d90/auth-test-model-097d90-kserve-67988d4b89-v48fv"] Oct 30 05:24:54.023192 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:24:54.023132 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40b7c36f_e157_4cc4_a04e_962fa1a7cc77.slice/crio-87a58ed989425512f80cf83358ca0d3ff135b2115c2185b488f416e1305086a6 WatchSource:0}: Error finding container 87a58ed989425512f80cf83358ca0d3ff135b2115c2185b488f416e1305086a6: Status 404 returned error can't find the container with id 87a58ed989425512f80cf83358ca0d3ff135b2115c2185b488f416e1305086a6 Oct 30 05:24:54.025320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:54.025276 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l"] Oct 30 05:24:54.384960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:54.384865 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t"] Oct 30 05:24:54.631959 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:54.631895 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" event={"ID":"40b7c36f-e157-4cc4-a04e-962fa1a7cc77","Type":"ContainerStarted","Data":"cdd98a7ccde20d19aec50711daee7bcb6864d351ebc62060b367d08ae445c250"} Oct 30 05:24:54.631959 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:54.631963 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" event={"ID":"40b7c36f-e157-4cc4-a04e-962fa1a7cc77","Type":"ContainerStarted","Data":"87a58ed989425512f80cf83358ca0d3ff135b2115c2185b488f416e1305086a6"} Oct 30 05:24:54.679453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:54.679347 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" podStartSLOduration=1.6793331550000001 podStartE2EDuration="1.679333155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:24:54.678239948 +0000 UTC m=+2419.206006733" watchObservedRunningTime="2026-10-30 05:24:54.679333155 +0000 UTC m=+2419.207099941" Oct 30 05:24:55.086518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.086481 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" Oct 30 05:24:55.118684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.118663 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" Oct 30 05:24:55.230565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.230484 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeea73b7-9493-4e07-9ede-23123f96f897-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeea73b7-9493-4e07-9ede-23123f96f897-plugins-config-volume\") pod \"aeea73b7-9493-4e07-9ede-23123f96f897\" (UID: \"aeea73b7-9493-4e07-9ede-23123f96f897\") " Oct 30 05:24:55.230565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.230524 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeea73b7-9493-4e07-9ede-23123f96f897-kube-api-access-tmdw9\" (UniqueName: \"kubernetes.io/projected/aeea73b7-9493-4e07-9ede-23123f96f897-kube-api-access-tmdw9\") pod \"aeea73b7-9493-4e07-9ede-23123f96f897\" (UID: \"aeea73b7-9493-4e07-9ede-23123f96f897\") " Oct 30 05:24:55.230774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.230618 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b10ce723-0350-4b2f-8180-32bea097068c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10ce723-0350-4b2f-8180-32bea097068c-plugins-config-volume\") pod \"b10ce723-0350-4b2f-8180-32bea097068c\" (UID: \"b10ce723-0350-4b2f-8180-32bea097068c\") " Oct 30 05:24:55.230918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.230895 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10ce723-0350-4b2f-8180-32bea097068c-kube-api-access-jw2gt\" (UniqueName: \"kubernetes.io/projected/b10ce723-0350-4b2f-8180-32bea097068c-kube-api-access-jw2gt\") pod \"b10ce723-0350-4b2f-8180-32bea097068c\" (UID: \"b10ce723-0350-4b2f-8180-32bea097068c\") " Oct 30 05:24:55.230994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.230929 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeea73b7-9493-4e07-9ede-23123f96f897-plugins-config-volume" pod "aeea73b7-9493-4e07-9ede-23123f96f897" (UID: "aeea73b7-9493-4e07-9ede-23123f96f897"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:24:55.230994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.230974 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10ce723-0350-4b2f-8180-32bea097068c-plugins-config-volume" pod "b10ce723-0350-4b2f-8180-32bea097068c" (UID: "b10ce723-0350-4b2f-8180-32bea097068c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:24:55.231573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.231554 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeea73b7-9493-4e07-9ede-23123f96f897-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:55.231664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.231576 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10ce723-0350-4b2f-8180-32bea097068c-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:55.233253 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.233216 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10ce723-0350-4b2f-8180-32bea097068c-kube-api-access-jw2gt" pod "b10ce723-0350-4b2f-8180-32bea097068c" (UID: "b10ce723-0350-4b2f-8180-32bea097068c"). InnerVolumeSpecName "kube-api-access-jw2gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:24:55.233600 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.233577 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeea73b7-9493-4e07-9ede-23123f96f897-kube-api-access-tmdw9" pod "aeea73b7-9493-4e07-9ede-23123f96f897" (UID: "aeea73b7-9493-4e07-9ede-23123f96f897"). InnerVolumeSpecName "kube-api-access-tmdw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:24:55.332991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.332954 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmdw9\" (UniqueName: \"kubernetes.io/projected/aeea73b7-9493-4e07-9ede-23123f96f897-kube-api-access-tmdw9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:55.332991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.332987 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw2gt\" (UniqueName: \"kubernetes.io/projected/b10ce723-0350-4b2f-8180-32bea097068c-kube-api-access-jw2gt\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:55.694943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.694821 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de242ad-68be-4de3-8d17-7d3d7c96ab32" path="/var/lib/kubelet/pods/3de242ad-68be-4de3-8d17-7d3d7c96ab32/volumes" Oct 30 05:24:55.696019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.695989 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:24:55.790599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.788005 6587 generic.go:356] "Generic (PLEG): container finished" podID="aeea73b7-9493-4e07-9ede-23123f96f897" containerID="ad8656fd05d3ab92806c38deb7628a8d68a27afcec8198923c2907c098de6b21" exitCode=0 Oct 30 05:24:55.790599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.788452 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" event={"ID":"aeea73b7-9493-4e07-9ede-23123f96f897","Type":"ContainerDied","Data":"ad8656fd05d3ab92806c38deb7628a8d68a27afcec8198923c2907c098de6b21"} Oct 30 05:24:55.790599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.788519 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" event={"ID":"aeea73b7-9493-4e07-9ede-23123f96f897","Type":"ContainerDied","Data":"0eceb360ec6869d128ffcd1427e8f4aac3c704fa6179d1903d0e004abf01dc24"} Oct 30 05:24:55.790599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.788548 6587 scope.go:122] "RemoveContainer" containerID="ad8656fd05d3ab92806c38deb7628a8d68a27afcec8198923c2907c098de6b21" Oct 30 05:24:55.790599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.788691 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7" Oct 30 05:24:55.813588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.813539 6587 generic.go:356] "Generic (PLEG): container finished" podID="b10ce723-0350-4b2f-8180-32bea097068c" containerID="9e66fa0ba50ae834f4cfaf302e00d302c7a296e71aa872a94464b64152d95187" exitCode=0 Oct 30 05:24:55.824273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.814627 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" Oct 30 05:24:55.824886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.824857 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" event={"ID":"b10ce723-0350-4b2f-8180-32bea097068c","Type":"ContainerDied","Data":"9e66fa0ba50ae834f4cfaf302e00d302c7a296e71aa872a94464b64152d95187"} Oct 30 05:24:55.825102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.825082 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc" event={"ID":"b10ce723-0350-4b2f-8180-32bea097068c","Type":"ContainerDied","Data":"8604234b439a6baf4af5c6edde1e74c426f8dc8f997e231f6fba693936c5b89a"} Oct 30 05:24:55.859373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.859307 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7"] Oct 30 05:24:55.860825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.860794 6587 cpu_manager.go:414] "containerMap: removing container" podUID="aeea73b7-9493-4e07-9ede-23123f96f897" containerName="payload-processing" Oct 30 05:24:55.860825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.860824 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeea73b7-9493-4e07-9ede-23123f96f897" containerName="payload-processing" Oct 30 05:24:55.861003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.860855 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b10ce723-0350-4b2f-8180-32bea097068c" containerName="payload-pre-processing" Oct 30 05:24:55.861003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.860864 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10ce723-0350-4b2f-8180-32bea097068c" containerName="payload-pre-processing" Oct 30 05:24:55.861133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.861116 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeea73b7-9493-4e07-9ede-23123f96f897" containerName="payload-processing" Oct 30 05:24:55.861205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.861140 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10ce723-0350-4b2f-8180-32bea097068c" containerName="payload-pre-processing" Oct 30 05:24:55.869238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.869212 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7"] Oct 30 05:24:55.869397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.869263 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc"] Oct 30 05:24:55.869545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.869522 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.870997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.870967 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-certs\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.871135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.871112 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-socket\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.871197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.871160 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-istio-token\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.871332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.871306 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ccq7\" (UniqueName: \"kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-kube-api-access-2ccq7\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.871469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.871446 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111bc68f-98c2-4041-a39f-daa438577eaf-istio-podinfo\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.871593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.871539 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-envoy\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.871593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.871570 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-credential-socket\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.871712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.871637 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-data\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.871775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.871721 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111bc68f-98c2-4041-a39f-daa438577eaf-istiod-ca-cert\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.872855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.872746 6587 scope.go:122] "RemoveContainer" containerID="ad8656fd05d3ab92806c38deb7628a8d68a27afcec8198923c2907c098de6b21" Oct 30 05:24:55.874170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.874150 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" Oct 30 05:24:55.874265 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:24:55.874170 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad8656fd05d3ab92806c38deb7628a8d68a27afcec8198923c2907c098de6b21\": container with ID starting with ad8656fd05d3ab92806c38deb7628a8d68a27afcec8198923c2907c098de6b21 not found: ID does not exist" containerID="ad8656fd05d3ab92806c38deb7628a8d68a27afcec8198923c2907c098de6b21" Oct 30 05:24:55.874265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.874200 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad8656fd05d3ab92806c38deb7628a8d68a27afcec8198923c2907c098de6b21"} err="failed to get container status \"ad8656fd05d3ab92806c38deb7628a8d68a27afcec8198923c2907c098de6b21\": rpc error: code = NotFound desc = could not find container \"ad8656fd05d3ab92806c38deb7628a8d68a27afcec8198923c2907c098de6b21\": container with ID starting with ad8656fd05d3ab92806c38deb7628a8d68a27afcec8198923c2907c098de6b21 not found: ID does not exist" Oct 30 05:24:55.874265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.874223 6587 scope.go:122] "RemoveContainer" containerID="9e66fa0ba50ae834f4cfaf302e00d302c7a296e71aa872a94464b64152d95187" Oct 30 05:24:55.893309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.893278 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:24:55.893466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.893430 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-60a531-openshift-default-dockercfg-v4sxt\"" Oct 30 05:24:55.893533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.893515 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:24:55.901880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.901858 6587 scope.go:122] "RemoveContainer" containerID="9e66fa0ba50ae834f4cfaf302e00d302c7a296e71aa872a94464b64152d95187" Oct 30 05:24:55.903332 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:24:55.903234 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e66fa0ba50ae834f4cfaf302e00d302c7a296e71aa872a94464b64152d95187\": container with ID starting with 9e66fa0ba50ae834f4cfaf302e00d302c7a296e71aa872a94464b64152d95187 not found: ID does not exist" containerID="9e66fa0ba50ae834f4cfaf302e00d302c7a296e71aa872a94464b64152d95187" Oct 30 05:24:55.903332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.903272 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e66fa0ba50ae834f4cfaf302e00d302c7a296e71aa872a94464b64152d95187"} err="failed to get container status \"9e66fa0ba50ae834f4cfaf302e00d302c7a296e71aa872a94464b64152d95187\": rpc error: code = NotFound desc = could not find container \"9e66fa0ba50ae834f4cfaf302e00d302c7a296e71aa872a94464b64152d95187\": container with ID starting with 9e66fa0ba50ae834f4cfaf302e00d302c7a296e71aa872a94464b64152d95187 not found: ID does not exist" Oct 30 05:24:55.913646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.913620 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc"] Oct 30 05:24:55.916813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.916790 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc"] Oct 30 05:24:55.938379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.927954 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc"] Oct 30 05:24:55.962000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.961909 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7"] Oct 30 05:24:55.968926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.968895 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7"] Oct 30 05:24:55.973460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.973434 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ccq7\" (UniqueName: \"kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-kube-api-access-2ccq7\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.973719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.973697 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111bc68f-98c2-4041-a39f-daa438577eaf-istio-podinfo\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.973873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.973853 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc\" (UID: \"ae76c413-f444-43bb-9103-7bba5c5a2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" Oct 30 05:24:55.974012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.973998 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-envoy\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.974149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.974133 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-credential-socket\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.974748 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.974521 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-credential-socket\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.975227 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.975160 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-data\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.975365 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.974891 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-data\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.975365 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.975271 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq7g6\" (UniqueName: \"kubernetes.io/projected/ae76c413-f444-43bb-9103-7bba5c5a2343-kube-api-access-lq7g6\") pod \"maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc\" (UID: \"ae76c413-f444-43bb-9103-7bba5c5a2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" Oct 30 05:24:55.975365 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.975342 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc\" (UID: \"ae76c413-f444-43bb-9103-7bba5c5a2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" Oct 30 05:24:55.975657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.975408 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111bc68f-98c2-4041-a39f-daa438577eaf-istiod-ca-cert\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.975657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.975514 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-certs\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.975775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.975664 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-socket\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.975775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.975699 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-istio-token\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.976212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.976178 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-certs\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.976329 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.976299 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-socket\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.976545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.976524 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111bc68f-98c2-4041-a39f-daa438577eaf-istiod-ca-cert\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.976967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.976941 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111bc68f-98c2-4041-a39f-daa438577eaf-istio-podinfo\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.977039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.977023 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-envoy\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:55.987770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.987749 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:24:55.988042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:55.988026 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:24:56.012030 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:24:56.011549 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:24:56.012769 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.012740 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ccq7\" (UniqueName: \"kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-kube-api-access-2ccq7\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:56.014791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.014746 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-istio-token\") pod \"e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:56.079439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.077094 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc\" (UID: \"ae76c413-f444-43bb-9103-7bba5c5a2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" Oct 30 05:24:56.079439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.077183 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq7g6\" (UniqueName: \"kubernetes.io/projected/ae76c413-f444-43bb-9103-7bba5c5a2343-kube-api-access-lq7g6\") pod \"maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc\" (UID: \"ae76c413-f444-43bb-9103-7bba5c5a2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" Oct 30 05:24:56.079439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.077242 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc\" (UID: \"ae76c413-f444-43bb-9103-7bba5c5a2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" Oct 30 05:24:56.079439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.078383 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc\" (UID: \"ae76c413-f444-43bb-9103-7bba5c5a2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" Oct 30 05:24:56.080656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.080623 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc\" (UID: \"ae76c413-f444-43bb-9103-7bba5c5a2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" Oct 30 05:24:56.102316 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.102288 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq7g6\" (UniqueName: \"kubernetes.io/projected/ae76c413-f444-43bb-9103-7bba5c5a2343-kube-api-access-lq7g6\") pod \"maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc\" (UID: \"ae76c413-f444-43bb-9103-7bba5c5a2343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" Oct 30 05:24:56.216218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.216141 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:56.217197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.217167 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" Oct 30 05:24:56.553167 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:24:56.553137 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111bc68f_98c2_4041_a39f_daa438577eaf.slice/crio-4416b2e73ad46c157f6ef305359980ade87342bb9b5ece900114e6134999d44d WatchSource:0}: Error finding container 4416b2e73ad46c157f6ef305359980ade87342bb9b5ece900114e6134999d44d: Status 404 returned error can't find the container with id 4416b2e73ad46c157f6ef305359980ade87342bb9b5ece900114e6134999d44d Oct 30 05:24:56.557492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.556910 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7"] Oct 30 05:24:56.557492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.556910 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:56.557492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.556993 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:56.557492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.557030 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:24:56.591344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.591321 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc"] Oct 30 05:24:56.598720 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:24:56.598694 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae76c413_f444_43bb_9103_7bba5c5a2343.slice/crio-af71dce3e473fb253df813eb995897bb8be1988bba60189b4c51ba23fb6b9734 WatchSource:0}: Error finding container af71dce3e473fb253df813eb995897bb8be1988bba60189b4c51ba23fb6b9734: Status 404 returned error can't find the container with id af71dce3e473fb253df813eb995897bb8be1988bba60189b4c51ba23fb6b9734 Oct 30 05:24:56.828769 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.828676 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" event={"ID":"111bc68f-98c2-4041-a39f-daa438577eaf","Type":"ContainerStarted","Data":"6f0835455192fd83e9ef561dd2773d9e3402b63514a24b6f502e56e637b020fa"} Oct 30 05:24:56.828769 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.828749 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" event={"ID":"111bc68f-98c2-4041-a39f-daa438577eaf","Type":"ContainerStarted","Data":"4416b2e73ad46c157f6ef305359980ade87342bb9b5ece900114e6134999d44d"} Oct 30 05:24:56.831267 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.831239 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" event={"ID":"ae76c413-f444-43bb-9103-7bba5c5a2343","Type":"ContainerStarted","Data":"af71dce3e473fb253df813eb995897bb8be1988bba60189b4c51ba23fb6b9734"} Oct 30 05:24:56.870048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.870010 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54"] Oct 30 05:24:56.874046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.874025 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" Oct 30 05:24:56.888977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.888950 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e56c03f-xss54\" (UID: \"0be2e889-114d-4222-b92e-ed319dab5220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" Oct 30 05:24:56.889144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.888982 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e56c03f-xss54\" (UID: \"0be2e889-114d-4222-b92e-ed319dab5220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" Oct 30 05:24:56.889271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.889239 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9znp9\" (UniqueName: \"kubernetes.io/projected/0be2e889-114d-4222-b92e-ed319dab5220-kube-api-access-9znp9\") pod \"maas-api-revoke-sub-4e56c03f-xss54\" (UID: \"0be2e889-114d-4222-b92e-ed319dab5220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" Oct 30 05:24:56.892587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.892537 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" podStartSLOduration=1.892523657 podStartE2EDuration="1.892523657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:24:56.889797339 +0000 UTC m=+2421.417564126" watchObservedRunningTime="2026-10-30 05:24:56.892523657 +0000 UTC m=+2421.420290452" Oct 30 05:24:56.900910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.900886 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54"] Oct 30 05:24:56.991808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.991770 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e56c03f-xss54\" (UID: \"0be2e889-114d-4222-b92e-ed319dab5220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" Oct 30 05:24:56.991808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.991819 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e56c03f-xss54\" (UID: \"0be2e889-114d-4222-b92e-ed319dab5220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" Oct 30 05:24:56.992040 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.992001 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9znp9\" (UniqueName: \"kubernetes.io/projected/0be2e889-114d-4222-b92e-ed319dab5220-kube-api-access-9znp9\") pod \"maas-api-revoke-sub-4e56c03f-xss54\" (UID: \"0be2e889-114d-4222-b92e-ed319dab5220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" Oct 30 05:24:56.992476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.992451 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e56c03f-xss54\" (UID: \"0be2e889-114d-4222-b92e-ed319dab5220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" Oct 30 05:24:56.994499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:56.994469 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e56c03f-xss54\" (UID: \"0be2e889-114d-4222-b92e-ed319dab5220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" Oct 30 05:24:57.116457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:57.116374 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9znp9\" (UniqueName: \"kubernetes.io/projected/0be2e889-114d-4222-b92e-ed319dab5220-kube-api-access-9znp9\") pod \"maas-api-revoke-sub-4e56c03f-xss54\" (UID: \"0be2e889-114d-4222-b92e-ed319dab5220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" Oct 30 05:24:57.189442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:57.189410 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" Oct 30 05:24:57.216275 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:57.216242 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:57.224727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:57.224338 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:57.495520 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:24:57.495485 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be2e889_114d_4222_b92e_ed319dab5220.slice/crio-a89d92387f97c0e99fec87153a45a0f4c9878fac286f30a3bf514cde9588b19c WatchSource:0}: Error finding container a89d92387f97c0e99fec87153a45a0f4c9878fac286f30a3bf514cde9588b19c: Status 404 returned error can't find the container with id a89d92387f97c0e99fec87153a45a0f4c9878fac286f30a3bf514cde9588b19c Oct 30 05:24:57.497298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:57.497268 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54"] Oct 30 05:24:57.584609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:57.584577 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeea73b7-9493-4e07-9ede-23123f96f897" path="/var/lib/kubelet/pods/aeea73b7-9493-4e07-9ede-23123f96f897/volumes" Oct 30 05:24:57.585084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:57.585070 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b10ce723-0350-4b2f-8180-32bea097068c" path="/var/lib/kubelet/pods/b10ce723-0350-4b2f-8180-32bea097068c/volumes" Oct 30 05:24:57.862304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:57.862257 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" event={"ID":"0be2e889-114d-4222-b92e-ed319dab5220","Type":"ContainerStarted","Data":"a89d92387f97c0e99fec87153a45a0f4c9878fac286f30a3bf514cde9588b19c"} Oct 30 05:24:57.864889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:57.864848 6587 generic.go:356] "Generic (PLEG): container finished" podID="ae76c413-f444-43bb-9103-7bba5c5a2343" containerID="9f5dbb228bd11ae6d8c71b7869b073e3b95b0dfc15bffbda75757740812eb6c8" exitCode=0 Oct 30 05:24:57.865033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:57.864929 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" event={"ID":"ae76c413-f444-43bb-9103-7bba5c5a2343","Type":"ContainerDied","Data":"9f5dbb228bd11ae6d8c71b7869b073e3b95b0dfc15bffbda75757740812eb6c8"} Oct 30 05:24:57.865179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:57.865157 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:57.866890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:57.866870 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:24:58.874952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:58.874916 6587 generic.go:356] "Generic (PLEG): container finished" podID="0be2e889-114d-4222-b92e-ed319dab5220" containerID="5e75bab47e1e84df7e285036b1f5be590a81bffdcc178130434980be468b7d62" exitCode=0 Oct 30 05:24:58.875415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:58.875001 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" event={"ID":"0be2e889-114d-4222-b92e-ed319dab5220","Type":"ContainerDied","Data":"5e75bab47e1e84df7e285036b1f5be590a81bffdcc178130434980be468b7d62"} Oct 30 05:24:59.044014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.043981 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" Oct 30 05:24:59.137607 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.137565 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-xks-ca\") pod \"ae76c413-f444-43bb-9103-7bba5c5a2343\" (UID: \"ae76c413-f444-43bb-9103-7bba5c5a2343\") " Oct 30 05:24:59.137867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.137844 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-openshift-ca\") pod \"ae76c413-f444-43bb-9103-7bba5c5a2343\" (UID: \"ae76c413-f444-43bb-9103-7bba5c5a2343\") " Oct 30 05:24:59.137938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.137911 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae76c413-f444-43bb-9103-7bba5c5a2343-kube-api-access-lq7g6\" (UniqueName: \"kubernetes.io/projected/ae76c413-f444-43bb-9103-7bba5c5a2343-kube-api-access-lq7g6\") pod \"ae76c413-f444-43bb-9103-7bba5c5a2343\" (UID: \"ae76c413-f444-43bb-9103-7bba5c5a2343\") " Oct 30 05:24:59.139645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.139611 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-openshift-ca" pod "ae76c413-f444-43bb-9103-7bba5c5a2343" (UID: "ae76c413-f444-43bb-9103-7bba5c5a2343"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:24:59.142865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.141549 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-xks-ca" pod "ae76c413-f444-43bb-9103-7bba5c5a2343" (UID: "ae76c413-f444-43bb-9103-7bba5c5a2343"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:24:59.142865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.141655 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae76c413-f444-43bb-9103-7bba5c5a2343-kube-api-access-lq7g6" pod "ae76c413-f444-43bb-9103-7bba5c5a2343" (UID: "ae76c413-f444-43bb-9103-7bba5c5a2343"). InnerVolumeSpecName "kube-api-access-lq7g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:24:59.240889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.240811 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:59.240889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.240839 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq7g6\" (UniqueName: \"kubernetes.io/projected/ae76c413-f444-43bb-9103-7bba5c5a2343-kube-api-access-lq7g6\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:59.240889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.240850 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae76c413-f444-43bb-9103-7bba5c5a2343-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:24:59.892682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.892645 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds"] Oct 30 05:24:59.894673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.894637 6587 cpu_manager.go:414] "containerMap: removing container" podUID="ae76c413-f444-43bb-9103-7bba5c5a2343" containerName="revoke-keys" Oct 30 05:24:59.894855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.894841 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae76c413-f444-43bb-9103-7bba5c5a2343" containerName="revoke-keys" Oct 30 05:24:59.928394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:24:59.925799 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae76c413-f444-43bb-9103-7bba5c5a2343" containerName="revoke-keys" Oct 30 05:25:00.021742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.018219 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" Oct 30 05:25:00.021742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.019509 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds"] Oct 30 05:25:00.038545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.031066 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" Oct 30 05:25:00.038545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.031129 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc" event={"ID":"ae76c413-f444-43bb-9103-7bba5c5a2343","Type":"ContainerDied","Data":"af71dce3e473fb253df813eb995897bb8be1988bba60189b4c51ba23fb6b9734"} Oct 30 05:25:00.038545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.031171 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af71dce3e473fb253df813eb995897bb8be1988bba60189b4c51ba23fb6b9734" Oct 30 05:25:00.046730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.042306 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628vf\" (UniqueName: \"kubernetes.io/projected/66c8e0c3-624f-44f9-b797-b7837a2deb53-kube-api-access-628vf\") pod \"payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds\" (UID: \"66c8e0c3-624f-44f9-b797-b7837a2deb53\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" Oct 30 05:25:00.046730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.042400 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c8e0c3-624f-44f9-b797-b7837a2deb53-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds\" (UID: \"66c8e0c3-624f-44f9-b797-b7837a2deb53\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" Oct 30 05:25:00.118878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.118849 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6b3855\"" Oct 30 05:25:00.119918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.119855 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6b3855-dockercfg-7nq7g\"" Oct 30 05:25:00.145295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.144106 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628vf\" (UniqueName: \"kubernetes.io/projected/66c8e0c3-624f-44f9-b797-b7837a2deb53-kube-api-access-628vf\") pod \"payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds\" (UID: \"66c8e0c3-624f-44f9-b797-b7837a2deb53\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" Oct 30 05:25:00.145295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.144160 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c8e0c3-624f-44f9-b797-b7837a2deb53-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds\" (UID: \"66c8e0c3-624f-44f9-b797-b7837a2deb53\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" Oct 30 05:25:00.145295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.144941 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c8e0c3-624f-44f9-b797-b7837a2deb53-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds\" (UID: \"66c8e0c3-624f-44f9-b797-b7837a2deb53\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" Oct 30 05:25:00.169787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.169752 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-628vf\" (UniqueName: \"kubernetes.io/projected/66c8e0c3-624f-44f9-b797-b7837a2deb53-kube-api-access-628vf\") pod \"payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds\" (UID: \"66c8e0c3-624f-44f9-b797-b7837a2deb53\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" Oct 30 05:25:00.225192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.225165 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" Oct 30 05:25:00.245371 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.245335 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-xks-ca\") pod \"0be2e889-114d-4222-b92e-ed319dab5220\" (UID: \"0be2e889-114d-4222-b92e-ed319dab5220\") " Oct 30 05:25:00.245540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.245522 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-openshift-ca\") pod \"0be2e889-114d-4222-b92e-ed319dab5220\" (UID: \"0be2e889-114d-4222-b92e-ed319dab5220\") " Oct 30 05:25:00.245621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.245567 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be2e889-114d-4222-b92e-ed319dab5220-kube-api-access-9znp9\" (UniqueName: \"kubernetes.io/projected/0be2e889-114d-4222-b92e-ed319dab5220-kube-api-access-9znp9\") pod \"0be2e889-114d-4222-b92e-ed319dab5220\" (UID: \"0be2e889-114d-4222-b92e-ed319dab5220\") " Oct 30 05:25:00.246234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.246204 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-openshift-ca" pod "0be2e889-114d-4222-b92e-ed319dab5220" (UID: "0be2e889-114d-4222-b92e-ed319dab5220"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:25:00.246343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.246321 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:00.247660 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.247633 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-xks-ca" pod "0be2e889-114d-4222-b92e-ed319dab5220" (UID: "0be2e889-114d-4222-b92e-ed319dab5220"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:25:00.248273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.248247 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be2e889-114d-4222-b92e-ed319dab5220-kube-api-access-9znp9" pod "0be2e889-114d-4222-b92e-ed319dab5220" (UID: "0be2e889-114d-4222-b92e-ed319dab5220"). InnerVolumeSpecName "kube-api-access-9znp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:00.348041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.348009 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be2e889-114d-4222-b92e-ed319dab5220-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:00.348041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.348038 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9znp9\" (UniqueName: \"kubernetes.io/projected/0be2e889-114d-4222-b92e-ed319dab5220-kube-api-access-9znp9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:00.354395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.354342 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" Oct 30 05:25:00.661548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.661482 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds"] Oct 30 05:25:00.662557 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:25:00.662525 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c8e0c3_624f_44f9_b797_b7837a2deb53.slice/crio-b834dac7c259d81ea8b1f5097c5baeef3e1930b26c727b01fcdc5eea5c4a398b WatchSource:0}: Error finding container b834dac7c259d81ea8b1f5097c5baeef3e1930b26c727b01fcdc5eea5c4a398b: Status 404 returned error can't find the container with id b834dac7c259d81ea8b1f5097c5baeef3e1930b26c727b01fcdc5eea5c4a398b Oct 30 05:25:00.664174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:00.664152 6587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:25:01.043676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:01.043577 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" event={"ID":"66c8e0c3-624f-44f9-b797-b7837a2deb53","Type":"ContainerStarted","Data":"b834dac7c259d81ea8b1f5097c5baeef3e1930b26c727b01fcdc5eea5c4a398b"} Oct 30 05:25:01.046040 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:01.046015 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" Oct 30 05:25:01.046180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:01.046034 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54" event={"ID":"0be2e889-114d-4222-b92e-ed319dab5220","Type":"ContainerDied","Data":"a89d92387f97c0e99fec87153a45a0f4c9878fac286f30a3bf514cde9588b19c"} Oct 30 05:25:01.046180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:01.046078 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a89d92387f97c0e99fec87153a45a0f4c9878fac286f30a3bf514cde9588b19c" Oct 30 05:25:01.048912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:01.048889 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck"] Oct 30 05:25:01.071735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:01.071709 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-892dc917-f86ck"] Oct 30 05:25:01.596002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:01.595946 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18fa2da-c745-43bf-a9cb-a2c8ab8641aa" path="/var/lib/kubelet/pods/a18fa2da-c745-43bf-a9cb-a2c8ab8641aa/volumes" Oct 30 05:25:03.085122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:03.085034 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" event={"ID":"66c8e0c3-624f-44f9-b797-b7837a2deb53","Type":"ContainerStarted","Data":"3f515f4285dbcf1db1d53b8430d28d0549c0d4ab0169a5dd9d5bc1811b31f957"} Oct 30 05:25:03.085554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:03.085258 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" Oct 30 05:25:03.176483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:03.176416 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" podStartSLOduration=2.033453195 podStartE2EDuration="4.17639719s" totalImagesPullingTime="2.142943995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:03.169602497 +0000 UTC m=+2427.697369284" watchObservedRunningTime="2026-10-30 05:25:03.17639719 +0000 UTC m=+2427.704163981" Oct 30 05:25:03.568912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:03.568879 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:25:03.584755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:03.584731 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:25:04.105071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:04.105038 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:25:06.339161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:06.339126 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6"] Oct 30 05:25:06.374268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:06.374227 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb"] Oct 30 05:25:07.283881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.283848 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g"] Oct 30 05:25:07.284737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.284715 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0be2e889-114d-4222-b92e-ed319dab5220" containerName="revoke-keys" Oct 30 05:25:07.284737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.284736 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be2e889-114d-4222-b92e-ed319dab5220" containerName="revoke-keys" Oct 30 05:25:07.284972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.284957 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be2e889-114d-4222-b92e-ed319dab5220" containerName="revoke-keys" Oct 30 05:25:07.287854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.287833 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" Oct 30 05:25:07.306425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.306400 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g"] Oct 30 05:25:07.318408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.315464 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:25:07.318408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.315531 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:25:07.318408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.315560 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:25:07.318408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.317500 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:25:07.437728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.437693 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk7dw\" (UniqueName: \"kubernetes.io/projected/58097ffd-8c5b-4a63-a0b7-7cfe31728523-kube-api-access-nk7dw\") pod \"maas-api-revoke-sub-7bcbe01d-f7d2g\" (UID: \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" Oct 30 05:25:07.438234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.437761 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bcbe01d-f7d2g\" (UID: \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" Oct 30 05:25:07.438470 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.438453 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bcbe01d-f7d2g\" (UID: \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" Oct 30 05:25:07.540422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.540287 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bcbe01d-f7d2g\" (UID: \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" Oct 30 05:25:07.540591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.540433 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk7dw\" (UniqueName: \"kubernetes.io/projected/58097ffd-8c5b-4a63-a0b7-7cfe31728523-kube-api-access-nk7dw\") pod \"maas-api-revoke-sub-7bcbe01d-f7d2g\" (UID: \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" Oct 30 05:25:07.540591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.540493 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bcbe01d-f7d2g\" (UID: \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" Oct 30 05:25:07.541216 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.541191 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bcbe01d-f7d2g\" (UID: \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" Oct 30 05:25:07.543005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.542981 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bcbe01d-f7d2g\" (UID: \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" Oct 30 05:25:07.564924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.564902 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk7dw\" (UniqueName: \"kubernetes.io/projected/58097ffd-8c5b-4a63-a0b7-7cfe31728523-kube-api-access-nk7dw\") pod \"maas-api-revoke-sub-7bcbe01d-f7d2g\" (UID: \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" Oct 30 05:25:07.603454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.603428 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" Oct 30 05:25:07.908910 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:25:07.908879 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58097ffd_8c5b_4a63_a0b7_7cfe31728523.slice/crio-e781c5c3125f73f5dd02fcc7840379344f84e22d9eeb69ed622cc63bbdab59bf WatchSource:0}: Error finding container e781c5c3125f73f5dd02fcc7840379344f84e22d9eeb69ed622cc63bbdab59bf: Status 404 returned error can't find the container with id e781c5c3125f73f5dd02fcc7840379344f84e22d9eeb69ed622cc63bbdab59bf Oct 30 05:25:07.910765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:07.910739 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g"] Oct 30 05:25:08.146209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:08.146169 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" event={"ID":"58097ffd-8c5b-4a63-a0b7-7cfe31728523","Type":"ContainerStarted","Data":"e781c5c3125f73f5dd02fcc7840379344f84e22d9eeb69ed622cc63bbdab59bf"} Oct 30 05:25:09.160725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.160687 6587 generic.go:356] "Generic (PLEG): container finished" podID="58097ffd-8c5b-4a63-a0b7-7cfe31728523" containerID="a5be515343348f7ffbacc91604f5b6c9b6281562135e7615d2755663eec275c7" exitCode=0 Oct 30 05:25:09.161191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.160819 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" event={"ID":"58097ffd-8c5b-4a63-a0b7-7cfe31728523","Type":"ContainerDied","Data":"a5be515343348f7ffbacc91604f5b6c9b6281562135e7615d2755663eec275c7"} Oct 30 05:25:09.408317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.408279 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" podUID="04dcecfd-1ed5-4221-a527-3aeb9adb49f3" containerName="main" containerID="cri-o://b1b039a76f4fc0c989754869e9e605e311360b6f42cde6d4c6d881777c971d15" gracePeriod=45 Oct 30 05:25:09.746849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.746818 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:25:09.895005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.894972 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-model-cache\") pod \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " Oct 30 05:25:09.895206 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.895011 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-home\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-home\") pod \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " Oct 30 05:25:09.895270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.895252 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-dshm\") pod \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " Oct 30 05:25:09.895422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.895335 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-model-cache" pod "04dcecfd-1ed5-4221-a527-3aeb9adb49f3" (UID: "04dcecfd-1ed5-4221-a527-3aeb9adb49f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:25:09.895422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.895413 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tls-certs\") pod \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " Oct 30 05:25:09.895562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.895419 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-home" pod "04dcecfd-1ed5-4221-a527-3aeb9adb49f3" (UID: "04dcecfd-1ed5-4221-a527-3aeb9adb49f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:25:09.895562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.895469 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tmp-dir\") pod \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " Oct 30 05:25:09.895562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.895546 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-kube-api-access-bvnkr\" (UniqueName: \"kubernetes.io/projected/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-kube-api-access-bvnkr\") pod \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\" (UID: \"04dcecfd-1ed5-4221-a527-3aeb9adb49f3\") " Oct 30 05:25:09.895786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.895761 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tmp-dir" pod "04dcecfd-1ed5-4221-a527-3aeb9adb49f3" (UID: "04dcecfd-1ed5-4221-a527-3aeb9adb49f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:25:09.896318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.896292 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:09.896467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.896322 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:09.896467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.896337 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:09.897670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.897645 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tls-certs" pod "04dcecfd-1ed5-4221-a527-3aeb9adb49f3" (UID: "04dcecfd-1ed5-4221-a527-3aeb9adb49f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:25:09.897759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.897678 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-dshm" pod "04dcecfd-1ed5-4221-a527-3aeb9adb49f3" (UID: "04dcecfd-1ed5-4221-a527-3aeb9adb49f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:25:09.897844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.897827 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-kube-api-access-bvnkr" pod "04dcecfd-1ed5-4221-a527-3aeb9adb49f3" (UID: "04dcecfd-1ed5-4221-a527-3aeb9adb49f3"). InnerVolumeSpecName "kube-api-access-bvnkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:09.997633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.997529 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:09.997633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.997568 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvnkr\" (UniqueName: \"kubernetes.io/projected/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-kube-api-access-bvnkr\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:09.997633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:09.997583 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04dcecfd-1ed5-4221-a527-3aeb9adb49f3-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:10.177584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.177552 6587 generic.go:356] "Generic (PLEG): container finished" podID="04dcecfd-1ed5-4221-a527-3aeb9adb49f3" containerID="b1b039a76f4fc0c989754869e9e605e311360b6f42cde6d4c6d881777c971d15" exitCode=0 Oct 30 05:25:10.178053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.177595 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" Oct 30 05:25:10.178053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.177632 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" event={"ID":"04dcecfd-1ed5-4221-a527-3aeb9adb49f3","Type":"ContainerDied","Data":"b1b039a76f4fc0c989754869e9e605e311360b6f42cde6d4c6d881777c971d15"} Oct 30 05:25:10.178053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.177679 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t" event={"ID":"04dcecfd-1ed5-4221-a527-3aeb9adb49f3","Type":"ContainerDied","Data":"feb5268a3d6c2d7d9affc6ed00ae9c2c79e184e3ac9722350e6666abd340bc07"} Oct 30 05:25:10.178053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.177696 6587 scope.go:122] "RemoveContainer" containerID="b1b039a76f4fc0c989754869e9e605e311360b6f42cde6d4c6d881777c971d15" Oct 30 05:25:10.191620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.191598 6587 scope.go:122] "RemoveContainer" containerID="b1b039a76f4fc0c989754869e9e605e311360b6f42cde6d4c6d881777c971d15" Oct 30 05:25:10.191923 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:25:10.191903 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b039a76f4fc0c989754869e9e605e311360b6f42cde6d4c6d881777c971d15\": container with ID starting with b1b039a76f4fc0c989754869e9e605e311360b6f42cde6d4c6d881777c971d15 not found: ID does not exist" containerID="b1b039a76f4fc0c989754869e9e605e311360b6f42cde6d4c6d881777c971d15" Oct 30 05:25:10.192014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.191934 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b039a76f4fc0c989754869e9e605e311360b6f42cde6d4c6d881777c971d15"} err="failed to get container status \"b1b039a76f4fc0c989754869e9e605e311360b6f42cde6d4c6d881777c971d15\": rpc error: code = NotFound desc = could not find container \"b1b039a76f4fc0c989754869e9e605e311360b6f42cde6d4c6d881777c971d15\": container with ID starting with b1b039a76f4fc0c989754869e9e605e311360b6f42cde6d4c6d881777c971d15 not found: ID does not exist" Oct 30 05:25:10.241291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.241158 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t"] Oct 30 05:25:10.288753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.288714 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e15c99/auth-test-model-e15c99-kserve-86bdbdfb97-7zn6t"] Oct 30 05:25:10.381734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.381708 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" Oct 30 05:25:10.509196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.509119 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-openshift-ca\") pod \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\" (UID: \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\") " Oct 30 05:25:10.509351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.509308 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-xks-ca\") pod \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\" (UID: \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\") " Oct 30 05:25:10.509351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.509332 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58097ffd-8c5b-4a63-a0b7-7cfe31728523-kube-api-access-nk7dw\" (UniqueName: \"kubernetes.io/projected/58097ffd-8c5b-4a63-a0b7-7cfe31728523-kube-api-access-nk7dw\") pod \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\" (UID: \"58097ffd-8c5b-4a63-a0b7-7cfe31728523\") " Oct 30 05:25:10.509560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.509533 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-openshift-ca" pod "58097ffd-8c5b-4a63-a0b7-7cfe31728523" (UID: "58097ffd-8c5b-4a63-a0b7-7cfe31728523"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:25:10.509918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.509898 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:10.511487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.511461 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-xks-ca" pod "58097ffd-8c5b-4a63-a0b7-7cfe31728523" (UID: "58097ffd-8c5b-4a63-a0b7-7cfe31728523"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:25:10.511656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.511495 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58097ffd-8c5b-4a63-a0b7-7cfe31728523-kube-api-access-nk7dw" pod "58097ffd-8c5b-4a63-a0b7-7cfe31728523" (UID: "58097ffd-8c5b-4a63-a0b7-7cfe31728523"). InnerVolumeSpecName "kube-api-access-nk7dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:10.611940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.611910 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58097ffd-8c5b-4a63-a0b7-7cfe31728523-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:10.611940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:10.611940 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk7dw\" (UniqueName: \"kubernetes.io/projected/58097ffd-8c5b-4a63-a0b7-7cfe31728523-kube-api-access-nk7dw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:11.189828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:11.188849 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" event={"ID":"58097ffd-8c5b-4a63-a0b7-7cfe31728523","Type":"ContainerDied","Data":"e781c5c3125f73f5dd02fcc7840379344f84e22d9eeb69ed622cc63bbdab59bf"} Oct 30 05:25:11.189828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:11.188916 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e781c5c3125f73f5dd02fcc7840379344f84e22d9eeb69ed622cc63bbdab59bf" Oct 30 05:25:11.190286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:11.190041 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g" Oct 30 05:25:11.340459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:11.340415 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" podUID="d453e4b7-43e6-4463-869e-20060976c7ca" containerName="payload-processing" containerID="cri-o://e63ef0f1bfc5f3df6be8b80a5cdefd671276f0dd2b8efd9629c3709c0fff31f5" gracePeriod=25 Oct 30 05:25:11.375205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:11.375151 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" podUID="659aff68-3d79-4f16-b239-23197429890c" containerName="payload-pre-processing" containerID="cri-o://6e12c3df4a138f8c16e2a4b3de3e40fceb674419e40bbc156364b3cbe4ec970f" gracePeriod=25 Oct 30 05:25:11.591461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:11.591428 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04dcecfd-1ed5-4221-a527-3aeb9adb49f3" path="/var/lib/kubelet/pods/04dcecfd-1ed5-4221-a527-3aeb9adb49f3/volumes" Oct 30 05:25:11.641926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:11.641893 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" Oct 30 05:25:11.728802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:11.728716 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d453e4b7-43e6-4463-869e-20060976c7ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d453e4b7-43e6-4463-869e-20060976c7ca-plugins-config-volume\") pod \"d453e4b7-43e6-4463-869e-20060976c7ca\" (UID: \"d453e4b7-43e6-4463-869e-20060976c7ca\") " Oct 30 05:25:11.728802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:11.728776 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d453e4b7-43e6-4463-869e-20060976c7ca-kube-api-access-kwxsj\" (UniqueName: \"kubernetes.io/projected/d453e4b7-43e6-4463-869e-20060976c7ca-kube-api-access-kwxsj\") pod \"d453e4b7-43e6-4463-869e-20060976c7ca\" (UID: \"d453e4b7-43e6-4463-869e-20060976c7ca\") " Oct 30 05:25:11.730390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:11.729262 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d453e4b7-43e6-4463-869e-20060976c7ca-plugins-config-volume" pod "d453e4b7-43e6-4463-869e-20060976c7ca" (UID: "d453e4b7-43e6-4463-869e-20060976c7ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:25:11.730390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:11.730101 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d453e4b7-43e6-4463-869e-20060976c7ca-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:11.733386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:11.732415 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d453e4b7-43e6-4463-869e-20060976c7ca-kube-api-access-kwxsj" pod "d453e4b7-43e6-4463-869e-20060976c7ca" (UID: "d453e4b7-43e6-4463-869e-20060976c7ca"). InnerVolumeSpecName "kube-api-access-kwxsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:11.833853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:11.832336 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwxsj\" (UniqueName: \"kubernetes.io/projected/d453e4b7-43e6-4463-869e-20060976c7ca-kube-api-access-kwxsj\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:12.043802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:12.043703 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" podUID="659aff68-3d79-4f16-b239-23197429890c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:25:12.209097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:12.209064 6587 generic.go:356] "Generic (PLEG): container finished" podID="d453e4b7-43e6-4463-869e-20060976c7ca" containerID="e63ef0f1bfc5f3df6be8b80a5cdefd671276f0dd2b8efd9629c3709c0fff31f5" exitCode=0 Oct 30 05:25:12.209688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:12.209127 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" Oct 30 05:25:12.209688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:12.209147 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" event={"ID":"d453e4b7-43e6-4463-869e-20060976c7ca","Type":"ContainerDied","Data":"e63ef0f1bfc5f3df6be8b80a5cdefd671276f0dd2b8efd9629c3709c0fff31f5"} Oct 30 05:25:12.209688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:12.209240 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6" event={"ID":"d453e4b7-43e6-4463-869e-20060976c7ca","Type":"ContainerDied","Data":"cc886c43ed7948abfdf708bef6275cd8ef9aad3444aefcdd46f98888e785cb91"} Oct 30 05:25:12.209688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:12.209283 6587 scope.go:122] "RemoveContainer" containerID="e63ef0f1bfc5f3df6be8b80a5cdefd671276f0dd2b8efd9629c3709c0fff31f5" Oct 30 05:25:12.223083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:12.223061 6587 scope.go:122] "RemoveContainer" containerID="e63ef0f1bfc5f3df6be8b80a5cdefd671276f0dd2b8efd9629c3709c0fff31f5" Oct 30 05:25:12.224299 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:25:12.223347 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e63ef0f1bfc5f3df6be8b80a5cdefd671276f0dd2b8efd9629c3709c0fff31f5\": container with ID starting with e63ef0f1bfc5f3df6be8b80a5cdefd671276f0dd2b8efd9629c3709c0fff31f5 not found: ID does not exist" containerID="e63ef0f1bfc5f3df6be8b80a5cdefd671276f0dd2b8efd9629c3709c0fff31f5" Oct 30 05:25:12.224299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:12.223401 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63ef0f1bfc5f3df6be8b80a5cdefd671276f0dd2b8efd9629c3709c0fff31f5"} err="failed to get container status \"e63ef0f1bfc5f3df6be8b80a5cdefd671276f0dd2b8efd9629c3709c0fff31f5\": rpc error: code = NotFound desc = could not find container \"e63ef0f1bfc5f3df6be8b80a5cdefd671276f0dd2b8efd9629c3709c0fff31f5\": container with ID starting with e63ef0f1bfc5f3df6be8b80a5cdefd671276f0dd2b8efd9629c3709c0fff31f5 not found: ID does not exist" Oct 30 05:25:12.268655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:12.268621 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6"] Oct 30 05:25:12.281990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:12.281960 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6"] Oct 30 05:25:13.592926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:13.592889 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d453e4b7-43e6-4463-869e-20060976c7ca" path="/var/lib/kubelet/pods/d453e4b7-43e6-4463-869e-20060976c7ca/volumes" Oct 30 05:25:14.095074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:14.095046 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" Oct 30 05:25:18.226175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.226091 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6"] Oct 30 05:25:18.226860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.226843 6587 cpu_manager.go:414] "containerMap: removing container" podUID="58097ffd-8c5b-4a63-a0b7-7cfe31728523" containerName="revoke-keys" Oct 30 05:25:18.226907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.226861 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58097ffd-8c5b-4a63-a0b7-7cfe31728523" containerName="revoke-keys" Oct 30 05:25:18.226907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.226872 6587 cpu_manager.go:414] "containerMap: removing container" podUID="04dcecfd-1ed5-4221-a527-3aeb9adb49f3" containerName="main" Oct 30 05:25:18.226907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.226877 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04dcecfd-1ed5-4221-a527-3aeb9adb49f3" containerName="main" Oct 30 05:25:18.226907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.226888 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d453e4b7-43e6-4463-869e-20060976c7ca" containerName="payload-processing" Oct 30 05:25:18.226907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.226893 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d453e4b7-43e6-4463-869e-20060976c7ca" containerName="payload-processing" Oct 30 05:25:18.227076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.227024 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="58097ffd-8c5b-4a63-a0b7-7cfe31728523" containerName="revoke-keys" Oct 30 05:25:18.227076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.227031 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="04dcecfd-1ed5-4221-a527-3aeb9adb49f3" containerName="main" Oct 30 05:25:18.227076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.227037 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d453e4b7-43e6-4463-869e-20060976c7ca" containerName="payload-processing" Oct 30 05:25:18.232424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.232406 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" Oct 30 05:25:18.255224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.255195 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6"] Oct 30 05:25:18.263050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.262830 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:25:18.263050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.262902 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:25:18.263050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.262936 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:25:18.263050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.263002 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:25:18.330785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.330744 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06eabedb-rn8z6\" (UID: \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" Oct 30 05:25:18.330980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.330955 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06eabedb-rn8z6\" (UID: \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" Oct 30 05:25:18.331087 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.331068 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w265\" (UniqueName: \"kubernetes.io/projected/8da8f630-2be6-4ec7-8c48-3c69b536dffd-kube-api-access-9w265\") pod \"maas-api-revoke-sub-06eabedb-rn8z6\" (UID: \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" Oct 30 05:25:18.432688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.432649 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06eabedb-rn8z6\" (UID: \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" Oct 30 05:25:18.432892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.432746 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w265\" (UniqueName: \"kubernetes.io/projected/8da8f630-2be6-4ec7-8c48-3c69b536dffd-kube-api-access-9w265\") pod \"maas-api-revoke-sub-06eabedb-rn8z6\" (UID: \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" Oct 30 05:25:18.432892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.432841 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06eabedb-rn8z6\" (UID: \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" Oct 30 05:25:18.433599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.433575 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06eabedb-rn8z6\" (UID: \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" Oct 30 05:25:18.435649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.435628 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06eabedb-rn8z6\" (UID: \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" Oct 30 05:25:18.454212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.454189 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w265\" (UniqueName: \"kubernetes.io/projected/8da8f630-2be6-4ec7-8c48-3c69b536dffd-kube-api-access-9w265\") pod \"maas-api-revoke-sub-06eabedb-rn8z6\" (UID: \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" Oct 30 05:25:18.547853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.547781 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" Oct 30 05:25:18.723264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.723229 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm"] Oct 30 05:25:18.730205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.730179 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.760768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.756803 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-45607d-openshift-default-dockercfg-dlgvx\"" Oct 30 05:25:18.760768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.759184 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm"] Oct 30 05:25:18.857621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.856645 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f131a03e-c992-40b1-a6e6-86d29073205b-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.857621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.856760 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.857621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.856896 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rgqp\" (UniqueName: \"kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-kube-api-access-5rgqp\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.857621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.856929 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.857621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.856952 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.857621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.856972 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.857621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.857027 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f131a03e-c992-40b1-a6e6-86d29073205b-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.857621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.857075 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.857621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.857193 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.975939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.975904 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f131a03e-c992-40b1-a6e6-86d29073205b-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.976233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.976206 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.976519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.976496 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.976724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.976703 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f131a03e-c992-40b1-a6e6-86d29073205b-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.976940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.976922 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.977209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.977188 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rgqp\" (UniqueName: \"kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-kube-api-access-5rgqp\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.977376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.977344 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.977503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.977486 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.977614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.977599 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.978769 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.978748 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f131a03e-c992-40b1-a6e6-86d29073205b-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.979123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.979105 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.979487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.979467 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.980251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.979758 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6"] Oct 30 05:25:18.980428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.980042 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:18.980808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:18.980789 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:19.027249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:19.000830 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:19.027249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:19.007175 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rgqp\" (UniqueName: \"kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-kube-api-access-5rgqp\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:19.027249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:19.023179 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:19.027249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:19.025508 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f131a03e-c992-40b1-a6e6-86d29073205b-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:19.067386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:19.055708 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:19.303886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:19.303845 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" event={"ID":"8da8f630-2be6-4ec7-8c48-3c69b536dffd","Type":"ContainerStarted","Data":"55101ed03521c3b71379ed2a268cc517daec173e630f9f63cff1bc723f296684"} Oct 30 05:25:19.446294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:19.446267 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm"] Oct 30 05:25:19.518451 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:25:19.518419 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf131a03e_c992_40b1_a6e6_86d29073205b.slice/crio-797fa622e7c099b4c6fa5a61632fbfcba591f0b81054b2981f8a4af0a00ed5cc WatchSource:0}: Error finding container 797fa622e7c099b4c6fa5a61632fbfcba591f0b81054b2981f8a4af0a00ed5cc: Status 404 returned error can't find the container with id 797fa622e7c099b4c6fa5a61632fbfcba591f0b81054b2981f8a4af0a00ed5cc Oct 30 05:25:19.521937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:19.521905 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:25:19.522056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:19.521985 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:25:19.522056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:19.522015 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:25:20.316446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:20.316413 6587 generic.go:356] "Generic (PLEG): container finished" podID="8da8f630-2be6-4ec7-8c48-3c69b536dffd" containerID="cd7199aae94b4cbebdc7ad4d0086cd1414e689d6efb8e9440032ad6a31fbf9f3" exitCode=0 Oct 30 05:25:20.316919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:20.316560 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" event={"ID":"8da8f630-2be6-4ec7-8c48-3c69b536dffd","Type":"ContainerDied","Data":"cd7199aae94b4cbebdc7ad4d0086cd1414e689d6efb8e9440032ad6a31fbf9f3"} Oct 30 05:25:20.318766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:20.318740 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" event={"ID":"f131a03e-c992-40b1-a6e6-86d29073205b","Type":"ContainerStarted","Data":"6512ca1ab0e81c9d97c1839699c9f2e89cce37f7685c36c414d15c8cf626f5c1"} Oct 30 05:25:20.318917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:20.318781 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" event={"ID":"f131a03e-c992-40b1-a6e6-86d29073205b","Type":"ContainerStarted","Data":"797fa622e7c099b4c6fa5a61632fbfcba591f0b81054b2981f8a4af0a00ed5cc"} Oct 30 05:25:20.409107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:20.409006 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" podStartSLOduration=2.408991639 podStartE2EDuration="2.408991639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:20.405959427 +0000 UTC m=+2444.933726214" watchObservedRunningTime="2026-10-30 05:25:20.408991639 +0000 UTC m=+2444.936758484" Oct 30 05:25:21.056899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.056864 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:21.064315 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.064288 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:21.328534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.328439 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:21.329572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.329554 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:25:21.538948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.538917 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" Oct 30 05:25:21.549380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.549341 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da8f630-2be6-4ec7-8c48-3c69b536dffd-kube-api-access-9w265\" (UniqueName: \"kubernetes.io/projected/8da8f630-2be6-4ec7-8c48-3c69b536dffd-kube-api-access-9w265\") pod \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\" (UID: \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\") " Oct 30 05:25:21.549518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.549489 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-openshift-ca\") pod \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\" (UID: \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\") " Oct 30 05:25:21.549556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.549547 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-xks-ca\") pod \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\" (UID: \"8da8f630-2be6-4ec7-8c48-3c69b536dffd\") " Oct 30 05:25:21.549859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.549829 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-openshift-ca" pod "8da8f630-2be6-4ec7-8c48-3c69b536dffd" (UID: "8da8f630-2be6-4ec7-8c48-3c69b536dffd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:25:21.550262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.550240 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:21.551542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.551519 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da8f630-2be6-4ec7-8c48-3c69b536dffd-kube-api-access-9w265" pod "8da8f630-2be6-4ec7-8c48-3c69b536dffd" (UID: "8da8f630-2be6-4ec7-8c48-3c69b536dffd"). InnerVolumeSpecName "kube-api-access-9w265". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:21.551673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.551659 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-xks-ca" pod "8da8f630-2be6-4ec7-8c48-3c69b536dffd" (UID: "8da8f630-2be6-4ec7-8c48-3c69b536dffd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:25:21.651400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.651314 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8da8f630-2be6-4ec7-8c48-3c69b536dffd-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:21.651400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:21.651352 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w265\" (UniqueName: \"kubernetes.io/projected/8da8f630-2be6-4ec7-8c48-3c69b536dffd-kube-api-access-9w265\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:22.043646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:22.043551 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" podUID="659aff68-3d79-4f16-b239-23197429890c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:25:22.340078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:22.340045 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" Oct 30 05:25:22.340553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:22.340069 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6" event={"ID":"8da8f630-2be6-4ec7-8c48-3c69b536dffd","Type":"ContainerDied","Data":"55101ed03521c3b71379ed2a268cc517daec173e630f9f63cff1bc723f296684"} Oct 30 05:25:22.340553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:22.340132 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55101ed03521c3b71379ed2a268cc517daec173e630f9f63cff1bc723f296684" Oct 30 05:25:24.867771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:24.867731 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m"] Oct 30 05:25:24.868525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:24.868507 6587 cpu_manager.go:414] "containerMap: removing container" podUID="8da8f630-2be6-4ec7-8c48-3c69b536dffd" containerName="revoke-keys" Oct 30 05:25:24.868525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:24.868525 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da8f630-2be6-4ec7-8c48-3c69b536dffd" containerName="revoke-keys" Oct 30 05:25:24.868672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:24.868662 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da8f630-2be6-4ec7-8c48-3c69b536dffd" containerName="revoke-keys" Oct 30 05:25:24.872753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:24.872734 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" Oct 30 05:25:24.891931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:24.891901 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m"] Oct 30 05:25:24.900644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:24.900616 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-45607d\"" Oct 30 05:25:24.903708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:24.903679 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5faab8-cb81-4304-8715-f3909474e267-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m\" (UID: \"fc5faab8-cb81-4304-8715-f3909474e267\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" Oct 30 05:25:24.904197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:24.904168 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpq7h\" (UniqueName: \"kubernetes.io/projected/fc5faab8-cb81-4304-8715-f3909474e267-kube-api-access-zpq7h\") pod \"payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m\" (UID: \"fc5faab8-cb81-4304-8715-f3909474e267\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" Oct 30 05:25:24.992409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:24.978836 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-45607d-dockercfg-mhfjf\"" Oct 30 05:25:25.062689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.006797 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpq7h\" (UniqueName: \"kubernetes.io/projected/fc5faab8-cb81-4304-8715-f3909474e267-kube-api-access-zpq7h\") pod \"payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m\" (UID: \"fc5faab8-cb81-4304-8715-f3909474e267\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" Oct 30 05:25:25.162438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.161506 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5faab8-cb81-4304-8715-f3909474e267-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m\" (UID: \"fc5faab8-cb81-4304-8715-f3909474e267\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" Oct 30 05:25:25.162620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.162601 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5faab8-cb81-4304-8715-f3909474e267-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m\" (UID: \"fc5faab8-cb81-4304-8715-f3909474e267\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" Oct 30 05:25:25.187381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.134255 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpq7h\" (UniqueName: \"kubernetes.io/projected/fc5faab8-cb81-4304-8715-f3909474e267-kube-api-access-zpq7h\") pod \"payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m\" (UID: \"fc5faab8-cb81-4304-8715-f3909474e267\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" Oct 30 05:25:25.187381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.177613 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5"] Oct 30 05:25:25.218768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.218731 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" Oct 30 05:25:25.225381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.222248 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5"] Oct 30 05:25:25.225381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.223969 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" Oct 30 05:25:25.264086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.264020 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf7x9\" (UniqueName: \"kubernetes.io/projected/57233b00-5af2-4532-a38b-5870d0181606-kube-api-access-lf7x9\") pod \"payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5\" (UID: \"57233b00-5af2-4532-a38b-5870d0181606\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" Oct 30 05:25:25.264086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.264063 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57233b00-5af2-4532-a38b-5870d0181606-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5\" (UID: \"57233b00-5af2-4532-a38b-5870d0181606\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" Oct 30 05:25:25.378432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.367468 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf7x9\" (UniqueName: \"kubernetes.io/projected/57233b00-5af2-4532-a38b-5870d0181606-kube-api-access-lf7x9\") pod \"payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5\" (UID: \"57233b00-5af2-4532-a38b-5870d0181606\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" Oct 30 05:25:25.378432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.367522 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57233b00-5af2-4532-a38b-5870d0181606-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5\" (UID: \"57233b00-5af2-4532-a38b-5870d0181606\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" Oct 30 05:25:25.378432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.377921 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57233b00-5af2-4532-a38b-5870d0181606-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5\" (UID: \"57233b00-5af2-4532-a38b-5870d0181606\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" Oct 30 05:25:25.389910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.389884 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf7x9\" (UniqueName: \"kubernetes.io/projected/57233b00-5af2-4532-a38b-5870d0181606-kube-api-access-lf7x9\") pod \"payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5\" (UID: \"57233b00-5af2-4532-a38b-5870d0181606\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" Oct 30 05:25:25.545926 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:25:25.545892 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5faab8_cb81_4304_8715_f3909474e267.slice/crio-fefcbc39387a8bef4a3420c348e638ab6869c7e8950b30e012b0b48022b8497b WatchSource:0}: Error finding container fefcbc39387a8bef4a3420c348e638ab6869c7e8950b30e012b0b48022b8497b: Status 404 returned error can't find the container with id fefcbc39387a8bef4a3420c348e638ab6869c7e8950b30e012b0b48022b8497b Oct 30 05:25:25.548190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.548163 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" Oct 30 05:25:25.548702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.548678 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m"] Oct 30 05:25:25.866532 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:25:25.866489 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57233b00_5af2_4532_a38b_5870d0181606.slice/crio-1dcd4b7b06e5a6761841d9697677fd17e2f01ff3dfeb2611c3c26760061a2069 WatchSource:0}: Error finding container 1dcd4b7b06e5a6761841d9697677fd17e2f01ff3dfeb2611c3c26760061a2069: Status 404 returned error can't find the container with id 1dcd4b7b06e5a6761841d9697677fd17e2f01ff3dfeb2611c3c26760061a2069 Oct 30 05:25:25.867486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:25.867461 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5"] Oct 30 05:25:25.989734 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:25:25.989687 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:25:26.393434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:26.393393 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" event={"ID":"fc5faab8-cb81-4304-8715-f3909474e267","Type":"ContainerStarted","Data":"38d789e2b785b4ccdce5a4ee29d2d2bf06378e599709546d4c07b6173efa3fdb"} Oct 30 05:25:26.393605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:26.393459 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" event={"ID":"fc5faab8-cb81-4304-8715-f3909474e267","Type":"ContainerStarted","Data":"fefcbc39387a8bef4a3420c348e638ab6869c7e8950b30e012b0b48022b8497b"} Oct 30 05:25:26.393605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:26.393486 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" Oct 30 05:25:26.395307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:26.395275 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" event={"ID":"57233b00-5af2-4532-a38b-5870d0181606","Type":"ContainerStarted","Data":"e9e34fe6504a2ded817d1e37112040b4e5e00128949ce6ab67498972f0b4f5e3"} Oct 30 05:25:26.395446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:26.395326 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" event={"ID":"57233b00-5af2-4532-a38b-5870d0181606","Type":"ContainerStarted","Data":"1dcd4b7b06e5a6761841d9697677fd17e2f01ff3dfeb2611c3c26760061a2069"} Oct 30 05:25:26.395446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:26.395345 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" Oct 30 05:25:26.436849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:26.436801 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" podStartSLOduration=2.436783783 podStartE2EDuration="2.436783783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:26.435085917 +0000 UTC m=+2450.962852702" watchObservedRunningTime="2026-10-30 05:25:26.436783783 +0000 UTC m=+2450.964550569" Oct 30 05:25:26.480078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:26.480034 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" podStartSLOduration=2.480021397 podStartE2EDuration="2.480021397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:26.477227791 +0000 UTC m=+2451.004994576" watchObservedRunningTime="2026-10-30 05:25:26.480021397 +0000 UTC m=+2451.007788182" Oct 30 05:25:28.794199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:28.794157 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm"] Oct 30 05:25:28.801770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:28.801748 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" Oct 30 05:25:28.819097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:28.819064 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm"] Oct 30 05:25:28.827534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:28.827458 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:25:28.827534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:28.827496 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:25:28.827534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:28.827529 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:25:28.827832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:28.827574 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:25:28.923903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:28.923867 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l26wq\" (UniqueName: \"kubernetes.io/projected/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-kube-api-access-l26wq\") pod \"maas-api-revoke-sub-cf86982c-86zlm\" (UID: \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" Oct 30 05:25:28.924116 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:28.924096 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf86982c-86zlm\" (UID: \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" Oct 30 05:25:28.924258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:28.924241 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf86982c-86zlm\" (UID: \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" Oct 30 05:25:29.026173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:29.026119 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l26wq\" (UniqueName: \"kubernetes.io/projected/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-kube-api-access-l26wq\") pod \"maas-api-revoke-sub-cf86982c-86zlm\" (UID: \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" Oct 30 05:25:29.026375 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:29.026349 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf86982c-86zlm\" (UID: \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" Oct 30 05:25:29.026542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:29.026526 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf86982c-86zlm\" (UID: \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" Oct 30 05:25:29.027290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:29.027264 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf86982c-86zlm\" (UID: \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" Oct 30 05:25:29.028790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:29.028762 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf86982c-86zlm\" (UID: \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" Oct 30 05:25:29.068387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:29.068327 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l26wq\" (UniqueName: \"kubernetes.io/projected/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-kube-api-access-l26wq\") pod \"maas-api-revoke-sub-cf86982c-86zlm\" (UID: \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" Oct 30 05:25:29.124434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:29.124400 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" Oct 30 05:25:29.445472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:29.445443 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm"] Oct 30 05:25:29.449730 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:25:29.449700 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18880bdf_d4f4_4510_9f9e_4ca7dbb47077.slice/crio-61121a776410155532152e881f97f43ad9c45ae9ec71c3c159e567eb052ad809 WatchSource:0}: Error finding container 61121a776410155532152e881f97f43ad9c45ae9ec71c3c159e567eb052ad809: Status 404 returned error can't find the container with id 61121a776410155532152e881f97f43ad9c45ae9ec71c3c159e567eb052ad809 Oct 30 05:25:30.439611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:30.439556 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" event={"ID":"18880bdf-d4f4-4510-9f9e-4ca7dbb47077","Type":"ContainerStarted","Data":"8c8c2e17b64efedf51abc3d5d80b555e75f276a7c319742b791dfa8e0ed3a203"} Oct 30 05:25:30.440079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:30.439627 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" event={"ID":"18880bdf-d4f4-4510-9f9e-4ca7dbb47077","Type":"ContainerStarted","Data":"61121a776410155532152e881f97f43ad9c45ae9ec71c3c159e567eb052ad809"} Oct 30 05:25:30.483095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:30.483039 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" podStartSLOduration=2.4830252809999998 podStartE2EDuration="2.483025281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:30.481839841 +0000 UTC m=+2455.009606631" watchObservedRunningTime="2026-10-30 05:25:30.483025281 +0000 UTC m=+2455.010792067" Oct 30 05:25:31.452483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:31.452444 6587 generic.go:356] "Generic (PLEG): container finished" podID="18880bdf-d4f4-4510-9f9e-4ca7dbb47077" containerID="8c8c2e17b64efedf51abc3d5d80b555e75f276a7c319742b791dfa8e0ed3a203" exitCode=0 Oct 30 05:25:31.452904 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:31.452535 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" event={"ID":"18880bdf-d4f4-4510-9f9e-4ca7dbb47077","Type":"ContainerDied","Data":"8c8c2e17b64efedf51abc3d5d80b555e75f276a7c319742b791dfa8e0ed3a203"} Oct 30 05:25:31.715679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:31.715647 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" Oct 30 05:25:31.896688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:31.896651 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659aff68-3d79-4f16-b239-23197429890c-kube-api-access-c9j8x\" (UniqueName: \"kubernetes.io/projected/659aff68-3d79-4f16-b239-23197429890c-kube-api-access-c9j8x\") pod \"659aff68-3d79-4f16-b239-23197429890c\" (UID: \"659aff68-3d79-4f16-b239-23197429890c\") " Oct 30 05:25:31.896861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:31.896726 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659aff68-3d79-4f16-b239-23197429890c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659aff68-3d79-4f16-b239-23197429890c-plugins-config-volume\") pod \"659aff68-3d79-4f16-b239-23197429890c\" (UID: \"659aff68-3d79-4f16-b239-23197429890c\") " Oct 30 05:25:31.897112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:31.897087 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659aff68-3d79-4f16-b239-23197429890c-plugins-config-volume" pod "659aff68-3d79-4f16-b239-23197429890c" (UID: "659aff68-3d79-4f16-b239-23197429890c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:25:31.897546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:31.897521 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659aff68-3d79-4f16-b239-23197429890c-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:31.898962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:31.898934 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659aff68-3d79-4f16-b239-23197429890c-kube-api-access-c9j8x" pod "659aff68-3d79-4f16-b239-23197429890c" (UID: "659aff68-3d79-4f16-b239-23197429890c"). InnerVolumeSpecName "kube-api-access-c9j8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:31.999643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:31.999557 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9j8x\" (UniqueName: \"kubernetes.io/projected/659aff68-3d79-4f16-b239-23197429890c-kube-api-access-c9j8x\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:32.196647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.196614 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw"] Oct 30 05:25:32.197472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.197455 6587 cpu_manager.go:414] "containerMap: removing container" podUID="659aff68-3d79-4f16-b239-23197429890c" containerName="payload-pre-processing" Oct 30 05:25:32.197544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.197472 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659aff68-3d79-4f16-b239-23197429890c" containerName="payload-pre-processing" Oct 30 05:25:32.197609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.197599 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="659aff68-3d79-4f16-b239-23197429890c" containerName="payload-pre-processing" Oct 30 05:25:32.201602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.201564 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.220793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.220746 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-674082\"/\"openshift-service-ca.crt\"" Oct 30 05:25:32.229747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.229710 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw"] Oct 30 05:25:32.234705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.234678 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-674082\"/\"kube-root-ca.crt\"" Oct 30 05:25:32.234861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.234783 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-674082\"/\"default-dockercfg-rrhv2\"" Oct 30 05:25:32.235017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.234995 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-674082\"/\"auth-test-model-674082-kserve-self-signed-certs\"" Oct 30 05:25:32.304525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.304492 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brlr\" (UniqueName: \"kubernetes.io/projected/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-kube-api-access-9brlr\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.304753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.304537 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-dshm\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.304753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.304591 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tls-certs\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.304898 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.304848 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-home\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.305048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.305027 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-model-cache\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.305154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.305069 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tmp-dir\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.407073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.407032 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tls-certs\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.407279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.407173 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-home\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.407407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.407294 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-model-cache\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.407407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.407323 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tmp-dir\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.407543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.407522 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brlr\" (UniqueName: \"kubernetes.io/projected/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-kube-api-access-9brlr\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.407602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.407569 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-dshm\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.407655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.407633 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-home\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.407715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.407699 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-model-cache\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.407784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.407759 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tmp-dir\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.409746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.409723 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-dshm\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.409846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.409821 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tls-certs\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.428968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.428945 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brlr\" (UniqueName: \"kubernetes.io/projected/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-kube-api-access-9brlr\") pod \"auth-test-model-674082-kserve-5d5d58d7c-96wmw\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.507273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.498948 6587 generic.go:356] "Generic (PLEG): container finished" podID="659aff68-3d79-4f16-b239-23197429890c" containerID="6e12c3df4a138f8c16e2a4b3de3e40fceb674419e40bbc156364b3cbe4ec970f" exitCode=0 Oct 30 05:25:32.507273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.499472 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" Oct 30 05:25:32.514547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.512411 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" event={"ID":"659aff68-3d79-4f16-b239-23197429890c","Type":"ContainerDied","Data":"6e12c3df4a138f8c16e2a4b3de3e40fceb674419e40bbc156364b3cbe4ec970f"} Oct 30 05:25:32.514547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.512483 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb" event={"ID":"659aff68-3d79-4f16-b239-23197429890c","Type":"ContainerDied","Data":"2fbbe8e7a9fb8648a70ee32f3cb5c66c1691984947a776f887554493e95167a0"} Oct 30 05:25:32.514547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.512518 6587 scope.go:122] "RemoveContainer" containerID="6e12c3df4a138f8c16e2a4b3de3e40fceb674419e40bbc156364b3cbe4ec970f" Oct 30 05:25:32.536774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.536508 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:32.649840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.649805 6587 scope.go:122] "RemoveContainer" containerID="6e12c3df4a138f8c16e2a4b3de3e40fceb674419e40bbc156364b3cbe4ec970f" Oct 30 05:25:32.652823 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:25:32.650245 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e12c3df4a138f8c16e2a4b3de3e40fceb674419e40bbc156364b3cbe4ec970f\": container with ID starting with 6e12c3df4a138f8c16e2a4b3de3e40fceb674419e40bbc156364b3cbe4ec970f not found: ID does not exist" containerID="6e12c3df4a138f8c16e2a4b3de3e40fceb674419e40bbc156364b3cbe4ec970f" Oct 30 05:25:32.652823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.650288 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e12c3df4a138f8c16e2a4b3de3e40fceb674419e40bbc156364b3cbe4ec970f"} err="failed to get container status \"6e12c3df4a138f8c16e2a4b3de3e40fceb674419e40bbc156364b3cbe4ec970f\": rpc error: code = NotFound desc = could not find container \"6e12c3df4a138f8c16e2a4b3de3e40fceb674419e40bbc156364b3cbe4ec970f\": container with ID starting with 6e12c3df4a138f8c16e2a4b3de3e40fceb674419e40bbc156364b3cbe4ec970f not found: ID does not exist" Oct 30 05:25:32.660344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.660314 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb"] Oct 30 05:25:32.672402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.670510 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb"] Oct 30 05:25:32.835538 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.835496 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" Oct 30 05:25:32.930903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.929937 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-xks-ca\") pod \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\" (UID: \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\") " Oct 30 05:25:32.930903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.930024 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-openshift-ca\") pod \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\" (UID: \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\") " Oct 30 05:25:32.930903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.930262 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-kube-api-access-l26wq\" (UniqueName: \"kubernetes.io/projected/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-kube-api-access-l26wq\") pod \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\" (UID: \"18880bdf-d4f4-4510-9f9e-4ca7dbb47077\") " Oct 30 05:25:32.932075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.932047 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-openshift-ca" pod "18880bdf-d4f4-4510-9f9e-4ca7dbb47077" (UID: "18880bdf-d4f4-4510-9f9e-4ca7dbb47077"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:25:32.937749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.937714 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-xks-ca" pod "18880bdf-d4f4-4510-9f9e-4ca7dbb47077" (UID: "18880bdf-d4f4-4510-9f9e-4ca7dbb47077"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:25:32.939728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.939697 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-kube-api-access-l26wq" pod "18880bdf-d4f4-4510-9f9e-4ca7dbb47077" (UID: "18880bdf-d4f4-4510-9f9e-4ca7dbb47077"). InnerVolumeSpecName "kube-api-access-l26wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:32.995533 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:25:32.995505 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfffc12ee_42fe_4e32_afb7_b8b35d6c8694.slice/crio-4c349c5556cdab5b3bc363d304ce5e9d8ec0495e5533f4f53417c16c1c655044 WatchSource:0}: Error finding container 4c349c5556cdab5b3bc363d304ce5e9d8ec0495e5533f4f53417c16c1c655044: Status 404 returned error can't find the container with id 4c349c5556cdab5b3bc363d304ce5e9d8ec0495e5533f4f53417c16c1c655044 Oct 30 05:25:32.998659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:32.998632 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw"] Oct 30 05:25:33.033062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:33.033032 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l26wq\" (UniqueName: \"kubernetes.io/projected/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-kube-api-access-l26wq\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:33.033062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:33.033063 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:33.033267 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:33.033072 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18880bdf-d4f4-4510-9f9e-4ca7dbb47077-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:25:33.510909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:33.510872 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" Oct 30 05:25:33.510909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:33.510890 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm" event={"ID":"18880bdf-d4f4-4510-9f9e-4ca7dbb47077","Type":"ContainerDied","Data":"61121a776410155532152e881f97f43ad9c45ae9ec71c3c159e567eb052ad809"} Oct 30 05:25:33.511501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:33.510944 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61121a776410155532152e881f97f43ad9c45ae9ec71c3c159e567eb052ad809" Oct 30 05:25:33.513122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:33.513094 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" event={"ID":"fffc12ee-42fe-4e32-afb7-b8b35d6c8694","Type":"ContainerStarted","Data":"fe3525c7118c3189f7faa18e7ed22dccc3e6e013e3f8c3f00983d8afee13a374"} Oct 30 05:25:33.513303 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:33.513131 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" event={"ID":"fffc12ee-42fe-4e32-afb7-b8b35d6c8694","Type":"ContainerStarted","Data":"4c349c5556cdab5b3bc363d304ce5e9d8ec0495e5533f4f53417c16c1c655044"} Oct 30 05:25:33.545745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:33.545720 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:33.562109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:33.562060 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" podStartSLOduration=1.562046761 podStartE2EDuration="1.562046761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:33.559817074 +0000 UTC m=+2458.087583860" watchObservedRunningTime="2026-10-30 05:25:33.562046761 +0000 UTC m=+2458.089813610" Oct 30 05:25:33.586211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:33.586178 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659aff68-3d79-4f16-b239-23197429890c" path="/var/lib/kubelet/pods/659aff68-3d79-4f16-b239-23197429890c/volumes" Oct 30 05:25:37.410279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:37.410243 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" Oct 30 05:25:37.411058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:37.410520 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" Oct 30 05:25:40.808264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:40.808218 6587 scope.go:122] "RemoveContainer" containerID="10714accd08f6fb92d1df617073b90fba4bf79b1268191f156c4bd42fa3cf7a3" Oct 30 05:25:40.820478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:40.820449 6587 scope.go:122] "RemoveContainer" containerID="5af747b8a1aeea5a2ea819109c3039c15ff9693c6edb4e675755ec8a675d0a9a" Oct 30 05:25:40.832632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:40.832608 6587 scope.go:122] "RemoveContainer" containerID="c6b96ea85922e1f71d04281b917a4f3da2188e2debcc46689b7c94af98e5e7c8" Oct 30 05:25:40.843516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:40.843349 6587 scope.go:122] "RemoveContainer" containerID="9d29f9997bf06563abd3eefe68822e3388e447a1d9d46a7783a48ea19d59b27e" Oct 30 05:25:42.539716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:42.539683 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:42.552663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:42.552630 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:42.621478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:42.621447 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:25:47.498523 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:47.498488 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc"] Oct 30 05:25:47.512326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:47.512295 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-68b893-mx9lc"] Oct 30 05:25:47.593609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:47.593571 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae76c413-f444-43bb-9103-7bba5c5a2343" path="/var/lib/kubelet/pods/ae76c413-f444-43bb-9103-7bba5c5a2343/volumes" Oct 30 05:25:52.030995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:52.030961 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm"] Oct 30 05:25:52.045585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:52.045533 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4453ce-2pplm"] Oct 30 05:25:53.585063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:53.585028 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa477b52-3248-43b7-811e-239378cf4438" path="/var/lib/kubelet/pods/aa477b52-3248-43b7-811e-239378cf4438/volumes" Oct 30 05:25:55.070195 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:55.070151 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7"] Oct 30 05:25:55.070782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:55.070431 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" containerID="cri-o://e17c36f726279343488d0bd32a10004ce2e5ced6a7d57f3a6fe0b1df5a0c4454" gracePeriod=30 Oct 30 05:25:55.473887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:55.473795 6587 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:25:55.473887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:55.473860 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:25:55.988401 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:25:55.988353 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:25:56.373091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.373056 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw"] Oct 30 05:25:56.374088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.374063 6587 cpu_manager.go:414] "containerMap: removing container" podUID="18880bdf-d4f4-4510-9f9e-4ca7dbb47077" containerName="revoke-keys" Oct 30 05:25:56.374216 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.374088 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18880bdf-d4f4-4510-9f9e-4ca7dbb47077" containerName="revoke-keys" Oct 30 05:25:56.374315 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.374301 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="18880bdf-d4f4-4510-9f9e-4ca7dbb47077" containerName="revoke-keys" Oct 30 05:25:56.381577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.381550 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.406113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.406078 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw"] Oct 30 05:25:56.417156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.415335 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-35a94e-openshift-default-dockercfg-s9msl\"" Oct 30 05:25:56.425179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.425147 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.425341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.425209 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.425341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.425242 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k64jw\" (UniqueName: \"kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-kube-api-access-k64jw\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.425636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.425613 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.425737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.425648 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.425737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.425681 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.425874 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.425757 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.425874 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.425804 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.425977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.425869 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.526857 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.526820 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.527043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.526865 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.527130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.527104 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.527249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.527223 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.527421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.527287 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.527421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.527400 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.527560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.527521 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.527560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.527526 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.527674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.527607 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.527674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.527638 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k64jw\" (UniqueName: \"kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-kube-api-access-k64jw\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.527789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.527673 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.528196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.528169 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.528196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.528190 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.528323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.528280 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.568381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.556604 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.579379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.574373 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k64jw\" (UniqueName: \"kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-kube-api-access-k64jw\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.602379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.598018 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.614396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.611685 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:56.714425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:56.707589 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:57.191450 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:25:57.191410 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24c4b5a_53e8_4f99_ba51_69ec7b6a994a.slice/crio-f2f45eddcd0f0f7277db6a84b7867c64b58533d021bf710d3c491141f9f59726 WatchSource:0}: Error finding container f2f45eddcd0f0f7277db6a84b7867c64b58533d021bf710d3c491141f9f59726: Status 404 returned error can't find the container with id f2f45eddcd0f0f7277db6a84b7867c64b58533d021bf710d3c491141f9f59726 Oct 30 05:25:57.191450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:57.191427 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw"] Oct 30 05:25:57.194456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:57.194422 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:25:57.194549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:57.194486 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:25:57.194549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:57.194516 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:25:57.806432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:57.806393 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" event={"ID":"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a","Type":"ContainerStarted","Data":"c3936e4be7a46060b4f86bd849a2f18c5bf415c3bc4e7dc9df76e8c6482bbf4a"} Oct 30 05:25:57.806885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:57.806454 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" event={"ID":"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a","Type":"ContainerStarted","Data":"f2f45eddcd0f0f7277db6a84b7867c64b58533d021bf710d3c491141f9f59726"} Oct 30 05:25:58.709146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:58.709118 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:58.715219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:58.715192 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:58.779256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:58.779199 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" podStartSLOduration=2.779184685 podStartE2EDuration="2.779184685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:57.88398086 +0000 UTC m=+2482.411747646" watchObservedRunningTime="2026-10-30 05:25:58.779184685 +0000 UTC m=+2483.306951529" Oct 30 05:25:58.818849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:58.818820 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:25:58.819828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:25:58.819806 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:26:00.381417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.381390 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:26:00.407099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.407070 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-kube-api-access-r5cln\" (UniqueName: \"kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-kube-api-access-r5cln\") pod \"040a272c-aca3-4af4-86db-3055cbe66486\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " Oct 30 05:26:00.407282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.407167 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-socket\") pod \"040a272c-aca3-4af4-86db-3055cbe66486\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " Oct 30 05:26:00.407282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.407228 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-istio-token\" (UniqueName: \"kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-istio-token\") pod \"040a272c-aca3-4af4-86db-3055cbe66486\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " Oct 30 05:26:00.407471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.407288 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-data\") pod \"040a272c-aca3-4af4-86db-3055cbe66486\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " Oct 30 05:26:00.407471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.407346 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-certs\") pod \"040a272c-aca3-4af4-86db-3055cbe66486\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " Oct 30 05:26:00.407471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.407390 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-envoy\") pod \"040a272c-aca3-4af4-86db-3055cbe66486\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " Oct 30 05:26:00.407471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.407443 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/040a272c-aca3-4af4-86db-3055cbe66486-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040a272c-aca3-4af4-86db-3055cbe66486-istio-podinfo\") pod \"040a272c-aca3-4af4-86db-3055cbe66486\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " Oct 30 05:26:00.407714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.407520 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040a272c-aca3-4af4-86db-3055cbe66486-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040a272c-aca3-4af4-86db-3055cbe66486-istiod-ca-cert\") pod \"040a272c-aca3-4af4-86db-3055cbe66486\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " Oct 30 05:26:00.407714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.407564 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-credential-socket\") pod \"040a272c-aca3-4af4-86db-3055cbe66486\" (UID: \"040a272c-aca3-4af4-86db-3055cbe66486\") " Oct 30 05:26:00.407941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.407917 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-data" pod "040a272c-aca3-4af4-86db-3055cbe66486" (UID: "040a272c-aca3-4af4-86db-3055cbe66486"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:00.408139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.408119 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-credential-socket" pod "040a272c-aca3-4af4-86db-3055cbe66486" (UID: "040a272c-aca3-4af4-86db-3055cbe66486"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:00.408239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.408221 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.408429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.408352 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-certs" pod "040a272c-aca3-4af4-86db-3055cbe66486" (UID: "040a272c-aca3-4af4-86db-3055cbe66486"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:00.410460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.410431 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040a272c-aca3-4af4-86db-3055cbe66486-istiod-ca-cert" pod "040a272c-aca3-4af4-86db-3055cbe66486" (UID: "040a272c-aca3-4af4-86db-3055cbe66486"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:26:00.410569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.410503 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-socket" pod "040a272c-aca3-4af4-86db-3055cbe66486" (UID: "040a272c-aca3-4af4-86db-3055cbe66486"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:00.411069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.411049 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-envoy" pod "040a272c-aca3-4af4-86db-3055cbe66486" (UID: "040a272c-aca3-4af4-86db-3055cbe66486"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:26:00.413113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.413087 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-kube-api-access-r5cln" pod "040a272c-aca3-4af4-86db-3055cbe66486" (UID: "040a272c-aca3-4af4-86db-3055cbe66486"). InnerVolumeSpecName "kube-api-access-r5cln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:26:00.413786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.413763 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-istio-token" pod "040a272c-aca3-4af4-86db-3055cbe66486" (UID: "040a272c-aca3-4af4-86db-3055cbe66486"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:26:00.415485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.415462 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/040a272c-aca3-4af4-86db-3055cbe66486-istio-podinfo" pod "040a272c-aca3-4af4-86db-3055cbe66486" (UID: "040a272c-aca3-4af4-86db-3055cbe66486"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:26:00.510823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.510789 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.510823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.510824 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.511036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.510839 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040a272c-aca3-4af4-86db-3055cbe66486-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.511036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.510852 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040a272c-aca3-4af4-86db-3055cbe66486-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.511036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.510865 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.511036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.510879 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5cln\" (UniqueName: \"kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-kube-api-access-r5cln\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.511036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.510893 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040a272c-aca3-4af4-86db-3055cbe66486-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.511036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.510906 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040a272c-aca3-4af4-86db-3055cbe66486-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:00.843390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.843338 6587 generic.go:356] "Generic (PLEG): container finished" podID="040a272c-aca3-4af4-86db-3055cbe66486" containerID="e17c36f726279343488d0bd32a10004ce2e5ced6a7d57f3a6fe0b1df5a0c4454" exitCode=0 Oct 30 05:26:00.843625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.843388 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" event={"ID":"040a272c-aca3-4af4-86db-3055cbe66486","Type":"ContainerDied","Data":"e17c36f726279343488d0bd32a10004ce2e5ced6a7d57f3a6fe0b1df5a0c4454"} Oct 30 05:26:00.843625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.843446 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" Oct 30 05:26:00.843625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.843451 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7" event={"ID":"040a272c-aca3-4af4-86db-3055cbe66486","Type":"ContainerDied","Data":"2c41bac0f9d161829f32fc8ff96ae1ae65edbf30bb77fc2546d8b87723dd6080"} Oct 30 05:26:00.843625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.843482 6587 scope.go:122] "RemoveContainer" containerID="e17c36f726279343488d0bd32a10004ce2e5ced6a7d57f3a6fe0b1df5a0c4454" Oct 30 05:26:00.857180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.857157 6587 scope.go:122] "RemoveContainer" containerID="0c00818e6ca0ce776a41115ffd356d70451429ff3d97332404c15ef5167596f9" Oct 30 05:26:00.868967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.868947 6587 scope.go:122] "RemoveContainer" containerID="e17c36f726279343488d0bd32a10004ce2e5ced6a7d57f3a6fe0b1df5a0c4454" Oct 30 05:26:00.869287 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:26:00.869238 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e17c36f726279343488d0bd32a10004ce2e5ced6a7d57f3a6fe0b1df5a0c4454\": container with ID starting with e17c36f726279343488d0bd32a10004ce2e5ced6a7d57f3a6fe0b1df5a0c4454 not found: ID does not exist" containerID="e17c36f726279343488d0bd32a10004ce2e5ced6a7d57f3a6fe0b1df5a0c4454" Oct 30 05:26:00.869429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.869297 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e17c36f726279343488d0bd32a10004ce2e5ced6a7d57f3a6fe0b1df5a0c4454"} err="failed to get container status \"e17c36f726279343488d0bd32a10004ce2e5ced6a7d57f3a6fe0b1df5a0c4454\": rpc error: code = NotFound desc = could not find container \"e17c36f726279343488d0bd32a10004ce2e5ced6a7d57f3a6fe0b1df5a0c4454\": container with ID starting with e17c36f726279343488d0bd32a10004ce2e5ced6a7d57f3a6fe0b1df5a0c4454 not found: ID does not exist" Oct 30 05:26:00.869429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.869316 6587 scope.go:122] "RemoveContainer" containerID="0c00818e6ca0ce776a41115ffd356d70451429ff3d97332404c15ef5167596f9" Oct 30 05:26:00.869693 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:26:00.869659 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c00818e6ca0ce776a41115ffd356d70451429ff3d97332404c15ef5167596f9\": container with ID starting with 0c00818e6ca0ce776a41115ffd356d70451429ff3d97332404c15ef5167596f9 not found: ID does not exist" containerID="0c00818e6ca0ce776a41115ffd356d70451429ff3d97332404c15ef5167596f9" Oct 30 05:26:00.869789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.869700 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c00818e6ca0ce776a41115ffd356d70451429ff3d97332404c15ef5167596f9"} err="failed to get container status \"0c00818e6ca0ce776a41115ffd356d70451429ff3d97332404c15ef5167596f9\": rpc error: code = NotFound desc = could not find container \"0c00818e6ca0ce776a41115ffd356d70451429ff3d97332404c15ef5167596f9\": container with ID starting with 0c00818e6ca0ce776a41115ffd356d70451429ff3d97332404c15ef5167596f9 not found: ID does not exist" Oct 30 05:26:00.910554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.910514 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7"] Oct 30 05:26:00.924134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:00.924098 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7"] Oct 30 05:26:01.583772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:01.583735 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040a272c-aca3-4af4-86db-3055cbe66486" path="/var/lib/kubelet/pods/040a272c-aca3-4af4-86db-3055cbe66486/volumes" Oct 30 05:26:02.356413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.356378 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz"] Oct 30 05:26:02.357149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.357133 6587 cpu_manager.go:414] "containerMap: removing container" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" Oct 30 05:26:02.357199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.357150 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" Oct 30 05:26:02.357199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.357173 6587 cpu_manager.go:414] "containerMap: removing container" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" Oct 30 05:26:02.357199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.357178 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" Oct 30 05:26:02.357379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.357352 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" Oct 30 05:26:02.357421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.357384 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="040a272c-aca3-4af4-86db-3055cbe66486" containerName="istio-proxy" Oct 30 05:26:02.361226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.361207 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" Oct 30 05:26:02.380030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.380006 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz"] Oct 30 05:26:02.386922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.386901 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-35a94e\"" Oct 30 05:26:02.440398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.426380 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-35a94e-dockercfg-r46lp\"" Oct 30 05:26:02.630187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.452892 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5ff08b-f239-4e07-863a-3653c81a6708-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz\" (UID: \"3c5ff08b-f239-4e07-863a-3653c81a6708\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" Oct 30 05:26:02.648604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.648568 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pttjj\" (UniqueName: \"kubernetes.io/projected/3c5ff08b-f239-4e07-863a-3653c81a6708-kube-api-access-pttjj\") pod \"payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz\" (UID: \"3c5ff08b-f239-4e07-863a-3653c81a6708\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" Oct 30 05:26:02.751670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.661451 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd"] Oct 30 05:26:02.751670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.726346 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd"] Oct 30 05:26:02.751670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.728767 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" Oct 30 05:26:02.760700 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.760667 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pttjj\" (UniqueName: \"kubernetes.io/projected/3c5ff08b-f239-4e07-863a-3653c81a6708-kube-api-access-pttjj\") pod \"payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz\" (UID: \"3c5ff08b-f239-4e07-863a-3653c81a6708\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" Oct 30 05:26:02.760908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.760891 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2zxz\" (UniqueName: \"kubernetes.io/projected/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-kube-api-access-f2zxz\") pod \"payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd\" (UID: \"3d330d3a-1853-4bc5-92a6-d7b13bcf2466\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" Oct 30 05:26:02.760987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.760949 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd\" (UID: \"3d330d3a-1853-4bc5-92a6-d7b13bcf2466\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" Oct 30 05:26:02.761129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.761113 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5ff08b-f239-4e07-863a-3653c81a6708-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz\" (UID: \"3c5ff08b-f239-4e07-863a-3653c81a6708\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" Oct 30 05:26:02.761904 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.761878 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5ff08b-f239-4e07-863a-3653c81a6708-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz\" (UID: \"3c5ff08b-f239-4e07-863a-3653c81a6708\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" Oct 30 05:26:02.785022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.784990 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pttjj\" (UniqueName: \"kubernetes.io/projected/3c5ff08b-f239-4e07-863a-3653c81a6708-kube-api-access-pttjj\") pod \"payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz\" (UID: \"3c5ff08b-f239-4e07-863a-3653c81a6708\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" Oct 30 05:26:02.863317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.863277 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2zxz\" (UniqueName: \"kubernetes.io/projected/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-kube-api-access-f2zxz\") pod \"payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd\" (UID: \"3d330d3a-1853-4bc5-92a6-d7b13bcf2466\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" Oct 30 05:26:02.863563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.863351 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd\" (UID: \"3d330d3a-1853-4bc5-92a6-d7b13bcf2466\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" Oct 30 05:26:02.864098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.864071 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd\" (UID: \"3d330d3a-1853-4bc5-92a6-d7b13bcf2466\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" Oct 30 05:26:02.899421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.899301 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2zxz\" (UniqueName: \"kubernetes.io/projected/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-kube-api-access-f2zxz\") pod \"payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd\" (UID: \"3d330d3a-1853-4bc5-92a6-d7b13bcf2466\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" Oct 30 05:26:02.977658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:02.977630 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" Oct 30 05:26:03.036406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.035338 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59"] Oct 30 05:26:03.041814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.041778 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45870a0c-zxm59"] Oct 30 05:26:03.058774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.056985 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" Oct 30 05:26:03.295154 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:26:03.295126 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5ff08b_f239_4e07_863a_3653c81a6708.slice/crio-b0bd31a676b030ca2d2c100da272723a3dcadecacaace560043b43b6ce1eedd9 WatchSource:0}: Error finding container b0bd31a676b030ca2d2c100da272723a3dcadecacaace560043b43b6ce1eedd9: Status 404 returned error can't find the container with id b0bd31a676b030ca2d2c100da272723a3dcadecacaace560043b43b6ce1eedd9 Oct 30 05:26:03.296728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.296700 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz"] Oct 30 05:26:03.411232 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:26:03.411204 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d330d3a_1853_4bc5_92a6_d7b13bcf2466.slice/crio-fa71f06a6edee646b7b1db561ae61478bc29d20124a5f8d9053745c1d63105ab WatchSource:0}: Error finding container fa71f06a6edee646b7b1db561ae61478bc29d20124a5f8d9053745c1d63105ab: Status 404 returned error can't find the container with id fa71f06a6edee646b7b1db561ae61478bc29d20124a5f8d9053745c1d63105ab Oct 30 05:26:03.413097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.413076 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd"] Oct 30 05:26:03.586687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.586648 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d624690-2e68-45a4-9e66-529b4de0b3a2" path="/var/lib/kubelet/pods/1d624690-2e68-45a4-9e66-529b4de0b3a2/volumes" Oct 30 05:26:03.876456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.876413 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" event={"ID":"3d330d3a-1853-4bc5-92a6-d7b13bcf2466","Type":"ContainerStarted","Data":"c5ce09fcf677c8c7754a797f2504397e08430bdd9aac783d4e6941e006dcface"} Oct 30 05:26:03.876940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.876471 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" event={"ID":"3d330d3a-1853-4bc5-92a6-d7b13bcf2466","Type":"ContainerStarted","Data":"fa71f06a6edee646b7b1db561ae61478bc29d20124a5f8d9053745c1d63105ab"} Oct 30 05:26:03.876940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.876509 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" Oct 30 05:26:03.878314 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.878284 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" event={"ID":"3c5ff08b-f239-4e07-863a-3653c81a6708","Type":"ContainerStarted","Data":"3b5a472e2c0750c8b11ad491bf2ca7da9e9f3bd5c5214959f3f5f5e67dd4f931"} Oct 30 05:26:03.878471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.878326 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" event={"ID":"3c5ff08b-f239-4e07-863a-3653c81a6708","Type":"ContainerStarted","Data":"b0bd31a676b030ca2d2c100da272723a3dcadecacaace560043b43b6ce1eedd9"} Oct 30 05:26:03.878471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.878387 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" Oct 30 05:26:03.923680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.923630 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" podStartSLOduration=1.923616031 podStartE2EDuration="1.923616031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:26:03.920479284 +0000 UTC m=+2488.448246073" watchObservedRunningTime="2026-10-30 05:26:03.923616031 +0000 UTC m=+2488.451382817" Oct 30 05:26:03.967301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:03.967242 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" podStartSLOduration=1.9672283720000001 podStartE2EDuration="1.967228372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:26:03.962904371 +0000 UTC m=+2488.490671157" watchObservedRunningTime="2026-10-30 05:26:03.967228372 +0000 UTC m=+2488.494995157" Oct 30 05:26:14.130104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.128421 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx"] Oct 30 05:26:14.146910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.146432 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.155385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.151678 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx"] Oct 30 05:26:14.172326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.165739 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-dfb231-openshift-default-dockercfg-t5h89\"" Oct 30 05:26:14.313316 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.313262 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-data\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.313570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.313352 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-token\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.313570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.313405 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e413931-a15a-4fb2-82ef-4e0096717c33-istiod-ca-cert\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.313570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.313527 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.313787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.313631 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-envoy\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.313787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.313691 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-credential-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.313787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.313731 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-podinfo\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.313975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.313827 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-certs\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.313975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.313888 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfd2q\" (UniqueName: \"kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-kube-api-access-cfd2q\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.415457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.415335 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-envoy\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.415457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.415444 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-credential-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.415719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.415484 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-podinfo\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.415719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.415585 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-certs\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.415719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.415649 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfd2q\" (UniqueName: \"kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-kube-api-access-cfd2q\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.415889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.415758 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-data\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.415889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.415820 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-token\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.415889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.415857 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e413931-a15a-4fb2-82ef-4e0096717c33-istiod-ca-cert\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.416072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.415985 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.420383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.416596 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.420383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.417622 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e413931-a15a-4fb2-82ef-4e0096717c33-istiod-ca-cert\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.420383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.418629 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-certs\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.420383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.418874 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-data\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.420383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.419448 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-credential-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.445381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.441937 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-token\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.445381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.445121 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfd2q\" (UniqueName: \"kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-kube-api-access-cfd2q\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.454993 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.454914 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-podinfo\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.455453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.455399 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-envoy\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.520179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.520147 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:14.909702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.905541 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" Oct 30 05:26:14.921381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:14.918740 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" Oct 30 05:26:15.237330 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:15.237299 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx"] Oct 30 05:26:15.238663 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:26:15.238635 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e413931_a15a_4fb2_82ef_4e0096717c33.slice/crio-4f5d19c0ffdae0bb92941653baad9068c9a94d154f50a258b42e2ee5092f8f49 WatchSource:0}: Error finding container 4f5d19c0ffdae0bb92941653baad9068c9a94d154f50a258b42e2ee5092f8f49: Status 404 returned error can't find the container with id 4f5d19c0ffdae0bb92941653baad9068c9a94d154f50a258b42e2ee5092f8f49 Oct 30 05:26:15.252303 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:15.252264 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:26:15.252581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:15.252555 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:26:15.252732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:15.252713 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:26:16.061224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:16.061190 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" event={"ID":"0e413931-a15a-4fb2-82ef-4e0096717c33","Type":"ContainerStarted","Data":"a8ddaa1951425ea4f8c5a558997284c641f84e111f873509902044c689ff91d1"} Oct 30 05:26:16.061437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:16.061245 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" event={"ID":"0e413931-a15a-4fb2-82ef-4e0096717c33","Type":"ContainerStarted","Data":"4f5d19c0ffdae0bb92941653baad9068c9a94d154f50a258b42e2ee5092f8f49"} Oct 30 05:26:16.530434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:16.523420 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:16.543606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:16.542475 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Oct 30 05:26:16.543606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:16.542539 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" podUID="0e413931-a15a-4fb2-82ef-4e0096717c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Oct 30 05:26:17.150492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.150428 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" podStartSLOduration=3.1504078189999998 podStartE2EDuration="3.150407819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:26:16.146320976 +0000 UTC m=+2500.674087765" watchObservedRunningTime="2026-10-30 05:26:17.150407819 +0000 UTC m=+2501.678174605" Oct 30 05:26:17.151320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.151290 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt"] Oct 30 05:26:17.162711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.162691 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt"] Oct 30 05:26:17.162972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.162920 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" Oct 30 05:26:17.181701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.181677 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:26:17.181851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.181745 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:26:17.181914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.181851 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:26:17.182076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.182062 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:26:17.281612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.281569 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b270a6-2n9bt\" (UID: \"025c2255-76a8-49c2-8956-3d81e7a4e725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" Oct 30 05:26:17.281790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.281771 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrlvd\" (UniqueName: \"kubernetes.io/projected/025c2255-76a8-49c2-8956-3d81e7a4e725-kube-api-access-vrlvd\") pod \"maas-api-revoke-sub-14b270a6-2n9bt\" (UID: \"025c2255-76a8-49c2-8956-3d81e7a4e725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" Oct 30 05:26:17.281919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.281903 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b270a6-2n9bt\" (UID: \"025c2255-76a8-49c2-8956-3d81e7a4e725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" Oct 30 05:26:17.383252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.383211 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrlvd\" (UniqueName: \"kubernetes.io/projected/025c2255-76a8-49c2-8956-3d81e7a4e725-kube-api-access-vrlvd\") pod \"maas-api-revoke-sub-14b270a6-2n9bt\" (UID: \"025c2255-76a8-49c2-8956-3d81e7a4e725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" Oct 30 05:26:17.383505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.383310 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b270a6-2n9bt\" (UID: \"025c2255-76a8-49c2-8956-3d81e7a4e725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" Oct 30 05:26:17.383583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.383549 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b270a6-2n9bt\" (UID: \"025c2255-76a8-49c2-8956-3d81e7a4e725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" Oct 30 05:26:17.384207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.384185 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b270a6-2n9bt\" (UID: \"025c2255-76a8-49c2-8956-3d81e7a4e725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" Oct 30 05:26:17.385808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.385781 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b270a6-2n9bt\" (UID: \"025c2255-76a8-49c2-8956-3d81e7a4e725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" Oct 30 05:26:17.405424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.405036 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrlvd\" (UniqueName: \"kubernetes.io/projected/025c2255-76a8-49c2-8956-3d81e7a4e725-kube-api-access-vrlvd\") pod \"maas-api-revoke-sub-14b270a6-2n9bt\" (UID: \"025c2255-76a8-49c2-8956-3d81e7a4e725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" Oct 30 05:26:17.491871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.491835 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" Oct 30 05:26:17.565384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.562464 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Oct 30 05:26:17.565384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:17.562524 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" podUID="0e413931-a15a-4fb2-82ef-4e0096717c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Oct 30 05:26:18.051400 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:26:18.034776 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025c2255_76a8_49c2_8956_3d81e7a4e725.slice/crio-091a2b9d567bca082b3bcecceb6714ae567efb5b24350b0275a8b23024c35590 WatchSource:0}: Error finding container 091a2b9d567bca082b3bcecceb6714ae567efb5b24350b0275a8b23024c35590: Status 404 returned error can't find the container with id 091a2b9d567bca082b3bcecceb6714ae567efb5b24350b0275a8b23024c35590 Oct 30 05:26:18.055376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:18.055329 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt"] Oct 30 05:26:18.084746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:18.084707 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" event={"ID":"025c2255-76a8-49c2-8956-3d81e7a4e725","Type":"ContainerStarted","Data":"091a2b9d567bca082b3bcecceb6714ae567efb5b24350b0275a8b23024c35590"} Oct 30 05:26:18.521781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:18.521744 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Oct 30 05:26:18.522080 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:18.522047 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" podUID="0e413931-a15a-4fb2-82ef-4e0096717c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Oct 30 05:26:19.096198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:19.096161 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" event={"ID":"025c2255-76a8-49c2-8956-3d81e7a4e725","Type":"ContainerStarted","Data":"2517c8ab27190320bc13e0671a92d5603cb8713ed70bb341655c9b85a376e586"} Oct 30 05:26:19.156684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:19.156429 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" podStartSLOduration=2.1564146060000002 podStartE2EDuration="2.156414606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:26:19.154740129 +0000 UTC m=+2503.682506915" watchObservedRunningTime="2026-10-30 05:26:19.156414606 +0000 UTC m=+2503.684181391" Oct 30 05:26:19.525478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:19.525392 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Oct 30 05:26:19.525738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:19.525706 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" podUID="0e413931-a15a-4fb2-82ef-4e0096717c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Oct 30 05:26:20.106694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:20.106656 6587 generic.go:356] "Generic (PLEG): container finished" podID="025c2255-76a8-49c2-8956-3d81e7a4e725" containerID="2517c8ab27190320bc13e0671a92d5603cb8713ed70bb341655c9b85a376e586" exitCode=0 Oct 30 05:26:20.107084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:20.106736 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" event={"ID":"025c2255-76a8-49c2-8956-3d81e7a4e725","Type":"ContainerDied","Data":"2517c8ab27190320bc13e0671a92d5603cb8713ed70bb341655c9b85a376e586"} Oct 30 05:26:20.522815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:20.522717 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Oct 30 05:26:20.522815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:20.522780 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" podUID="0e413931-a15a-4fb2-82ef-4e0096717c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Oct 30 05:26:21.462173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:21.462134 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" Oct 30 05:26:21.514425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:21.507799 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-openshift-ca\") pod \"025c2255-76a8-49c2-8956-3d81e7a4e725\" (UID: \"025c2255-76a8-49c2-8956-3d81e7a4e725\") " Oct 30 05:26:21.514425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:21.507852 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025c2255-76a8-49c2-8956-3d81e7a4e725-kube-api-access-vrlvd\" (UniqueName: \"kubernetes.io/projected/025c2255-76a8-49c2-8956-3d81e7a4e725-kube-api-access-vrlvd\") pod \"025c2255-76a8-49c2-8956-3d81e7a4e725\" (UID: \"025c2255-76a8-49c2-8956-3d81e7a4e725\") " Oct 30 05:26:21.514425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:21.508013 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-xks-ca\") pod \"025c2255-76a8-49c2-8956-3d81e7a4e725\" (UID: \"025c2255-76a8-49c2-8956-3d81e7a4e725\") " Oct 30 05:26:21.514425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:21.511608 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-openshift-ca" pod "025c2255-76a8-49c2-8956-3d81e7a4e725" (UID: "025c2255-76a8-49c2-8956-3d81e7a4e725"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:26:21.514425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:21.511911 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-xks-ca" pod "025c2255-76a8-49c2-8956-3d81e7a4e725" (UID: "025c2255-76a8-49c2-8956-3d81e7a4e725"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:26:21.514425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:21.513921 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025c2255-76a8-49c2-8956-3d81e7a4e725-kube-api-access-vrlvd" pod "025c2255-76a8-49c2-8956-3d81e7a4e725" (UID: "025c2255-76a8-49c2-8956-3d81e7a4e725"). InnerVolumeSpecName "kube-api-access-vrlvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:26:21.534619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:21.531779 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Oct 30 05:26:21.534619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:21.531842 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" podUID="0e413931-a15a-4fb2-82ef-4e0096717c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Oct 30 05:26:21.611847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:21.611816 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:21.611847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:21.611845 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrlvd\" (UniqueName: \"kubernetes.io/projected/025c2255-76a8-49c2-8956-3d81e7a4e725-kube-api-access-vrlvd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:21.612069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:21.611862 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025c2255-76a8-49c2-8956-3d81e7a4e725-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:22.148901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:22.148862 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" Oct 30 05:26:22.149127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:22.148895 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt" event={"ID":"025c2255-76a8-49c2-8956-3d81e7a4e725","Type":"ContainerDied","Data":"091a2b9d567bca082b3bcecceb6714ae567efb5b24350b0275a8b23024c35590"} Oct 30 05:26:22.149127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:22.148959 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="091a2b9d567bca082b3bcecceb6714ae567efb5b24350b0275a8b23024c35590" Oct 30 05:26:22.527485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:22.527395 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:22.528225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:22.528204 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:22.529232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:22.529209 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:26:25.988931 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:26:25.988896 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:26:27.553460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.553421 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq"] Oct 30 05:26:27.554723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.554700 6587 cpu_manager.go:414] "containerMap: removing container" podUID="025c2255-76a8-49c2-8956-3d81e7a4e725" containerName="revoke-keys" Oct 30 05:26:27.554783 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.554727 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025c2255-76a8-49c2-8956-3d81e7a4e725" containerName="revoke-keys" Oct 30 05:26:27.554946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.554933 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="025c2255-76a8-49c2-8956-3d81e7a4e725" containerName="revoke-keys" Oct 30 05:26:27.559737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.559717 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" Oct 30 05:26:27.576471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.576444 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:26:27.584641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.584609 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:26:27.584776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.584748 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:26:27.585345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.585322 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:26:27.589760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.587088 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq"] Oct 30 05:26:27.726890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.726844 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7526ac7b-76wgq\" (UID: \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" Oct 30 05:26:27.727105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.726915 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twg6l\" (UniqueName: \"kubernetes.io/projected/8dfc61e5-0c2d-44a8-9398-b35decd958dd-kube-api-access-twg6l\") pod \"maas-api-revoke-sub-7526ac7b-76wgq\" (UID: \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" Oct 30 05:26:27.727105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.727082 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7526ac7b-76wgq\" (UID: \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" Oct 30 05:26:27.828846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.828751 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7526ac7b-76wgq\" (UID: \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" Oct 30 05:26:27.829029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.828857 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twg6l\" (UniqueName: \"kubernetes.io/projected/8dfc61e5-0c2d-44a8-9398-b35decd958dd-kube-api-access-twg6l\") pod \"maas-api-revoke-sub-7526ac7b-76wgq\" (UID: \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" Oct 30 05:26:27.829029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.828994 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7526ac7b-76wgq\" (UID: \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" Oct 30 05:26:27.829703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.829672 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7526ac7b-76wgq\" (UID: \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" Oct 30 05:26:27.831266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.831238 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7526ac7b-76wgq\" (UID: \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" Oct 30 05:26:27.852220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.852191 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twg6l\" (UniqueName: \"kubernetes.io/projected/8dfc61e5-0c2d-44a8-9398-b35decd958dd-kube-api-access-twg6l\") pod \"maas-api-revoke-sub-7526ac7b-76wgq\" (UID: \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" Oct 30 05:26:27.878025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:27.877998 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" Oct 30 05:26:28.192919 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:26:28.192889 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfc61e5_0c2d_44a8_9398_b35decd958dd.slice/crio-15080626965e572edf18508a6e4b649f738dbdc2393f70304be47562612f13df WatchSource:0}: Error finding container 15080626965e572edf18508a6e4b649f738dbdc2393f70304be47562612f13df: Status 404 returned error can't find the container with id 15080626965e572edf18508a6e4b649f738dbdc2393f70304be47562612f13df Oct 30 05:26:28.195415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:28.195387 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq"] Oct 30 05:26:28.207645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:28.207605 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" event={"ID":"8dfc61e5-0c2d-44a8-9398-b35decd958dd","Type":"ContainerStarted","Data":"15080626965e572edf18508a6e4b649f738dbdc2393f70304be47562612f13df"} Oct 30 05:26:29.243382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:29.242858 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" event={"ID":"8dfc61e5-0c2d-44a8-9398-b35decd958dd","Type":"ContainerStarted","Data":"901aa29d8026cc580053f05145635a39295436a52417fd8d84144646b39088fa"} Oct 30 05:26:29.290001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:29.289896 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" podStartSLOduration=2.289875886 podStartE2EDuration="2.289875886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:26:29.286737106 +0000 UTC m=+2513.814503891" watchObservedRunningTime="2026-10-30 05:26:29.289875886 +0000 UTC m=+2513.817642674" Oct 30 05:26:30.259044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:30.259000 6587 generic.go:356] "Generic (PLEG): container finished" podID="8dfc61e5-0c2d-44a8-9398-b35decd958dd" containerID="901aa29d8026cc580053f05145635a39295436a52417fd8d84144646b39088fa" exitCode=0 Oct 30 05:26:30.259533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:30.259342 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" event={"ID":"8dfc61e5-0c2d-44a8-9398-b35decd958dd","Type":"ContainerDied","Data":"901aa29d8026cc580053f05145635a39295436a52417fd8d84144646b39088fa"} Oct 30 05:26:31.582608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:31.582515 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" Oct 30 05:26:31.672159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:31.672125 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-xks-ca\") pod \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\" (UID: \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\") " Oct 30 05:26:31.672410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:31.672184 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-openshift-ca\") pod \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\" (UID: \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\") " Oct 30 05:26:31.672512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:31.672433 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dfc61e5-0c2d-44a8-9398-b35decd958dd-kube-api-access-twg6l\" (UniqueName: \"kubernetes.io/projected/8dfc61e5-0c2d-44a8-9398-b35decd958dd-kube-api-access-twg6l\") pod \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\" (UID: \"8dfc61e5-0c2d-44a8-9398-b35decd958dd\") " Oct 30 05:26:31.674255 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:31.674225 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-openshift-ca" pod "8dfc61e5-0c2d-44a8-9398-b35decd958dd" (UID: "8dfc61e5-0c2d-44a8-9398-b35decd958dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:26:31.676166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:31.676137 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dfc61e5-0c2d-44a8-9398-b35decd958dd-kube-api-access-twg6l" pod "8dfc61e5-0c2d-44a8-9398-b35decd958dd" (UID: "8dfc61e5-0c2d-44a8-9398-b35decd958dd"). InnerVolumeSpecName "kube-api-access-twg6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:26:31.676497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:31.676474 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-xks-ca" pod "8dfc61e5-0c2d-44a8-9398-b35decd958dd" (UID: "8dfc61e5-0c2d-44a8-9398-b35decd958dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:26:31.779391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:31.776795 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twg6l\" (UniqueName: \"kubernetes.io/projected/8dfc61e5-0c2d-44a8-9398-b35decd958dd-kube-api-access-twg6l\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:31.779391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:31.776833 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:31.779391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:31.776849 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dfc61e5-0c2d-44a8-9398-b35decd958dd-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:26:32.365640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:32.363905 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" Oct 30 05:26:32.365640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:32.363922 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq" event={"ID":"8dfc61e5-0c2d-44a8-9398-b35decd958dd","Type":"ContainerDied","Data":"15080626965e572edf18508a6e4b649f738dbdc2393f70304be47562612f13df"} Oct 30 05:26:32.365640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:32.363967 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15080626965e572edf18508a6e4b649f738dbdc2393f70304be47562612f13df" Oct 30 05:26:40.442543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:40.442454 6587 generic.go:356] "Generic (PLEG): container finished" podID="111bc68f-98c2-4041-a39f-daa438577eaf" containerID="6f0835455192fd83e9ef561dd2773d9e3402b63514a24b6f502e56e637b020fa" exitCode=0 Oct 30 05:26:40.442543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:40.442525 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" event={"ID":"111bc68f-98c2-4041-a39f-daa438577eaf","Type":"ContainerDied","Data":"6f0835455192fd83e9ef561dd2773d9e3402b63514a24b6f502e56e637b020fa"} Oct 30 05:26:40.443016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:40.442996 6587 scope.go:122] "RemoveContainer" containerID="6f0835455192fd83e9ef561dd2773d9e3402b63514a24b6f502e56e637b020fa" Oct 30 05:26:40.444866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:40.444833 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:26:40.445007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:40.444923 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:26:40.445007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:40.444993 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:26:40.945380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:40.945081 6587 scope.go:122] "RemoveContainer" containerID="41e9a278a522be8368a4f4dba7ae69e5e42ef20ea9367767fff13e7e07a5a052" Oct 30 05:26:40.974648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:40.974629 6587 scope.go:122] "RemoveContainer" containerID="d2efa9df0a43aa5b31c4f7ec1bd9f35451abb070693b96b6c3c7928d86e8c378" Oct 30 05:26:40.986497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:40.986472 6587 scope.go:122] "RemoveContainer" containerID="8c98b567fbaf66daf1704d25ad8ea97375d5c5bb5a9573e8c67fdb655eaa9126" Oct 30 05:26:40.997487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:40.997463 6587 scope.go:122] "RemoveContainer" containerID="d29ffc2c9cd06d155a68f0f4703c652662a02cee93dacaf115873e27933a34ee" Oct 30 05:26:41.453742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:41.453707 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" event={"ID":"111bc68f-98c2-4041-a39f-daa438577eaf","Type":"ContainerStarted","Data":"c623275b4b69848e013f2bf0bf22db1d354e77da1fe75b9b10e686051786e1ae"} Oct 30 05:26:42.222626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:42.222150 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:26:42.222626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:42.222272 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Oct 30 05:26:42.222626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:42.222313 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" podUID="111bc68f-98c2-4041-a39f-daa438577eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Oct 30 05:26:42.879951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:42.878295 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:26:43.238563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:43.238472 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Oct 30 05:26:43.238861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:43.238827 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" podUID="111bc68f-98c2-4041-a39f-daa438577eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Oct 30 05:26:44.223385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:44.219711 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Oct 30 05:26:44.223385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:44.219772 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" podUID="111bc68f-98c2-4041-a39f-daa438577eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Oct 30 05:26:45.219384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:45.217747 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Oct 30 05:26:45.219384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:45.217824 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" podUID="111bc68f-98c2-4041-a39f-daa438577eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Oct 30 05:26:46.223384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:46.219676 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Oct 30 05:26:46.223384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:46.219746 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" podUID="111bc68f-98c2-4041-a39f-daa438577eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Oct 30 05:26:47.031646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:47.031558 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp"] Oct 30 05:26:47.045119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:47.045086 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758ee42e-xw5bp"] Oct 30 05:26:47.222657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:47.222623 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:26:47.223897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:47.223875 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:26:47.585560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:47.585528 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc232fb-c42a-4876-9aa3-e64af1c1484a" path="/var/lib/kubelet/pods/2dc232fb-c42a-4876-9aa3-e64af1c1484a/volumes" Oct 30 05:26:49.886802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:49.886761 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6"] Oct 30 05:26:54.918169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:54.918132 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5"] Oct 30 05:26:54.953084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:54.953050 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m"] Oct 30 05:26:55.987274 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:26:55.987238 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:26:59.918780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:59.918733 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" podUID="57233b00-5af2-4532-a38b-5870d0181606" containerName="payload-processing" containerID="cri-o://e9e34fe6504a2ded817d1e37112040b4e5e00128949ce6ab67498972f0b4f5e3" gracePeriod=25 Oct 30 05:26:59.954212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:26:59.954167 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" podUID="fc5faab8-cb81-4304-8715-f3909474e267" containerName="payload-pre-processing" containerID="cri-o://38d789e2b785b4ccdce5a4ee29d2d2bf06378e599709546d4c07b6173efa3fdb" gracePeriod=25 Oct 30 05:27:00.268124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.267699 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" Oct 30 05:27:00.317918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.317891 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" Oct 30 05:27:00.355033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.354995 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5faab8-cb81-4304-8715-f3909474e267-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5faab8-cb81-4304-8715-f3909474e267-plugins-config-volume\") pod \"fc5faab8-cb81-4304-8715-f3909474e267\" (UID: \"fc5faab8-cb81-4304-8715-f3909474e267\") " Oct 30 05:27:00.355229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.355199 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57233b00-5af2-4532-a38b-5870d0181606-kube-api-access-lf7x9\" (UniqueName: \"kubernetes.io/projected/57233b00-5af2-4532-a38b-5870d0181606-kube-api-access-lf7x9\") pod \"57233b00-5af2-4532-a38b-5870d0181606\" (UID: \"57233b00-5af2-4532-a38b-5870d0181606\") " Oct 30 05:27:00.355405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.355338 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5faab8-cb81-4304-8715-f3909474e267-plugins-config-volume" pod "fc5faab8-cb81-4304-8715-f3909474e267" (UID: "fc5faab8-cb81-4304-8715-f3909474e267"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:27:00.355507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.355454 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5faab8-cb81-4304-8715-f3909474e267-kube-api-access-zpq7h\" (UniqueName: \"kubernetes.io/projected/fc5faab8-cb81-4304-8715-f3909474e267-kube-api-access-zpq7h\") pod \"fc5faab8-cb81-4304-8715-f3909474e267\" (UID: \"fc5faab8-cb81-4304-8715-f3909474e267\") " Oct 30 05:27:00.355608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.355578 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57233b00-5af2-4532-a38b-5870d0181606-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57233b00-5af2-4532-a38b-5870d0181606-plugins-config-volume\") pod \"57233b00-5af2-4532-a38b-5870d0181606\" (UID: \"57233b00-5af2-4532-a38b-5870d0181606\") " Oct 30 05:27:00.356049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.356022 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57233b00-5af2-4532-a38b-5870d0181606-plugins-config-volume" pod "57233b00-5af2-4532-a38b-5870d0181606" (UID: "57233b00-5af2-4532-a38b-5870d0181606"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:27:00.356534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.356507 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57233b00-5af2-4532-a38b-5870d0181606-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:00.356635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.356539 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5faab8-cb81-4304-8715-f3909474e267-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:00.358512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.358334 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57233b00-5af2-4532-a38b-5870d0181606-kube-api-access-lf7x9" pod "57233b00-5af2-4532-a38b-5870d0181606" (UID: "57233b00-5af2-4532-a38b-5870d0181606"). InnerVolumeSpecName "kube-api-access-lf7x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:27:00.358512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.358428 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5faab8-cb81-4304-8715-f3909474e267-kube-api-access-zpq7h" pod "fc5faab8-cb81-4304-8715-f3909474e267" (UID: "fc5faab8-cb81-4304-8715-f3909474e267"). InnerVolumeSpecName "kube-api-access-zpq7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:27:00.457940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.457847 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf7x9\" (UniqueName: \"kubernetes.io/projected/57233b00-5af2-4532-a38b-5870d0181606-kube-api-access-lf7x9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:00.457940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.457884 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpq7h\" (UniqueName: \"kubernetes.io/projected/fc5faab8-cb81-4304-8715-f3909474e267-kube-api-access-zpq7h\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:00.735701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.735603 6587 generic.go:356] "Generic (PLEG): container finished" podID="fc5faab8-cb81-4304-8715-f3909474e267" containerID="38d789e2b785b4ccdce5a4ee29d2d2bf06378e599709546d4c07b6173efa3fdb" exitCode=0 Oct 30 05:27:00.735701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.735648 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" Oct 30 05:27:00.735947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.735815 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" event={"ID":"fc5faab8-cb81-4304-8715-f3909474e267","Type":"ContainerDied","Data":"38d789e2b785b4ccdce5a4ee29d2d2bf06378e599709546d4c07b6173efa3fdb"} Oct 30 05:27:00.735947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.735863 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m" event={"ID":"fc5faab8-cb81-4304-8715-f3909474e267","Type":"ContainerDied","Data":"fefcbc39387a8bef4a3420c348e638ab6869c7e8950b30e012b0b48022b8497b"} Oct 30 05:27:00.735947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.735892 6587 scope.go:122] "RemoveContainer" containerID="38d789e2b785b4ccdce5a4ee29d2d2bf06378e599709546d4c07b6173efa3fdb" Oct 30 05:27:00.743686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.743585 6587 generic.go:356] "Generic (PLEG): container finished" podID="57233b00-5af2-4532-a38b-5870d0181606" containerID="e9e34fe6504a2ded817d1e37112040b4e5e00128949ce6ab67498972f0b4f5e3" exitCode=0 Oct 30 05:27:00.743831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.743792 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" event={"ID":"57233b00-5af2-4532-a38b-5870d0181606","Type":"ContainerDied","Data":"e9e34fe6504a2ded817d1e37112040b4e5e00128949ce6ab67498972f0b4f5e3"} Oct 30 05:27:00.743903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.743850 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" event={"ID":"57233b00-5af2-4532-a38b-5870d0181606","Type":"ContainerDied","Data":"1dcd4b7b06e5a6761841d9697677fd17e2f01ff3dfeb2611c3c26760061a2069"} Oct 30 05:27:00.743984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.743956 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5" Oct 30 05:27:00.756937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.756921 6587 scope.go:122] "RemoveContainer" containerID="38d789e2b785b4ccdce5a4ee29d2d2bf06378e599709546d4c07b6173efa3fdb" Oct 30 05:27:00.757233 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:27:00.757205 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d789e2b785b4ccdce5a4ee29d2d2bf06378e599709546d4c07b6173efa3fdb\": container with ID starting with 38d789e2b785b4ccdce5a4ee29d2d2bf06378e599709546d4c07b6173efa3fdb not found: ID does not exist" containerID="38d789e2b785b4ccdce5a4ee29d2d2bf06378e599709546d4c07b6173efa3fdb" Oct 30 05:27:00.757300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.757244 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d789e2b785b4ccdce5a4ee29d2d2bf06378e599709546d4c07b6173efa3fdb"} err="failed to get container status \"38d789e2b785b4ccdce5a4ee29d2d2bf06378e599709546d4c07b6173efa3fdb\": rpc error: code = NotFound desc = could not find container \"38d789e2b785b4ccdce5a4ee29d2d2bf06378e599709546d4c07b6173efa3fdb\": container with ID starting with 38d789e2b785b4ccdce5a4ee29d2d2bf06378e599709546d4c07b6173efa3fdb not found: ID does not exist" Oct 30 05:27:00.757300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.757276 6587 scope.go:122] "RemoveContainer" containerID="e9e34fe6504a2ded817d1e37112040b4e5e00128949ce6ab67498972f0b4f5e3" Oct 30 05:27:00.768484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.768465 6587 scope.go:122] "RemoveContainer" containerID="e9e34fe6504a2ded817d1e37112040b4e5e00128949ce6ab67498972f0b4f5e3" Oct 30 05:27:00.768753 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:27:00.768734 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e34fe6504a2ded817d1e37112040b4e5e00128949ce6ab67498972f0b4f5e3\": container with ID starting with e9e34fe6504a2ded817d1e37112040b4e5e00128949ce6ab67498972f0b4f5e3 not found: ID does not exist" containerID="e9e34fe6504a2ded817d1e37112040b4e5e00128949ce6ab67498972f0b4f5e3" Oct 30 05:27:00.768807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.768760 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e34fe6504a2ded817d1e37112040b4e5e00128949ce6ab67498972f0b4f5e3"} err="failed to get container status \"e9e34fe6504a2ded817d1e37112040b4e5e00128949ce6ab67498972f0b4f5e3\": rpc error: code = NotFound desc = could not find container \"e9e34fe6504a2ded817d1e37112040b4e5e00128949ce6ab67498972f0b4f5e3\": container with ID starting with e9e34fe6504a2ded817d1e37112040b4e5e00128949ce6ab67498972f0b4f5e3 not found: ID does not exist" Oct 30 05:27:00.807546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.807517 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m"] Oct 30 05:27:00.820926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.820902 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m"] Oct 30 05:27:00.878201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.878163 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5"] Oct 30 05:27:00.889579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:00.889551 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5"] Oct 30 05:27:01.034710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:01.034622 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9"] Oct 30 05:27:01.039450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:01.039421 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a39058-bb2d9"] Oct 30 05:27:01.584395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:01.584323 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57233b00-5af2-4532-a38b-5870d0181606" path="/var/lib/kubelet/pods/57233b00-5af2-4532-a38b-5870d0181606/volumes" Oct 30 05:27:01.584882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:01.584862 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750ce4e5-54b1-40a0-95e0-c496b89f9b4e" path="/var/lib/kubelet/pods/750ce4e5-54b1-40a0-95e0-c496b89f9b4e/volumes" Oct 30 05:27:01.585387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:01.585339 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5faab8-cb81-4304-8715-f3909474e267" path="/var/lib/kubelet/pods/fc5faab8-cb81-4304-8715-f3909474e267/volumes" Oct 30 05:27:04.914794 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:04.914744 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" podUID="d0fd7250-35d9-482a-a71b-2be4c8c3ad6e" containerName="main" containerID="cri-o://f288c5770f5a090499c2eeb8efb6b895b432a1b459b158167772100e4d55ae78" gracePeriod=45 Oct 30 05:27:05.208142 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.208114 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:27:05.362384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.341954 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-model-cache\") pod \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " Oct 30 05:27:05.362384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.342177 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tmp-dir\") pod \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " Oct 30 05:27:05.362384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.342228 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-dshm\") pod \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " Oct 30 05:27:05.362384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.342489 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-kube-api-access-cd6dm\" (UniqueName: \"kubernetes.io/projected/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-kube-api-access-cd6dm\") pod \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " Oct 30 05:27:05.362384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.342531 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tls-certs\") pod \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " Oct 30 05:27:05.362384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.342588 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-home\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-home\") pod \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\" (UID: \"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e\") " Oct 30 05:27:05.363304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.363006 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-home" pod "d0fd7250-35d9-482a-a71b-2be4c8c3ad6e" (UID: "d0fd7250-35d9-482a-a71b-2be4c8c3ad6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:27:05.363752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.363553 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-model-cache" pod "d0fd7250-35d9-482a-a71b-2be4c8c3ad6e" (UID: "d0fd7250-35d9-482a-a71b-2be4c8c3ad6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:27:05.364190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.363886 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tmp-dir" pod "d0fd7250-35d9-482a-a71b-2be4c8c3ad6e" (UID: "d0fd7250-35d9-482a-a71b-2be4c8c3ad6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:27:05.396457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.393587 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-kube-api-access-cd6dm" pod "d0fd7250-35d9-482a-a71b-2be4c8c3ad6e" (UID: "d0fd7250-35d9-482a-a71b-2be4c8c3ad6e"). InnerVolumeSpecName "kube-api-access-cd6dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:27:05.396457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.394375 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-dshm" pod "d0fd7250-35d9-482a-a71b-2be4c8c3ad6e" (UID: "d0fd7250-35d9-482a-a71b-2be4c8c3ad6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:27:05.405394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.404861 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tls-certs" pod "d0fd7250-35d9-482a-a71b-2be4c8c3ad6e" (UID: "d0fd7250-35d9-482a-a71b-2be4c8c3ad6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:27:05.455391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.444826 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:05.455391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.444859 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:05.455391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.444873 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd6dm\" (UniqueName: \"kubernetes.io/projected/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-kube-api-access-cd6dm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:05.455391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.444886 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:05.455391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.444900 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:05.455391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.444911 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:05.823382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.818910 6587 generic.go:356] "Generic (PLEG): container finished" podID="d0fd7250-35d9-482a-a71b-2be4c8c3ad6e" containerID="f288c5770f5a090499c2eeb8efb6b895b432a1b459b158167772100e4d55ae78" exitCode=0 Oct 30 05:27:05.823382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.819185 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" event={"ID":"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e","Type":"ContainerDied","Data":"f288c5770f5a090499c2eeb8efb6b895b432a1b459b158167772100e4d55ae78"} Oct 30 05:27:05.823382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.819221 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" event={"ID":"d0fd7250-35d9-482a-a71b-2be4c8c3ad6e","Type":"ContainerDied","Data":"bf1ceefed9032fa43624ee07b1b652d4de907b5529cdd272c1250405a286b1cb"} Oct 30 05:27:05.823382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.819241 6587 scope.go:122] "RemoveContainer" containerID="f288c5770f5a090499c2eeb8efb6b895b432a1b459b158167772100e4d55ae78" Oct 30 05:27:05.823382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.819420 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6" Oct 30 05:27:05.846461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.846432 6587 scope.go:122] "RemoveContainer" containerID="f288c5770f5a090499c2eeb8efb6b895b432a1b459b158167772100e4d55ae78" Oct 30 05:27:05.846826 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:27:05.846797 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f288c5770f5a090499c2eeb8efb6b895b432a1b459b158167772100e4d55ae78\": container with ID starting with f288c5770f5a090499c2eeb8efb6b895b432a1b459b158167772100e4d55ae78 not found: ID does not exist" containerID="f288c5770f5a090499c2eeb8efb6b895b432a1b459b158167772100e4d55ae78" Oct 30 05:27:05.846923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.846834 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f288c5770f5a090499c2eeb8efb6b895b432a1b459b158167772100e4d55ae78"} err="failed to get container status \"f288c5770f5a090499c2eeb8efb6b895b432a1b459b158167772100e4d55ae78\": rpc error: code = NotFound desc = could not find container \"f288c5770f5a090499c2eeb8efb6b895b432a1b459b158167772100e4d55ae78\": container with ID starting with f288c5770f5a090499c2eeb8efb6b895b432a1b459b158167772100e4d55ae78 not found: ID does not exist" Oct 30 05:27:05.891818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.891767 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6"] Oct 30 05:27:05.899331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:05.899294 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34722e/auth-test-model-34722e-kserve-764b8744bf-qn5q6"] Oct 30 05:27:07.592221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:07.592168 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0fd7250-35d9-482a-a71b-2be4c8c3ad6e" path="/var/lib/kubelet/pods/d0fd7250-35d9-482a-a71b-2be4c8c3ad6e/volumes" Oct 30 05:27:10.670974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.670921 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq"] Oct 30 05:27:10.675760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.675734 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d0fd7250-35d9-482a-a71b-2be4c8c3ad6e" containerName="main" Oct 30 05:27:10.675933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.675919 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fd7250-35d9-482a-a71b-2be4c8c3ad6e" containerName="main" Oct 30 05:27:10.676036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.676024 6587 cpu_manager.go:414] "containerMap: removing container" podUID="57233b00-5af2-4532-a38b-5870d0181606" containerName="payload-processing" Oct 30 05:27:10.676125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.676113 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57233b00-5af2-4532-a38b-5870d0181606" containerName="payload-processing" Oct 30 05:27:10.676215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.676205 6587 cpu_manager.go:414] "containerMap: removing container" podUID="8dfc61e5-0c2d-44a8-9398-b35decd958dd" containerName="revoke-keys" Oct 30 05:27:10.676298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.676288 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dfc61e5-0c2d-44a8-9398-b35decd958dd" containerName="revoke-keys" Oct 30 05:27:10.676477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.676460 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fc5faab8-cb81-4304-8715-f3909474e267" containerName="payload-pre-processing" Oct 30 05:27:10.676597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.676583 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5faab8-cb81-4304-8715-f3909474e267" containerName="payload-pre-processing" Oct 30 05:27:10.676938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.676919 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0fd7250-35d9-482a-a71b-2be4c8c3ad6e" containerName="main" Oct 30 05:27:10.677061 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.677049 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="57233b00-5af2-4532-a38b-5870d0181606" containerName="payload-processing" Oct 30 05:27:10.677156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.677146 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dfc61e5-0c2d-44a8-9398-b35decd958dd" containerName="revoke-keys" Oct 30 05:27:10.677253 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.677243 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5faab8-cb81-4304-8715-f3909474e267" containerName="payload-pre-processing" Oct 30 05:27:10.681707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.681678 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" Oct 30 05:27:10.689806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.689782 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq"] Oct 30 05:27:10.700280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.700255 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:27:10.714057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.714034 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:27:10.714184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.714155 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:27:10.714285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.714270 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:27:10.804118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.804077 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csfkc\" (UniqueName: \"kubernetes.io/projected/b888b790-c2be-440a-bc24-b9a7dc7d3509-kube-api-access-csfkc\") pod \"maas-api-revoke-sub-af9c958d-qgdcq\" (UID: \"b888b790-c2be-440a-bc24-b9a7dc7d3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" Oct 30 05:27:10.804325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.804132 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af9c958d-qgdcq\" (UID: \"b888b790-c2be-440a-bc24-b9a7dc7d3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" Oct 30 05:27:10.804469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.804444 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af9c958d-qgdcq\" (UID: \"b888b790-c2be-440a-bc24-b9a7dc7d3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" Oct 30 05:27:10.906194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.906023 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af9c958d-qgdcq\" (UID: \"b888b790-c2be-440a-bc24-b9a7dc7d3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" Oct 30 05:27:10.906194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.906173 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csfkc\" (UniqueName: \"kubernetes.io/projected/b888b790-c2be-440a-bc24-b9a7dc7d3509-kube-api-access-csfkc\") pod \"maas-api-revoke-sub-af9c958d-qgdcq\" (UID: \"b888b790-c2be-440a-bc24-b9a7dc7d3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" Oct 30 05:27:10.906490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.906212 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af9c958d-qgdcq\" (UID: \"b888b790-c2be-440a-bc24-b9a7dc7d3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" Oct 30 05:27:10.906852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.906828 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af9c958d-qgdcq\" (UID: \"b888b790-c2be-440a-bc24-b9a7dc7d3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" Oct 30 05:27:10.908755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.908732 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af9c958d-qgdcq\" (UID: \"b888b790-c2be-440a-bc24-b9a7dc7d3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" Oct 30 05:27:10.957432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.949131 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csfkc\" (UniqueName: \"kubernetes.io/projected/b888b790-c2be-440a-bc24-b9a7dc7d3509-kube-api-access-csfkc\") pod \"maas-api-revoke-sub-af9c958d-qgdcq\" (UID: \"b888b790-c2be-440a-bc24-b9a7dc7d3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" Oct 30 05:27:10.999122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:10.999061 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" Oct 30 05:27:11.789094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:11.789057 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq"] Oct 30 05:27:12.107641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:12.107601 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" event={"ID":"b888b790-c2be-440a-bc24-b9a7dc7d3509","Type":"ContainerStarted","Data":"7681a5bf73d43bee44a5fbc1ed6d80fed0884f1188d7591ab8ae4bd76dff9ea3"} Oct 30 05:27:13.016435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.002763 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv"] Oct 30 05:27:13.024644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.024613 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv"] Oct 30 05:27:13.025003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.024927 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" Oct 30 05:27:13.065688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.065649 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-60a531\"" Oct 30 05:27:13.065881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.065770 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-60a531-dockercfg-8rsmg\"" Oct 30 05:27:13.113665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.113630 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv\" (UID: \"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" Oct 30 05:27:13.113816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.113687 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xqqc\" (UniqueName: \"kubernetes.io/projected/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-kube-api-access-6xqqc\") pod \"payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv\" (UID: \"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" Oct 30 05:27:13.125214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.125140 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" event={"ID":"b888b790-c2be-440a-bc24-b9a7dc7d3509","Type":"ContainerStarted","Data":"2a88aba419aaaaeec25cd83733da58d911aa9c139ae44f085c5369e2b6b717fc"} Oct 30 05:27:13.177791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.177727 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" podStartSLOduration=3.17771067 podStartE2EDuration="3.17771067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:13.174035608 +0000 UTC m=+2557.701802395" watchObservedRunningTime="2026-10-30 05:27:13.17771067 +0000 UTC m=+2557.705477455" Oct 30 05:27:13.221864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.215496 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv\" (UID: \"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" Oct 30 05:27:13.221864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.215582 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xqqc\" (UniqueName: \"kubernetes.io/projected/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-kube-api-access-6xqqc\") pod \"payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv\" (UID: \"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" Oct 30 05:27:13.221864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.216338 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv\" (UID: \"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" Oct 30 05:27:13.242372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.242316 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xqqc\" (UniqueName: \"kubernetes.io/projected/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-kube-api-access-6xqqc\") pod \"payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv\" (UID: \"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" Oct 30 05:27:13.368409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.368374 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" Oct 30 05:27:13.715059 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:27:13.715008 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9396c9bf_5ba4_4baf_accb_f63fd7f0d69a.slice/crio-78d17c83664e4621dc09bbba23023349cec25a44904f5a5b650e8e287a130916 WatchSource:0}: Error finding container 78d17c83664e4621dc09bbba23023349cec25a44904f5a5b650e8e287a130916: Status 404 returned error can't find the container with id 78d17c83664e4621dc09bbba23023349cec25a44904f5a5b650e8e287a130916 Oct 30 05:27:13.720666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:13.716739 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv"] Oct 30 05:27:14.150113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:14.149145 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" event={"ID":"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a","Type":"ContainerStarted","Data":"0f7a40811e85d5a066e68586bd083fa443ea710d08a1529cde50ff955365ac66"} Oct 30 05:27:14.150113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:14.149231 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" event={"ID":"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a","Type":"ContainerStarted","Data":"78d17c83664e4621dc09bbba23023349cec25a44904f5a5b650e8e287a130916"} Oct 30 05:27:14.150113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:14.149327 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" Oct 30 05:27:14.152503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:14.152477 6587 generic.go:356] "Generic (PLEG): container finished" podID="b888b790-c2be-440a-bc24-b9a7dc7d3509" containerID="2a88aba419aaaaeec25cd83733da58d911aa9c139ae44f085c5369e2b6b717fc" exitCode=0 Oct 30 05:27:14.152760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:14.152620 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" event={"ID":"b888b790-c2be-440a-bc24-b9a7dc7d3509","Type":"ContainerDied","Data":"2a88aba419aaaaeec25cd83733da58d911aa9c139ae44f085c5369e2b6b717fc"} Oct 30 05:27:14.195253 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:14.195199 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" podStartSLOduration=2.195184419 podStartE2EDuration="2.195184419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:14.192317672 +0000 UTC m=+2558.720084461" watchObservedRunningTime="2026-10-30 05:27:14.195184419 +0000 UTC m=+2558.722951205" Oct 30 05:27:14.863737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:14.863495 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:27:14.863737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:14.863571 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:27:15.106523 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:15.106467 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r"] Oct 30 05:27:15.633481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:15.633450 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" Oct 30 05:27:15.673038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:15.673003 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b888b790-c2be-440a-bc24-b9a7dc7d3509-kube-api-access-csfkc\" (UniqueName: \"kubernetes.io/projected/b888b790-c2be-440a-bc24-b9a7dc7d3509-kube-api-access-csfkc\") pod \"b888b790-c2be-440a-bc24-b9a7dc7d3509\" (UID: \"b888b790-c2be-440a-bc24-b9a7dc7d3509\") " Oct 30 05:27:15.673214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:15.673171 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-openshift-ca\") pod \"b888b790-c2be-440a-bc24-b9a7dc7d3509\" (UID: \"b888b790-c2be-440a-bc24-b9a7dc7d3509\") " Oct 30 05:27:15.673384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:15.673350 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-xks-ca\") pod \"b888b790-c2be-440a-bc24-b9a7dc7d3509\" (UID: \"b888b790-c2be-440a-bc24-b9a7dc7d3509\") " Oct 30 05:27:15.674209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:15.674176 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-openshift-ca" pod "b888b790-c2be-440a-bc24-b9a7dc7d3509" (UID: "b888b790-c2be-440a-bc24-b9a7dc7d3509"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:27:15.675213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:15.675186 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:15.676256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:15.676232 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b888b790-c2be-440a-bc24-b9a7dc7d3509-kube-api-access-csfkc" pod "b888b790-c2be-440a-bc24-b9a7dc7d3509" (UID: "b888b790-c2be-440a-bc24-b9a7dc7d3509"). InnerVolumeSpecName "kube-api-access-csfkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:27:15.684764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:15.684732 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-xks-ca" pod "b888b790-c2be-440a-bc24-b9a7dc7d3509" (UID: "b888b790-c2be-440a-bc24-b9a7dc7d3509"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:27:15.776523 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:15.776492 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b888b790-c2be-440a-bc24-b9a7dc7d3509-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:15.776523 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:15.776519 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csfkc\" (UniqueName: \"kubernetes.io/projected/b888b790-c2be-440a-bc24-b9a7dc7d3509-kube-api-access-csfkc\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:16.182944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:16.182895 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" event={"ID":"b888b790-c2be-440a-bc24-b9a7dc7d3509","Type":"ContainerDied","Data":"7681a5bf73d43bee44a5fbc1ed6d80fed0884f1188d7591ab8ae4bd76dff9ea3"} Oct 30 05:27:16.183263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:16.182980 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq" Oct 30 05:27:16.183263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:16.183002 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7681a5bf73d43bee44a5fbc1ed6d80fed0884f1188d7591ab8ae4bd76dff9ea3" Oct 30 05:27:19.241533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.241488 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd"] Oct 30 05:27:19.275136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.275095 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b888b790-c2be-440a-bc24-b9a7dc7d3509" containerName="revoke-keys" Oct 30 05:27:19.275136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.275125 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b888b790-c2be-440a-bc24-b9a7dc7d3509" containerName="revoke-keys" Oct 30 05:27:19.275407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.275386 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b888b790-c2be-440a-bc24-b9a7dc7d3509" containerName="revoke-keys" Oct 30 05:27:19.511195 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.511106 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd"] Oct 30 05:27:19.511398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.511214 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.575469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.575425 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.575892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.575862 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8qb\" (UniqueName: \"kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-kube-api-access-qn8qb\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.576104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.576083 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.576212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.576148 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.576280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.576258 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.576340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.576288 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.576451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.576422 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.576586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.576559 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.576674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.576639 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.678134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.678096 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.678334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.678142 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.678334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.678188 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.678334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.678263 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.678334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.678310 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.678602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.678458 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.678602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.678562 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8qb\" (UniqueName: \"kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-kube-api-access-qn8qb\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.678714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.678614 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.678714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.678665 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.678714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.678674 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.678891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.678723 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.678891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.678696 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.679009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.678969 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.679226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.679206 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.694115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.694080 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.694291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.694269 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.699869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.699840 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.699988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.699953 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8qb\" (UniqueName: \"kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-kube-api-access-qn8qb\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:19.834488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:19.834445 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:21.022519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.022473 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv"] Oct 30 05:27:21.221516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.221479 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv"] Oct 30 05:27:21.221732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.221603 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" Oct 30 05:27:21.238764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.238735 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:27:21.238934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.238789 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:27:21.252351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.252319 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:27:21.252351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.252334 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:27:21.311341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.311306 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftxz8\" (UniqueName: \"kubernetes.io/projected/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-kube-api-access-ftxz8\") pod \"maas-api-revoke-sub-785e362b-6gbsv\" (UID: \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" Oct 30 05:27:21.311579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.311408 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-785e362b-6gbsv\" (UID: \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" Oct 30 05:27:21.311647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.311568 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-785e362b-6gbsv\" (UID: \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" Oct 30 05:27:21.413591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.413554 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftxz8\" (UniqueName: \"kubernetes.io/projected/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-kube-api-access-ftxz8\") pod \"maas-api-revoke-sub-785e362b-6gbsv\" (UID: \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" Oct 30 05:27:21.413775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.413611 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-785e362b-6gbsv\" (UID: \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" Oct 30 05:27:21.413775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.413676 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-785e362b-6gbsv\" (UID: \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" Oct 30 05:27:21.414546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.414521 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-785e362b-6gbsv\" (UID: \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" Oct 30 05:27:21.416555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.416521 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-785e362b-6gbsv\" (UID: \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" Oct 30 05:27:21.435094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.435065 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftxz8\" (UniqueName: \"kubernetes.io/projected/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-kube-api-access-ftxz8\") pod \"maas-api-revoke-sub-785e362b-6gbsv\" (UID: \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" Oct 30 05:27:21.466459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.466425 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" Oct 30 05:27:21.851156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.851114 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:21.851285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.851213 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:21.851285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.851257 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:21.854692 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.854651 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd"] Oct 30 05:27:21.870335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:21.870308 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv"] Oct 30 05:27:21.875632 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:27:21.875592 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a5d829_ba9b_4533_9ce0_003d04c3e1d5.slice/crio-e59799b2fbb6958283b2334867b2551f958b68176408066fdef1d8b58af22a15 WatchSource:0}: Error finding container e59799b2fbb6958283b2334867b2551f958b68176408066fdef1d8b58af22a15: Status 404 returned error can't find the container with id e59799b2fbb6958283b2334867b2551f958b68176408066fdef1d8b58af22a15 Oct 30 05:27:22.303232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:22.303199 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" event={"ID":"fdb49b21-bb35-4f43-8ddd-2a8104a04509","Type":"ContainerStarted","Data":"dc48d6602fca7c1aaa44b533a74741ca899c766c1e1fde737603d731f8456539"} Oct 30 05:27:22.303698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:22.303258 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" event={"ID":"fdb49b21-bb35-4f43-8ddd-2a8104a04509","Type":"ContainerStarted","Data":"d764f859b41bcf476256360a566b95f3e9e053b40c74d5257c6d958a998ba1ea"} Oct 30 05:27:22.304729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:22.304703 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" event={"ID":"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5","Type":"ContainerStarted","Data":"e59799b2fbb6958283b2334867b2551f958b68176408066fdef1d8b58af22a15"} Oct 30 05:27:22.369537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:22.369497 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:27:22.369712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:22.369553 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:27:22.372845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:22.372791 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" podStartSLOduration=3.372773198 podStartE2EDuration="3.372773198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:22.365695454 +0000 UTC m=+2566.893462241" watchObservedRunningTime="2026-10-30 05:27:22.372773198 +0000 UTC m=+2566.900539985" Oct 30 05:27:22.834911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:22.834872 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:22.837129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:22.837098 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Oct 30 05:27:22.837270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:22.837155 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" podUID="fdb49b21-bb35-4f43-8ddd-2a8104a04509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Oct 30 05:27:23.324388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:23.322969 6587 generic.go:356] "Generic (PLEG): container finished" podID="b7a5d829-ba9b-4533-9ce0-003d04c3e1d5" containerID="966d35b84119d5597bf8402afb586458fe9aad99f91500da47a28753fb30c258" exitCode=0 Oct 30 05:27:23.335030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:23.327141 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" event={"ID":"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5","Type":"ContainerDied","Data":"966d35b84119d5597bf8402afb586458fe9aad99f91500da47a28753fb30c258"} Oct 30 05:27:23.846383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:23.837813 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Oct 30 05:27:23.846383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:23.837878 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" podUID="fdb49b21-bb35-4f43-8ddd-2a8104a04509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Oct 30 05:27:24.052876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:24.052806 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4"] Oct 30 05:27:24.061199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:24.061159 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53e0a2e-ztcj4"] Oct 30 05:27:24.843393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:24.841475 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Oct 30 05:27:24.843393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:24.841540 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" podUID="fdb49b21-bb35-4f43-8ddd-2a8104a04509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Oct 30 05:27:24.881389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:24.872532 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" Oct 30 05:27:24.959432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:24.959177 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-xks-ca\") pod \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\" (UID: \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\") " Oct 30 05:27:24.959432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:24.959260 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-openshift-ca\") pod \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\" (UID: \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\") " Oct 30 05:27:24.959432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:24.959418 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-kube-api-access-ftxz8\" (UniqueName: \"kubernetes.io/projected/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-kube-api-access-ftxz8\") pod \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\" (UID: \"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5\") " Oct 30 05:27:24.970767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:24.970710 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-kube-api-access-ftxz8" pod "b7a5d829-ba9b-4533-9ce0-003d04c3e1d5" (UID: "b7a5d829-ba9b-4533-9ce0-003d04c3e1d5"). InnerVolumeSpecName "kube-api-access-ftxz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:27:24.971412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:24.971333 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-openshift-ca" pod "b7a5d829-ba9b-4533-9ce0-003d04c3e1d5" (UID: "b7a5d829-ba9b-4533-9ce0-003d04c3e1d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:27:24.971973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:24.971940 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-xks-ca" pod "b7a5d829-ba9b-4533-9ce0-003d04c3e1d5" (UID: "b7a5d829-ba9b-4533-9ce0-003d04c3e1d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:27:25.062376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:25.062006 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:25.062376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:25.062045 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:25.062376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:25.062063 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftxz8\" (UniqueName: \"kubernetes.io/projected/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5-kube-api-access-ftxz8\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:25.169474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:25.169423 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" Oct 30 05:27:25.406038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:25.406009 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" Oct 30 05:27:25.406038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:25.406027 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv" event={"ID":"b7a5d829-ba9b-4533-9ce0-003d04c3e1d5","Type":"ContainerDied","Data":"e59799b2fbb6958283b2334867b2551f958b68176408066fdef1d8b58af22a15"} Oct 30 05:27:25.406302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:25.406151 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e59799b2fbb6958283b2334867b2551f958b68176408066fdef1d8b58af22a15" Oct 30 05:27:25.588581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:25.588541 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b567ca-fe72-4ebe-a47a-4df8699bbc55" path="/var/lib/kubelet/pods/b8b567ca-fe72-4ebe-a47a-4df8699bbc55/volumes" Oct 30 05:27:25.855384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:25.836754 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Oct 30 05:27:25.855384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:25.836840 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" podUID="fdb49b21-bb35-4f43-8ddd-2a8104a04509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Oct 30 05:27:26.672053 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:27:26.672007 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:27:26.809393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:26.806338 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz"] Oct 30 05:27:26.809393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:26.807624 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b7a5d829-ba9b-4533-9ce0-003d04c3e1d5" containerName="revoke-keys" Oct 30 05:27:26.809393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:26.807647 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a5d829-ba9b-4533-9ce0-003d04c3e1d5" containerName="revoke-keys" Oct 30 05:27:26.809393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:26.807865 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a5d829-ba9b-4533-9ce0-003d04c3e1d5" containerName="revoke-keys" Oct 30 05:27:26.846379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:26.846191 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Oct 30 05:27:26.846379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:26.846252 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" podUID="fdb49b21-bb35-4f43-8ddd-2a8104a04509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Oct 30 05:27:26.863762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:26.863615 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz"] Oct 30 05:27:26.864340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:26.863736 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.008805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.008722 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.008805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.008765 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.008805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.008799 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.009191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.008864 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.009191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.008902 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ddbb1e-572b-46df-a64e-2b8e966fe680-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.009191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.008940 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.009191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.008975 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.009191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.009021 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.009191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.009106 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bttt\" (UniqueName: \"kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-kube-api-access-7bttt\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.111106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.111065 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bttt\" (UniqueName: \"kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-kube-api-access-7bttt\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.111323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.111230 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.111323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.111277 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.111473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.111321 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.111473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.111459 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.111595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.111534 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ddbb1e-572b-46df-a64e-2b8e966fe680-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.111655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.111608 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.111715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.111678 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.111777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.111719 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.111777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.111736 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.112010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.111981 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.112150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.112078 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.112228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.112151 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.112502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.112474 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ddbb1e-572b-46df-a64e-2b8e966fe680-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.114725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.114689 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.114986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.114971 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.151381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.136990 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.151381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.138346 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bttt\" (UniqueName: \"kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-kube-api-access-7bttt\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.200121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.200091 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:27.770297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.770263 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz"] Oct 30 05:27:27.771084 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:27:27.771046 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ddbb1e_572b_46df_a64e_2b8e966fe680.slice/crio-a2575ae4d7ee5c240801a9191826c59d90116c75889f2fa51d4ebca5980f47a9 WatchSource:0}: Error finding container a2575ae4d7ee5c240801a9191826c59d90116c75889f2fa51d4ebca5980f47a9: Status 404 returned error can't find the container with id a2575ae4d7ee5c240801a9191826c59d90116c75889f2fa51d4ebca5980f47a9 Oct 30 05:27:27.778516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.778011 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:27.778516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.778104 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:27.778516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.778147 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:27.839165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.838717 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Oct 30 05:27:27.839165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:27.838777 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" podUID="fdb49b21-bb35-4f43-8ddd-2a8104a04509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Oct 30 05:27:28.493458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:28.491584 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" event={"ID":"10ddbb1e-572b-46df-a64e-2b8e966fe680","Type":"ContainerStarted","Data":"0f73de0764fda050572c150e3ca627d6763120b900bb48e77832f43be38ec0f4"} Oct 30 05:27:28.493458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:28.491690 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" event={"ID":"10ddbb1e-572b-46df-a64e-2b8e966fe680","Type":"ContainerStarted","Data":"a2575ae4d7ee5c240801a9191826c59d90116c75889f2fa51d4ebca5980f47a9"} Oct 30 05:27:28.899114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:28.899073 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:28.899679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:28.899605 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:28.900907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:28.900884 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:27:28.949481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:28.949428 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" podStartSLOduration=2.949413958 podStartE2EDuration="2.949413958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:28.569186208 +0000 UTC m=+2573.096952994" watchObservedRunningTime="2026-10-30 05:27:28.949413958 +0000 UTC m=+2573.477180744" Oct 30 05:27:29.202949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:29.202858 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:29.207996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:29.207639 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Oct 30 05:27:29.207996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:29.207717 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" podUID="10ddbb1e-572b-46df-a64e-2b8e966fe680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Oct 30 05:27:30.068274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:30.067520 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:27:30.068274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:30.067599 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:27:30.216158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:30.216111 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Oct 30 05:27:30.216497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:30.216462 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" podUID="10ddbb1e-572b-46df-a64e-2b8e966fe680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Oct 30 05:27:30.555817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:30.555765 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" podUID="de134eba-c93e-4e36-b6bc-4a7dbe2c3685" containerName="main" containerID="cri-o://d68a92d32fb5afd2703261a5402f4b61b5ff478fdd0f8ec583c19ee5476b2369" gracePeriod=45 Oct 30 05:27:31.200819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.200741 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Oct 30 05:27:31.200819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.200798 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" podUID="10ddbb1e-572b-46df-a64e-2b8e966fe680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Oct 30 05:27:31.304421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.304393 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:27:31.376906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.376877 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-model-cache\") pod \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " Oct 30 05:27:31.377072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.376983 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-kube-api-access-pp9rf\" (UniqueName: \"kubernetes.io/projected/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-kube-api-access-pp9rf\") pod \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " Oct 30 05:27:31.377072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.377055 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-dshm\") pod \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " Oct 30 05:27:31.377192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.377125 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tls-certs\" (UniqueName: \"kubernetes.io/secret/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tls-certs\") pod \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " Oct 30 05:27:31.377192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.377170 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-home\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-home\") pod \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " Oct 30 05:27:31.377305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.377235 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tmp-dir\") pod \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\" (UID: \"de134eba-c93e-4e36-b6bc-4a7dbe2c3685\") " Oct 30 05:27:31.378221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.377999 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-model-cache" pod "de134eba-c93e-4e36-b6bc-4a7dbe2c3685" (UID: "de134eba-c93e-4e36-b6bc-4a7dbe2c3685"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:27:31.378629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.378612 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-home" pod "de134eba-c93e-4e36-b6bc-4a7dbe2c3685" (UID: "de134eba-c93e-4e36-b6bc-4a7dbe2c3685"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:27:31.378949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.378925 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tmp-dir" pod "de134eba-c93e-4e36-b6bc-4a7dbe2c3685" (UID: "de134eba-c93e-4e36-b6bc-4a7dbe2c3685"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:27:31.380278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.380255 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:31.380438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.380424 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:31.380579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.380564 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:31.391010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.390966 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tls-certs" pod "de134eba-c93e-4e36-b6bc-4a7dbe2c3685" (UID: "de134eba-c93e-4e36-b6bc-4a7dbe2c3685"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:27:31.401401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.401284 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-kube-api-access-pp9rf" pod "de134eba-c93e-4e36-b6bc-4a7dbe2c3685" (UID: "de134eba-c93e-4e36-b6bc-4a7dbe2c3685"). InnerVolumeSpecName "kube-api-access-pp9rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:27:31.405402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.405381 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-dshm" pod "de134eba-c93e-4e36-b6bc-4a7dbe2c3685" (UID: "de134eba-c93e-4e36-b6bc-4a7dbe2c3685"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:27:31.481990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.481952 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp9rf\" (UniqueName: \"kubernetes.io/projected/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-kube-api-access-pp9rf\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:31.481990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.481987 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:31.481990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.481999 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de134eba-c93e-4e36-b6bc-4a7dbe2c3685-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:31.558384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.552834 6587 generic.go:356] "Generic (PLEG): container finished" podID="f131a03e-c992-40b1-a6e6-86d29073205b" containerID="6512ca1ab0e81c9d97c1839699c9f2e89cce37f7685c36c414d15c8cf626f5c1" exitCode=0 Oct 30 05:27:31.558384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.553290 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" event={"ID":"f131a03e-c992-40b1-a6e6-86d29073205b","Type":"ContainerDied","Data":"6512ca1ab0e81c9d97c1839699c9f2e89cce37f7685c36c414d15c8cf626f5c1"} Oct 30 05:27:31.558384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.554120 6587 scope.go:122] "RemoveContainer" containerID="6512ca1ab0e81c9d97c1839699c9f2e89cce37f7685c36c414d15c8cf626f5c1" Oct 30 05:27:31.563516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.563480 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:31.563660 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.563558 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:31.563660 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.563591 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:31.570790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.570764 6587 generic.go:356] "Generic (PLEG): container finished" podID="de134eba-c93e-4e36-b6bc-4a7dbe2c3685" containerID="d68a92d32fb5afd2703261a5402f4b61b5ff478fdd0f8ec583c19ee5476b2369" exitCode=0 Oct 30 05:27:31.570920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.570878 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" event={"ID":"de134eba-c93e-4e36-b6bc-4a7dbe2c3685","Type":"ContainerDied","Data":"d68a92d32fb5afd2703261a5402f4b61b5ff478fdd0f8ec583c19ee5476b2369"} Oct 30 05:27:31.570920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.570909 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" event={"ID":"de134eba-c93e-4e36-b6bc-4a7dbe2c3685","Type":"ContainerDied","Data":"a4ace8868c1ec32d6355cf39b2b0f01177a1747fb61fe6627dc0babcbdd9d295"} Oct 30 05:27:31.571051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.570930 6587 scope.go:122] "RemoveContainer" containerID="d68a92d32fb5afd2703261a5402f4b61b5ff478fdd0f8ec583c19ee5476b2369" Oct 30 05:27:31.571114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.571097 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" Oct 30 05:27:31.664963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.664410 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r"] Oct 30 05:27:31.669025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.667705 6587 scope.go:122] "RemoveContainer" containerID="d68a92d32fb5afd2703261a5402f4b61b5ff478fdd0f8ec583c19ee5476b2369" Oct 30 05:27:31.669025 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:27:31.668117 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68a92d32fb5afd2703261a5402f4b61b5ff478fdd0f8ec583c19ee5476b2369\": container with ID starting with d68a92d32fb5afd2703261a5402f4b61b5ff478fdd0f8ec583c19ee5476b2369 not found: ID does not exist" containerID="d68a92d32fb5afd2703261a5402f4b61b5ff478fdd0f8ec583c19ee5476b2369" Oct 30 05:27:31.669025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.668154 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68a92d32fb5afd2703261a5402f4b61b5ff478fdd0f8ec583c19ee5476b2369"} err="failed to get container status \"d68a92d32fb5afd2703261a5402f4b61b5ff478fdd0f8ec583c19ee5476b2369\": rpc error: code = NotFound desc = could not find container \"d68a92d32fb5afd2703261a5402f4b61b5ff478fdd0f8ec583c19ee5476b2369\": container with ID starting with d68a92d32fb5afd2703261a5402f4b61b5ff478fdd0f8ec583c19ee5476b2369 not found: ID does not exist" Oct 30 05:27:31.672887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:31.671921 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r"] Oct 30 05:27:32.179313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:32.179222 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-0dba31/auth-test-model-0dba31-kserve-bf4884c8f-9ng6r" podUID="de134eba-c93e-4e36-b6bc-4a7dbe2c3685" containerName="main" probeResult="failure" output="Get \"https://10.134.1.16:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:27:32.201220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:32.201186 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Oct 30 05:27:32.201742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:32.201246 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" podUID="10ddbb1e-572b-46df-a64e-2b8e966fe680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Oct 30 05:27:32.605379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:32.605205 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" event={"ID":"f131a03e-c992-40b1-a6e6-86d29073205b","Type":"ContainerStarted","Data":"4ff405235d16322288fb6bc325830e7fc88a00682afe4462c15cebe05bc130fc"} Oct 30 05:27:33.068382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.067441 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:27:33.076545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.076505 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 05:27:33.076717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.076574 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 05:27:33.201149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.201052 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Oct 30 05:27:33.201149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.201117 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" podUID="10ddbb1e-572b-46df-a64e-2b8e966fe680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Oct 30 05:27:33.483549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.483382 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd"] Oct 30 05:27:33.484493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.484468 6587 cpu_manager.go:414] "containerMap: removing container" podUID="de134eba-c93e-4e36-b6bc-4a7dbe2c3685" containerName="main" Oct 30 05:27:33.484801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.484780 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de134eba-c93e-4e36-b6bc-4a7dbe2c3685" containerName="main" Oct 30 05:27:33.485165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.485143 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="de134eba-c93e-4e36-b6bc-4a7dbe2c3685" containerName="main" Oct 30 05:27:33.578159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.578127 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd"] Oct 30 05:27:33.578589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.578543 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:33.851441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.689043 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-9dc17a-openshift-default-dockercfg-k442k\"" Oct 30 05:27:33.865252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.853152 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de134eba-c93e-4e36-b6bc-4a7dbe2c3685" path="/var/lib/kubelet/pods/de134eba-c93e-4e36-b6bc-4a7dbe2c3685/volumes" Oct 30 05:27:33.865252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.854713 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:33.865252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.854813 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:33.865252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.854845 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:33.865252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.854900 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:33.865252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.854956 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:33.865252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.854981 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c2xb\" (UniqueName: \"kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-kube-api-access-2c2xb\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:33.865252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.855024 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:33.865252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.855127 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:33.865252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:33.855205 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.007251 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.007311 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.007344 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c2xb\" (UniqueName: \"kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-kube-api-access-2c2xb\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.007505 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.007659 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.007773 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.007817 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.007979 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.008020 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.009302 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.009707 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.010599 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.010856 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.023889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.011216 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.034924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.032274 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.068436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.049054 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.073200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.072134 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c2xb\" (UniqueName: \"kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-kube-api-access-2c2xb\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.073200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.072258 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 05:27:34.073200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.072304 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 05:27:34.077186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.077114 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:34.082455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.082419 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s"] Oct 30 05:27:34.092413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.091669 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f"] Oct 30 05:27:34.097607 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.097568 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3b14f40-8zw8f"] Oct 30 05:27:34.113300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.113201 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" containerID="cri-o://d6b099e09fee607d4fc2f46adcafa58f1d9766c952269c0be47116710649b530" gracePeriod=30 Oct 30 05:27:34.202088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.202058 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Oct 30 05:27:34.202473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.202115 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" podUID="10ddbb1e-572b-46df-a64e-2b8e966fe680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Oct 30 05:27:34.338382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:34.334967 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:35.080033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:35.070610 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 05:27:35.080033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:35.070671 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 05:27:35.080033 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:27:35.079970 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9e5f93_4df2_4f1d_90a7_5e5e5ddf22ce.slice/crio-bf21dc9579e27b1e916fe70d05d94aa6194118d050bd18ee389085f1dbff67b1 WatchSource:0}: Error finding container bf21dc9579e27b1e916fe70d05d94aa6194118d050bd18ee389085f1dbff67b1: Status 404 returned error can't find the container with id bf21dc9579e27b1e916fe70d05d94aa6194118d050bd18ee389085f1dbff67b1 Oct 30 05:27:35.081141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:35.081106 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd"] Oct 30 05:27:35.084311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:35.084273 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:35.084449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:35.084381 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:35.084449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:35.084427 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:35.214553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:35.214515 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Oct 30 05:27:35.215031 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:35.214586 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" podUID="10ddbb1e-572b-46df-a64e-2b8e966fe680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Oct 30 05:27:35.294273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:35.294236 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" event={"ID":"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce","Type":"ContainerStarted","Data":"bf21dc9579e27b1e916fe70d05d94aa6194118d050bd18ee389085f1dbff67b1"} Oct 30 05:27:35.613648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:35.613140 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4605208-cb14-449a-b590-132cdb64945e" path="/var/lib/kubelet/pods/b4605208-cb14-449a-b590-132cdb64945e/volumes" Oct 30 05:27:36.065377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.064594 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 05:27:36.065377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.064659 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 05:27:36.206760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.206723 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:36.207094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.207070 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:36.208038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.208018 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:27:36.306449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.306417 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" event={"ID":"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce","Type":"ContainerStarted","Data":"74a7eb4b2804ef7810b12a38693e254f7ba3c99d867fec953b2e2ac263e973ca"} Oct 30 05:27:36.308855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.308799 6587 generic.go:356] "Generic (PLEG): container finished" podID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerID="c3936e4be7a46060b4f86bd849a2f18c5bf415c3bc4e7dc9df76e8c6482bbf4a" exitCode=0 Oct 30 05:27:36.309057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.308854 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" event={"ID":"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a","Type":"ContainerDied","Data":"c3936e4be7a46060b4f86bd849a2f18c5bf415c3bc4e7dc9df76e8c6482bbf4a"} Oct 30 05:27:36.309413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.309388 6587 scope.go:122] "RemoveContainer" containerID="c3936e4be7a46060b4f86bd849a2f18c5bf415c3bc4e7dc9df76e8c6482bbf4a" Oct 30 05:27:36.311567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.311534 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:36.311648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.311618 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:36.311703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.311664 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:36.335541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.334451 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:27:36.336053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.336000 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:36.359422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.359393 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:36.443404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.443319 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" podStartSLOduration=3.443299862 podStartE2EDuration="3.443299862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:36.427431457 +0000 UTC m=+2580.955198245" watchObservedRunningTime="2026-10-30 05:27:36.443299862 +0000 UTC m=+2580.971066648" Oct 30 05:27:36.449271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.443791 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w"] Oct 30 05:27:36.450389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.450347 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" Oct 30 05:27:36.478972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.478109 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ec92c7\"" Oct 30 05:27:36.483492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.483043 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w"] Oct 30 05:27:36.541009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.540976 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm"] Oct 30 05:27:36.561380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.559144 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ec92c7-dockercfg-n98rg\"" Oct 30 05:27:36.561380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.561139 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5b2919-e724-4011-824d-b6f83bd67b83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w\" (UID: \"de5b2919-e724-4011-824d-b6f83bd67b83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" Oct 30 05:27:36.561380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.561179 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8rc\" (UniqueName: \"kubernetes.io/projected/de5b2919-e724-4011-824d-b6f83bd67b83-kube-api-access-wf8rc\") pod \"payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w\" (UID: \"de5b2919-e724-4011-824d-b6f83bd67b83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" Oct 30 05:27:36.575317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.575248 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" Oct 30 05:27:36.585796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.585757 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm"] Oct 30 05:27:36.946439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.939580 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm\" (UID: \"d67d5b5e-c62d-4024-b9ab-3a6b501acf89\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" Oct 30 05:27:36.946439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.939661 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq44v\" (UniqueName: \"kubernetes.io/projected/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-kube-api-access-kq44v\") pod \"payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm\" (UID: \"d67d5b5e-c62d-4024-b9ab-3a6b501acf89\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" Oct 30 05:27:36.946439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.939889 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5b2919-e724-4011-824d-b6f83bd67b83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w\" (UID: \"de5b2919-e724-4011-824d-b6f83bd67b83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" Oct 30 05:27:36.946439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.939918 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8rc\" (UniqueName: \"kubernetes.io/projected/de5b2919-e724-4011-824d-b6f83bd67b83-kube-api-access-wf8rc\") pod \"payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w\" (UID: \"de5b2919-e724-4011-824d-b6f83bd67b83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" Oct 30 05:27:36.963382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.952146 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5b2919-e724-4011-824d-b6f83bd67b83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w\" (UID: \"de5b2919-e724-4011-824d-b6f83bd67b83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" Oct 30 05:27:36.987386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:36.986672 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8rc\" (UniqueName: \"kubernetes.io/projected/de5b2919-e724-4011-824d-b6f83bd67b83-kube-api-access-wf8rc\") pod \"payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w\" (UID: \"de5b2919-e724-4011-824d-b6f83bd67b83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" Oct 30 05:27:37.045380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.044372 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm\" (UID: \"d67d5b5e-c62d-4024-b9ab-3a6b501acf89\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" Oct 30 05:27:37.045380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.044440 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq44v\" (UniqueName: \"kubernetes.io/projected/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-kube-api-access-kq44v\") pod \"payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm\" (UID: \"d67d5b5e-c62d-4024-b9ab-3a6b501acf89\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" Oct 30 05:27:37.056386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.045788 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm\" (UID: \"d67d5b5e-c62d-4024-b9ab-3a6b501acf89\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" Oct 30 05:27:37.075522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.075477 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 05:27:37.075799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.075750 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 05:27:37.078085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.078061 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq44v\" (UniqueName: \"kubernetes.io/projected/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-kube-api-access-kq44v\") pod \"payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm\" (UID: \"d67d5b5e-c62d-4024-b9ab-3a6b501acf89\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" Oct 30 05:27:37.109990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.109963 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" Oct 30 05:27:37.198534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.198460 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" Oct 30 05:27:37.408517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.388158 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" event={"ID":"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a","Type":"ContainerStarted","Data":"62ed9e0c8d5903e83e05f3186145a82721261c2c668309cdb18537a48a403536"} Oct 30 05:27:37.408517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.389468 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:37.410770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.410739 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:27:37.710072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.709498 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:27:37.711015 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.710965 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body= Oct 30 05:27:37.711167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.711033 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" Oct 30 05:27:37.713661 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:27:37.713625 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5b2919_e724_4011_824d_b6f83bd67b83.slice/crio-9f7448579f9fb104f0c6a5f3901821889725550aa53acccd4a0aeb8f6eb16d65 WatchSource:0}: Error finding container 9f7448579f9fb104f0c6a5f3901821889725550aa53acccd4a0aeb8f6eb16d65: Status 404 returned error can't find the container with id 9f7448579f9fb104f0c6a5f3901821889725550aa53acccd4a0aeb8f6eb16d65 Oct 30 05:27:37.714941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.714600 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w"] Oct 30 05:27:37.891779 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:37.883217 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm"] Oct 30 05:27:38.066394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:38.064947 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 05:27:38.066394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:38.065025 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 05:27:38.397715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:38.397684 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" event={"ID":"d67d5b5e-c62d-4024-b9ab-3a6b501acf89","Type":"ContainerStarted","Data":"0b3162da12f9550ab3d4fe7f169dda0994fc37dc5b427db46d0bd9a4ba6ab5de"} Oct 30 05:27:38.397966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:38.397952 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" event={"ID":"d67d5b5e-c62d-4024-b9ab-3a6b501acf89","Type":"ContainerStarted","Data":"5f36c51723cd0bc675e73ca1287845c5e736015c9373476ddcfd9ca7c0533a7c"} Oct 30 05:27:38.398083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:38.398073 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" Oct 30 05:27:38.399611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:38.399579 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" event={"ID":"de5b2919-e724-4011-824d-b6f83bd67b83","Type":"ContainerStarted","Data":"c406e462d79852fbaeaeccc518ca00e0fbec51988f36d241c4f4bed627061930"} Oct 30 05:27:38.399790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:38.399743 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" event={"ID":"de5b2919-e724-4011-824d-b6f83bd67b83","Type":"ContainerStarted","Data":"9f7448579f9fb104f0c6a5f3901821889725550aa53acccd4a0aeb8f6eb16d65"} Oct 30 05:27:38.399790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:38.399784 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" Oct 30 05:27:38.452383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:38.452132 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" podStartSLOduration=2.45211422 podStartE2EDuration="2.45211422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:38.450873438 +0000 UTC m=+2582.978640225" watchObservedRunningTime="2026-10-30 05:27:38.45211422 +0000 UTC m=+2582.979881009" Oct 30 05:27:38.504493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:38.504446 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" podStartSLOduration=2.504433439 podStartE2EDuration="2.504433439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:38.50071833 +0000 UTC m=+2583.028485117" watchObservedRunningTime="2026-10-30 05:27:38.504433439 +0000 UTC m=+2583.032200225" Oct 30 05:27:38.709283 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:38.709195 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body= Oct 30 05:27:38.709283 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:38.709246 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" Oct 30 05:27:39.081200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.081051 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:27:39.083540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.083510 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:27:39.355713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.355622 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj"] Oct 30 05:27:39.362500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.360947 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" Oct 30 05:27:39.362500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.362274 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj"] Oct 30 05:27:39.378697 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.378482 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-45607d-dockercfg-kw4lv\"" Oct 30 05:27:39.378916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.378880 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-45607d\"" Oct 30 05:27:39.443429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.431033 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67tmq\" (UniqueName: \"kubernetes.io/projected/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-kube-api-access-67tmq\") pod \"payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj\" (UID: \"6ac662ee-f41a-4ffb-8d99-2e2806ec05db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" Oct 30 05:27:39.443429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.431215 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj\" (UID: \"6ac662ee-f41a-4ffb-8d99-2e2806ec05db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" Oct 30 05:27:39.586380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.562102 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67tmq\" (UniqueName: \"kubernetes.io/projected/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-kube-api-access-67tmq\") pod \"payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj\" (UID: \"6ac662ee-f41a-4ffb-8d99-2e2806ec05db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" Oct 30 05:27:39.586380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.562283 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj\" (UID: \"6ac662ee-f41a-4ffb-8d99-2e2806ec05db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" Oct 30 05:27:39.586380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.563576 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj\" (UID: \"6ac662ee-f41a-4ffb-8d99-2e2806ec05db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" Oct 30 05:27:39.806853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.710149 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67tmq\" (UniqueName: \"kubernetes.io/projected/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-kube-api-access-67tmq\") pod \"payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj\" (UID: \"6ac662ee-f41a-4ffb-8d99-2e2806ec05db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" Oct 30 05:27:39.806853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.712596 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body= Oct 30 05:27:39.806853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.712643 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" Oct 30 05:27:39.806853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.746554 6587 generic.go:356] "Generic (PLEG): container finished" podID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerID="d6b099e09fee607d4fc2f46adcafa58f1d9766c952269c0be47116710649b530" exitCode=0 Oct 30 05:27:39.878665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.878613 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h"] Oct 30 05:27:39.880715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.880290 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:27:39.886195 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.886169 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" Oct 30 05:27:39.887093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.886056 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" event={"ID":"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d","Type":"ContainerDied","Data":"d6b099e09fee607d4fc2f46adcafa58f1d9766c952269c0be47116710649b530"} Oct 30 05:27:39.887330 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.887311 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h"] Oct 30 05:27:39.887620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:39.887604 6587 scope.go:122] "RemoveContainer" containerID="d6b099e09fee607d4fc2f46adcafa58f1d9766c952269c0be47116710649b530" Oct 30 05:27:40.009491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.008807 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" Oct 30 05:27:40.012666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.012629 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-token\" (UniqueName: \"kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-token\") pod \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " Oct 30 05:27:40.012821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.012688 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-credential-socket\") pod \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " Oct 30 05:27:40.012821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.012788 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-data\") pod \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " Oct 30 05:27:40.012962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.012846 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-socket\") pod \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " Oct 30 05:27:40.013023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.012986 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istiod-ca-cert\") pod \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " Oct 30 05:27:40.013220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.013094 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-envoy\") pod \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " Oct 30 05:27:40.013220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.013197 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-podinfo\") pod \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " Oct 30 05:27:40.013394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.013242 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-kube-api-access-dv48n\" (UniqueName: \"kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-kube-api-access-dv48n\") pod \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " Oct 30 05:27:40.013394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.013305 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-certs\") pod \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\" (UID: \"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d\") " Oct 30 05:27:40.013755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.013721 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a13271-a884-4256-8ae5-f24d9a61c7c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h\" (UID: \"82a13271-a884-4256-8ae5-f24d9a61c7c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" Oct 30 05:27:40.013914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.013895 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9lgk\" (UniqueName: \"kubernetes.io/projected/82a13271-a884-4256-8ae5-f24d9a61c7c6-kube-api-access-c9lgk\") pod \"payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h\" (UID: \"82a13271-a884-4256-8ae5-f24d9a61c7c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" Oct 30 05:27:40.069631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.068627 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-data" pod "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" (UID: "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:27:40.069631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.069518 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-token" pod "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" (UID: "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:27:40.071559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.070522 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-certs" pod "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" (UID: "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:27:40.071559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.070540 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istiod-ca-cert" pod "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" (UID: "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:27:40.071559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.070720 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-socket" pod "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" (UID: "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:27:40.071559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.070867 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-credential-socket" pod "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" (UID: "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:27:40.072256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.072208 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-kube-api-access-dv48n" pod "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" (UID: "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d"). InnerVolumeSpecName "kube-api-access-dv48n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:27:40.072781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.072757 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-podinfo" pod "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" (UID: "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:27:40.073986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.073860 6587 scope.go:122] "RemoveContainer" containerID="dd4db8126824b483c87e4f80ff2c90a5acaa3b1f79d4e9729222d31794d52103" Oct 30 05:27:40.076237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.076214 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-envoy" pod "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" (UID: "58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:27:40.155814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.135458 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a13271-a884-4256-8ae5-f24d9a61c7c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h\" (UID: \"82a13271-a884-4256-8ae5-f24d9a61c7c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" Oct 30 05:27:40.155814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.135674 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9lgk\" (UniqueName: \"kubernetes.io/projected/82a13271-a884-4256-8ae5-f24d9a61c7c6-kube-api-access-c9lgk\") pod \"payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h\" (UID: \"82a13271-a884-4256-8ae5-f24d9a61c7c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" Oct 30 05:27:40.155814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.136054 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:40.155814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.136080 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:40.155814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.136096 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:40.155814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.136113 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:40.155814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.136128 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:40.155814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.136142 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv48n\" (UniqueName: \"kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-kube-api-access-dv48n\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:40.155814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.136157 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:40.155814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.136172 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:40.155814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.136185 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:27:40.155814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.137544 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a13271-a884-4256-8ae5-f24d9a61c7c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h\" (UID: \"82a13271-a884-4256-8ae5-f24d9a61c7c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" Oct 30 05:27:40.170533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.170177 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9lgk\" (UniqueName: \"kubernetes.io/projected/82a13271-a884-4256-8ae5-f24d9a61c7c6-kube-api-access-c9lgk\") pod \"payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h\" (UID: \"82a13271-a884-4256-8ae5-f24d9a61c7c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" Oct 30 05:27:40.249381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.248584 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" Oct 30 05:27:40.462542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.462512 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj"] Oct 30 05:27:40.613429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.611150 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h"] Oct 30 05:27:40.710023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.709994 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body= Oct 30 05:27:40.710155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.710041 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" Oct 30 05:27:40.789131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.789089 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" event={"ID":"82a13271-a884-4256-8ae5-f24d9a61c7c6","Type":"ContainerStarted","Data":"4ec234737db6bd8e9f9ebdb77d66c3f7f04339304092fdb0a7ba36c1ce2ecac5"} Oct 30 05:27:40.789323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.789190 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" event={"ID":"82a13271-a884-4256-8ae5-f24d9a61c7c6","Type":"ContainerStarted","Data":"5a7090d5e157d5fb5c90098c6a805ab92f38b4398acd73f9011fdd943e46de3c"} Oct 30 05:27:40.790409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.790383 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" Oct 30 05:27:40.794199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.794168 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" event={"ID":"6ac662ee-f41a-4ffb-8d99-2e2806ec05db","Type":"ContainerStarted","Data":"3322dea5a4de53119c143ae77795e28ab6a9862de853ab4cf8e4cac244a2fc8a"} Oct 30 05:27:40.794320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.794229 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" event={"ID":"6ac662ee-f41a-4ffb-8d99-2e2806ec05db","Type":"ContainerStarted","Data":"c7e912b9108c0c152f24eab35ec3bd29af6e715eab5e5ae9a3a9594be88f5aa0"} Oct 30 05:27:40.794686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.794667 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" Oct 30 05:27:40.798982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.798959 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" event={"ID":"58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d","Type":"ContainerDied","Data":"08c7b22aed689244620d9bbfa6857a934e4ea4bc07635b5da61e6a9cdc4c6712"} Oct 30 05:27:40.799116 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.799038 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s" Oct 30 05:27:40.870601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.870479 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" podStartSLOduration=1.8704595529999999 podStartE2EDuration="1.870459553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:40.865270509 +0000 UTC m=+2585.393037297" watchObservedRunningTime="2026-10-30 05:27:40.870459553 +0000 UTC m=+2585.398226341" Oct 30 05:27:40.952232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.952198 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s"] Oct 30 05:27:40.961666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.961637 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s"] Oct 30 05:27:40.975005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:40.974944 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" podStartSLOduration=1.974925437 podStartE2EDuration="1.974925437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:40.968627661 +0000 UTC m=+2585.496394449" watchObservedRunningTime="2026-10-30 05:27:40.974925437 +0000 UTC m=+2585.502692225" Oct 30 05:27:41.030513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.015345 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb"] Oct 30 05:27:41.030513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.027864 6587 cpu_manager.go:414] "containerMap: removing container" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" Oct 30 05:27:41.030513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.027893 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" Oct 30 05:27:41.030513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.027951 6587 cpu_manager.go:414] "containerMap: removing container" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" Oct 30 05:27:41.030513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.027960 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" Oct 30 05:27:41.030513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.028158 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" Oct 30 05:27:41.056409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.055647 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb"] Oct 30 05:27:41.056409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.055772 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" Oct 30 05:27:41.072538 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.072513 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-9dc17a\"" Oct 30 05:27:41.077554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.077514 6587 scope.go:122] "RemoveContainer" containerID="d47242dbff5841b0cf5ec24013200e860871254dab009a6336fac13fee9e9db8" Oct 30 05:27:41.079600 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.079575 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z284l\" (UniqueName: \"kubernetes.io/projected/3baee78b-9919-4aab-a971-7251ce5ecd29-kube-api-access-z284l\") pod \"payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb\" (UID: \"3baee78b-9919-4aab-a971-7251ce5ecd29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" Oct 30 05:27:41.079761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.079743 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3baee78b-9919-4aab-a971-7251ce5ecd29-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb\" (UID: \"3baee78b-9919-4aab-a971-7251ce5ecd29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" Oct 30 05:27:41.105453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.104816 6587 scope.go:122] "RemoveContainer" containerID="f46e3426009c373d03045ff75bd8873a8c6a2a9e67bc72f4a40f1ed2ebf00ada" Oct 30 05:27:41.167849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.141848 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-9dc17a-dockercfg-zvzrk\"" Oct 30 05:27:41.218380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.201347 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z284l\" (UniqueName: \"kubernetes.io/projected/3baee78b-9919-4aab-a971-7251ce5ecd29-kube-api-access-z284l\") pod \"payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb\" (UID: \"3baee78b-9919-4aab-a971-7251ce5ecd29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" Oct 30 05:27:41.218726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.212138 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h"] Oct 30 05:27:41.247635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.219034 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3baee78b-9919-4aab-a971-7251ce5ecd29-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb\" (UID: \"3baee78b-9919-4aab-a971-7251ce5ecd29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" Oct 30 05:27:41.248663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.219935 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3baee78b-9919-4aab-a971-7251ce5ecd29-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb\" (UID: \"3baee78b-9919-4aab-a971-7251ce5ecd29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" Oct 30 05:27:41.278845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.278806 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" containerName="istio-proxy" Oct 30 05:27:41.332190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.329049 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z284l\" (UniqueName: \"kubernetes.io/projected/3baee78b-9919-4aab-a971-7251ce5ecd29-kube-api-access-z284l\") pod \"payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb\" (UID: \"3baee78b-9919-4aab-a971-7251ce5ecd29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" Oct 30 05:27:41.332190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.330223 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" Oct 30 05:27:41.362063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.357383 6587 scope.go:122] "RemoveContainer" containerID="856a0aa5777eca5db9fc5d51db6aa3000b617435d594b567b6505db9f393e112" Oct 30 05:27:41.375076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.374813 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" Oct 30 05:27:41.502351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.502180 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h"] Oct 30 05:27:41.589715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.589675 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/573bf036-fa9c-437d-bb41-8c93e82cb568-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h\" (UID: \"573bf036-fa9c-437d-bb41-8c93e82cb568\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" Oct 30 05:27:41.589883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.589763 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpgdg\" (UniqueName: \"kubernetes.io/projected/573bf036-fa9c-437d-bb41-8c93e82cb568-kube-api-access-cpgdg\") pod \"payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h\" (UID: \"573bf036-fa9c-437d-bb41-8c93e82cb568\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" Oct 30 05:27:41.597832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.597808 6587 scope.go:122] "RemoveContainer" containerID="b056bee6225a6c1f6ea45e4f07007f37ef6d4dd10f835fb6c7ab86e523926a3f" Oct 30 05:27:41.611801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.611702 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d" path="/var/lib/kubelet/pods/58a9f78e-8bc2-4bf0-adf1-eab0ca0bf81d/volumes" Oct 30 05:27:41.691752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.691725 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/573bf036-fa9c-437d-bb41-8c93e82cb568-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h\" (UID: \"573bf036-fa9c-437d-bb41-8c93e82cb568\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" Oct 30 05:27:41.692202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.692183 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpgdg\" (UniqueName: \"kubernetes.io/projected/573bf036-fa9c-437d-bb41-8c93e82cb568-kube-api-access-cpgdg\") pod \"payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h\" (UID: \"573bf036-fa9c-437d-bb41-8c93e82cb568\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" Oct 30 05:27:41.693199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.693169 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/573bf036-fa9c-437d-bb41-8c93e82cb568-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h\" (UID: \"573bf036-fa9c-437d-bb41-8c93e82cb568\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" Oct 30 05:27:41.709038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.709006 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body= Oct 30 05:27:41.709175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.709060 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" Oct 30 05:27:41.714073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.714036 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpgdg\" (UniqueName: \"kubernetes.io/projected/573bf036-fa9c-437d-bb41-8c93e82cb568-kube-api-access-cpgdg\") pod \"payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h\" (UID: \"573bf036-fa9c-437d-bb41-8c93e82cb568\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" Oct 30 05:27:41.803412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.803261 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" Oct 30 05:27:41.975581 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:27:41.974569 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3baee78b_9919_4aab_a971_7251ce5ecd29.slice/crio-ac10d6036a68c667802e76c7643dc5842b2550a2aa2509ddbd591b6288351bc3 WatchSource:0}: Error finding container ac10d6036a68c667802e76c7643dc5842b2550a2aa2509ddbd591b6288351bc3: Status 404 returned error can't find the container with id ac10d6036a68c667802e76c7643dc5842b2550a2aa2509ddbd591b6288351bc3 Oct 30 05:27:41.982063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:41.982033 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb"] Oct 30 05:27:42.188841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:42.188626 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h"] Oct 30 05:27:42.709392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:42.708909 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body= Oct 30 05:27:42.709392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:42.708977 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" Oct 30 05:27:42.852385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:42.848643 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" event={"ID":"3baee78b-9919-4aab-a971-7251ce5ecd29","Type":"ContainerStarted","Data":"6e22817b359afc91383c98a429fbdc0ec8328068e508c3a7e88820d081b19c20"} Oct 30 05:27:42.852385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:42.848712 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" event={"ID":"3baee78b-9919-4aab-a971-7251ce5ecd29","Type":"ContainerStarted","Data":"ac10d6036a68c667802e76c7643dc5842b2550a2aa2509ddbd591b6288351bc3"} Oct 30 05:27:42.852385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:42.849933 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" Oct 30 05:27:42.859406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:42.858305 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" event={"ID":"573bf036-fa9c-437d-bb41-8c93e82cb568","Type":"ContainerStarted","Data":"44955a785f067f7768fe299ca8a4626208e6dab73568b43355a178577236f20a"} Oct 30 05:27:42.859406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:42.858351 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" event={"ID":"573bf036-fa9c-437d-bb41-8c93e82cb568","Type":"ContainerStarted","Data":"0a729eb7a00b91d957b31e55066295d00a430e01d8242c42a00f264a1ec2eff3"} Oct 30 05:27:42.936625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:42.930542 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" podStartSLOduration=2.930522121 podStartE2EDuration="2.930522121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:42.910663935 +0000 UTC m=+2587.438430720" watchObservedRunningTime="2026-10-30 05:27:42.930522121 +0000 UTC m=+2587.458288910" Oct 30 05:27:42.984320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:42.984201 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" podStartSLOduration=1.9841877989999999 podStartE2EDuration="1.984187799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:42.979833373 +0000 UTC m=+2587.507600160" watchObservedRunningTime="2026-10-30 05:27:42.984187799 +0000 UTC m=+2587.511954582" Oct 30 05:27:43.743713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:43.743678 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:27:43.744782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:43.744697 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:27:43.746536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:43.746514 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:27:43.869755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:43.869724 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" Oct 30 05:27:48.281446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:48.281409 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8"] Oct 30 05:27:49.334697 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.334659 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2"] Oct 30 05:27:49.344905 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.344877 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.375415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.371395 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2"] Oct 30 05:27:49.474850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.474813 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.475046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.474948 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgg4n\" (UniqueName: \"kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-kube-api-access-zgg4n\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.475207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.475183 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.475403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.475340 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.475502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.475412 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.475502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.475449 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.475502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.475477 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.475616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.475534 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.475616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.475561 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.576564 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.576530 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.576759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.576658 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.576759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.576706 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.576759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.576740 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.576961 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.576765 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.576961 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.576806 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.576961 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.576842 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.576961 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.576890 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.577182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.576988 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgg4n\" (UniqueName: \"kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-kube-api-access-zgg4n\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.577413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.577372 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.577413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.577398 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.577666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.577639 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.577740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.577671 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.577740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.577682 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.579476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.579452 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.579708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.579687 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.598710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.598646 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.598809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.598709 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgg4n\" (UniqueName: \"kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-kube-api-access-zgg4n\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.674865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.674833 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:49.727993 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.726987 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s"] Oct 30 05:27:49.734701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.734669 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.754315 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.754235 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s"] Oct 30 05:27:49.755429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.755133 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-34722e\"/\"sub-test-model-34722e-kserve-self-signed-certs\"" Oct 30 05:27:49.758198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.757975 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-34722e\"/\"default-dockercfg-4fbl7\"" Oct 30 05:27:49.758385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.758008 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-34722e\"/\"kube-root-ca.crt\"" Oct 30 05:27:49.758506 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.758076 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-34722e\"/\"openshift-service-ca.crt\"" Oct 30 05:27:49.759051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.759036 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" Oct 30 05:27:49.759590 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.759567 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" Oct 30 05:27:49.883109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.883027 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-tmp-dir\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.883109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.883082 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-dshm\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.883109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.883104 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-home\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.883449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.883213 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c382def9-ebcf-44e3-a70f-cca002550278-tls-certs\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.883449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.883248 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-model-cache\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.883449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.883296 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8qz8\" (UniqueName: \"kubernetes.io/projected/c382def9-ebcf-44e3-a70f-cca002550278-kube-api-access-w8qz8\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.985124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.985092 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-tmp-dir\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.985321 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.985151 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-dshm\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.985321 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.985180 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-home\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.985321 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.985277 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c382def9-ebcf-44e3-a70f-cca002550278-tls-certs\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.985321 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.985312 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-model-cache\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.985601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.985377 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qz8\" (UniqueName: \"kubernetes.io/projected/c382def9-ebcf-44e3-a70f-cca002550278-kube-api-access-w8qz8\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.985666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.985606 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-tmp-dir\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.985666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.985652 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-home\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.985762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.985721 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-model-cache\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.987555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.987532 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-dshm\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:49.987819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:49.987800 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c382def9-ebcf-44e3-a70f-cca002550278-tls-certs\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:50.007543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:50.007515 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8qz8\" (UniqueName: \"kubernetes.io/projected/c382def9-ebcf-44e3-a70f-cca002550278-kube-api-access-w8qz8\") pod \"sub-test-model-34722e-kserve-5cc88f6d79-5s64s\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:50.106392 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:27:50.100867 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd599f4e_76ee_4ac0_ac7b_95c766cca3f9.slice/crio-e334320f8b3af8d225c85ff293f58db68a0dedb1cc40c2b1289f921656070824 WatchSource:0}: Error finding container e334320f8b3af8d225c85ff293f58db68a0dedb1cc40c2b1289f921656070824: Status 404 returned error can't find the container with id e334320f8b3af8d225c85ff293f58db68a0dedb1cc40c2b1289f921656070824 Oct 30 05:27:50.106392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:50.102448 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:50.171613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:50.170680 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2"] Oct 30 05:27:50.194527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:50.194485 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:50.194652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:50.194633 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:50.194743 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:50.194721 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:27:50.504161 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:27:50.504120 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc382def9_ebcf_44e3_a70f_cca002550278.slice/crio-eb428c40a144a116932480c0369728439d183e3776dee4c17739e3e9096bed19 WatchSource:0}: Error finding container eb428c40a144a116932480c0369728439d183e3776dee4c17739e3e9096bed19: Status 404 returned error can't find the container with id eb428c40a144a116932480c0369728439d183e3776dee4c17739e3e9096bed19 Oct 30 05:27:50.508867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:50.508838 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s"] Oct 30 05:27:50.942780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:50.942729 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" event={"ID":"c382def9-ebcf-44e3-a70f-cca002550278","Type":"ContainerStarted","Data":"ff3c5b9c4b60163c97a925d956d395d44e76ec3b91645cb148567b979d3f60a5"} Oct 30 05:27:50.942780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:50.942787 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" event={"ID":"c382def9-ebcf-44e3-a70f-cca002550278","Type":"ContainerStarted","Data":"eb428c40a144a116932480c0369728439d183e3776dee4c17739e3e9096bed19"} Oct 30 05:27:50.944857 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:50.944823 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" event={"ID":"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9","Type":"ContainerStarted","Data":"551e859e4d30609a03548d99aa818ce0969bed5b0ef1ac984b7232c56937a361"} Oct 30 05:27:50.945039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:50.944881 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" event={"ID":"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9","Type":"ContainerStarted","Data":"e334320f8b3af8d225c85ff293f58db68a0dedb1cc40c2b1289f921656070824"} Oct 30 05:27:50.986823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:50.986773 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" podStartSLOduration=1.986760978 podStartE2EDuration="1.986760978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:50.985667313 +0000 UTC m=+2595.513434100" watchObservedRunningTime="2026-10-30 05:27:50.986760978 +0000 UTC m=+2595.514527764" Oct 30 05:27:51.036434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:51.036377 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" podStartSLOduration=2.036348967 podStartE2EDuration="2.036348967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:51.031651059 +0000 UTC m=+2595.559417846" watchObservedRunningTime="2026-10-30 05:27:51.036348967 +0000 UTC m=+2595.564115752" Oct 30 05:27:51.106295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:51.106256 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:27:51.675457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:51.675424 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:51.677350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:51.677323 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 05:27:51.677512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:51.677401 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" podUID="cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 05:27:51.813262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:51.813224 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" Oct 30 05:27:52.675854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:52.675815 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 05:27:52.676281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:52.675880 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" podUID="cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 05:27:52.861267 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:52.861228 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" Oct 30 05:27:53.675641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:53.675611 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 05:27:53.675850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:53.675661 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" podUID="cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 05:27:54.675513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:54.675478 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 05:27:54.675899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:54.675548 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" podUID="cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 05:27:54.880644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:54.880610 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" Oct 30 05:27:54.881066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:54.881042 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" Oct 30 05:27:55.675317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:55.675281 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 05:27:55.675643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:55.675605 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" podUID="cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 05:27:56.005447 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:27:56.002566 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:27:56.675441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:56.675401 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 05:27:56.675956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:56.675475 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" podUID="cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 05:27:57.681065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:57.681035 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:57.681543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:57.681398 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:27:57.682138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:27:57.682113 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:28:00.106191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:00.106159 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:28:00.120367 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:00.120328 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:28:01.087715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:01.087688 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:28:02.388380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.385736 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd"] Oct 30 05:28:02.416675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.416645 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.472383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.465013 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2d21ab-openshift-default-dockercfg-85kql\"" Oct 30 05:28:02.535450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.519592 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.535791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.535764 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.536043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.536015 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.536335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.536310 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.536545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.536526 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.536671 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.536654 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.536809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.536794 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.544395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.537035 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxss\" (UniqueName: \"kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-kube-api-access-txxss\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.545210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.545185 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9342248-e93d-4bb7-8f9e-1a20117d05be-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.663436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.653722 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.663436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.653779 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.663436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.653832 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.663436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.653961 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxss\" (UniqueName: \"kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-kube-api-access-txxss\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.663436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.654024 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9342248-e93d-4bb7-8f9e-1a20117d05be-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.681473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.671669 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.681473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.671744 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.681473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.671882 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.681473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.672088 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.681473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.672725 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.682463 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.682223 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.683837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.683571 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9342248-e93d-4bb7-8f9e-1a20117d05be-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.684572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.684251 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.687434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.687205 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.691414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.691104 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.696766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.696744 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd"] Oct 30 05:28:02.714008 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.713970 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.715317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.715292 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.722986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.722494 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxss\" (UniqueName: \"kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-kube-api-access-txxss\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:02.943281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:02.943187 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:03.343385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:03.329476 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" podUID="1732cb93-3b7c-4eea-a360-8c4cf9e6943d" containerName="main" containerID="cri-o://6c12bcafa8459adab42be854b0459096ef0c01938899301258ef1e97d26fa356" gracePeriod=45 Oct 30 05:28:03.714797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:03.710142 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:03.714797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:03.710251 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:03.714797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:03.710291 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:03.778168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:03.778130 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd"] Oct 30 05:28:04.265435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:04.259996 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" podUID="1732cb93-3b7c-4eea-a360-8c4cf9e6943d" containerName="main" probeResult="failure" output="Get \"https://10.134.1.18:8000/ready\": dial tcp 10.134.1.18:8000: connect: connection refused" Oct 30 05:28:04.425384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:04.422083 6587 generic.go:356] "Generic (PLEG): container finished" podID="1732cb93-3b7c-4eea-a360-8c4cf9e6943d" containerID="6c12bcafa8459adab42be854b0459096ef0c01938899301258ef1e97d26fa356" exitCode=0 Oct 30 05:28:04.425384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:04.422209 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" event={"ID":"1732cb93-3b7c-4eea-a360-8c4cf9e6943d","Type":"ContainerDied","Data":"6c12bcafa8459adab42be854b0459096ef0c01938899301258ef1e97d26fa356"} Oct 30 05:28:04.434428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:04.434391 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" event={"ID":"a9342248-e93d-4bb7-8f9e-1a20117d05be","Type":"ContainerStarted","Data":"22dea87412cbf5bbc4a0e512fbb02cf95405e767b261bdb3c9c07677b580e6e2"} Oct 30 05:28:04.434702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:04.434685 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" event={"ID":"a9342248-e93d-4bb7-8f9e-1a20117d05be","Type":"ContainerStarted","Data":"fa790f818ed13046496bb1a81416bce69f370828eee240b4c9943133181ff3da"} Oct 30 05:28:04.628379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:04.610939 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" podStartSLOduration=2.6109124870000002 podStartE2EDuration="2.610912487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:04.553543147 +0000 UTC m=+2609.081309933" watchObservedRunningTime="2026-10-30 05:28:04.610912487 +0000 UTC m=+2609.138679273" Oct 30 05:28:04.758393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:04.756215 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:28:04.960450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:04.951613 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:05.107382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.105730 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" podUID="c382def9-ebcf-44e3-a70f-cca002550278" containerName="main" probeResult="failure" output="Get \"https://10.134.1.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:28:05.166381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.165951 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" podUID="40b7c36f-e157-4cc4-a04e-962fa1a7cc77" containerName="main" probeResult="failure" output="Get \"https://10.134.1.19:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:28:05.230426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.229467 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:28:05.369432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.366024 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-model-cache\") pod \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " Oct 30 05:28:05.369432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.366133 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tmp-dir\") pod \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " Oct 30 05:28:05.369432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.366233 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tls-certs\" (UniqueName: \"kubernetes.io/secret/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tls-certs\") pod \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " Oct 30 05:28:05.369432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.366307 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-home\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-home\") pod \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " Oct 30 05:28:05.369432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.366379 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-kube-api-access-j8l26\" (UniqueName: \"kubernetes.io/projected/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-kube-api-access-j8l26\") pod \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " Oct 30 05:28:05.369432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.366548 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-dshm\") pod \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\" (UID: \"1732cb93-3b7c-4eea-a360-8c4cf9e6943d\") " Oct 30 05:28:05.369432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.366705 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tmp-dir" pod "1732cb93-3b7c-4eea-a360-8c4cf9e6943d" (UID: "1732cb93-3b7c-4eea-a360-8c4cf9e6943d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:05.369432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.366951 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-model-cache" pod "1732cb93-3b7c-4eea-a360-8c4cf9e6943d" (UID: "1732cb93-3b7c-4eea-a360-8c4cf9e6943d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:05.369432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.367697 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-home" pod "1732cb93-3b7c-4eea-a360-8c4cf9e6943d" (UID: "1732cb93-3b7c-4eea-a360-8c4cf9e6943d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:05.383382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.370679 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tls-certs" pod "1732cb93-3b7c-4eea-a360-8c4cf9e6943d" (UID: "1732cb93-3b7c-4eea-a360-8c4cf9e6943d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:28:05.383382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.373694 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-dshm" pod "1732cb93-3b7c-4eea-a360-8c4cf9e6943d" (UID: "1732cb93-3b7c-4eea-a360-8c4cf9e6943d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:05.383382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.374426 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:05.383382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.374452 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:05.383382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.382525 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-kube-api-access-j8l26" pod "1732cb93-3b7c-4eea-a360-8c4cf9e6943d" (UID: "1732cb93-3b7c-4eea-a360-8c4cf9e6943d"). InnerVolumeSpecName "kube-api-access-j8l26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:28:05.463381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.462352 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" Oct 30 05:28:05.463585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.463506 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8" event={"ID":"1732cb93-3b7c-4eea-a360-8c4cf9e6943d","Type":"ContainerDied","Data":"f6fe1bd2fd8c45e8d9e4295c6fb1f135d874e74068ef660e62ff6636ebf88736"} Oct 30 05:28:05.463660 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.463626 6587 scope.go:122] "RemoveContainer" containerID="6c12bcafa8459adab42be854b0459096ef0c01938899301258ef1e97d26fa356" Oct 30 05:28:05.483527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.483441 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:05.483527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.483478 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:05.483527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.483492 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8l26\" (UniqueName: \"kubernetes.io/projected/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-kube-api-access-j8l26\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:05.483527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.483504 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1732cb93-3b7c-4eea-a360-8c4cf9e6943d-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:05.552424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.552348 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8"] Oct 30 05:28:05.559453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.559419 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ff4760/auth-test-model-ff4760-kserve-6d4dd4797-fldz8"] Oct 30 05:28:05.592453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.592344 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1732cb93-3b7c-4eea-a360-8c4cf9e6943d" path="/var/lib/kubelet/pods/1732cb93-3b7c-4eea-a360-8c4cf9e6943d/volumes" Oct 30 05:28:05.950679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:05.950642 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:06.473667 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:06.473504 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:06.474562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:06.474377 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:28:07.506156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.506122 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg"] Oct 30 05:28:07.507078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.507057 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1732cb93-3b7c-4eea-a360-8c4cf9e6943d" containerName="main" Oct 30 05:28:07.507078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.507077 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1732cb93-3b7c-4eea-a360-8c4cf9e6943d" containerName="main" Oct 30 05:28:07.507322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.507306 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1732cb93-3b7c-4eea-a360-8c4cf9e6943d" containerName="main" Oct 30 05:28:07.521423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.521392 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" Oct 30 05:28:07.554615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.552666 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:28:07.554615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.553436 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:28:07.554615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.553526 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:28:07.607393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.580595 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:28:07.658760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.653484 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e19f2ea-cnnqg\" (UID: \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" Oct 30 05:28:07.658760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.653677 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85f2x\" (UniqueName: \"kubernetes.io/projected/027c5a3b-eabe-4622-a1e2-7294bfe4edda-kube-api-access-85f2x\") pod \"maas-api-revoke-sub-1e19f2ea-cnnqg\" (UID: \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" Oct 30 05:28:07.658760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.653751 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e19f2ea-cnnqg\" (UID: \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" Oct 30 05:28:07.755824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.755786 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e19f2ea-cnnqg\" (UID: \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" Oct 30 05:28:07.756845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.756772 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85f2x\" (UniqueName: \"kubernetes.io/projected/027c5a3b-eabe-4622-a1e2-7294bfe4edda-kube-api-access-85f2x\") pod \"maas-api-revoke-sub-1e19f2ea-cnnqg\" (UID: \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" Oct 30 05:28:07.757600 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.757577 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e19f2ea-cnnqg\" (UID: \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" Oct 30 05:28:07.758656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.758458 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e19f2ea-cnnqg\" (UID: \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" Oct 30 05:28:07.776730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.776579 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e19f2ea-cnnqg\" (UID: \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" Oct 30 05:28:07.781417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.780789 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85f2x\" (UniqueName: \"kubernetes.io/projected/027c5a3b-eabe-4622-a1e2-7294bfe4edda-kube-api-access-85f2x\") pod \"maas-api-revoke-sub-1e19f2ea-cnnqg\" (UID: \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" Oct 30 05:28:07.803375 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.802200 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg"] Oct 30 05:28:07.964313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:07.964282 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" Oct 30 05:28:08.274561 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:28:08.274520 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027c5a3b_eabe_4622_a1e2_7294bfe4edda.slice/crio-ef1137fdd87875ef6755f48101871372bb16270775663a24055c06065ed42e97 WatchSource:0}: Error finding container ef1137fdd87875ef6755f48101871372bb16270775663a24055c06065ed42e97: Status 404 returned error can't find the container with id ef1137fdd87875ef6755f48101871372bb16270775663a24055c06065ed42e97 Oct 30 05:28:08.275632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:08.275609 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg"] Oct 30 05:28:08.511376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:08.508824 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" event={"ID":"027c5a3b-eabe-4622-a1e2-7294bfe4edda","Type":"ContainerStarted","Data":"ef1137fdd87875ef6755f48101871372bb16270775663a24055c06065ed42e97"} Oct 30 05:28:09.243632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.243596 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs"] Oct 30 05:28:09.248569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.248549 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" Oct 30 05:28:09.265543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.265500 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2d21ab-dockercfg-bh6dj\"" Oct 30 05:28:09.272041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.271740 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs"] Oct 30 05:28:09.278184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.278156 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2d21ab\"" Oct 30 05:28:09.319078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.319044 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9"] Oct 30 05:28:09.325963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.325942 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" Oct 30 05:28:09.339699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.339659 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9"] Oct 30 05:28:09.507438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.504513 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bz46\" (UniqueName: \"kubernetes.io/projected/046e8b73-ad3b-4126-85be-742df2a606cd-kube-api-access-8bz46\") pod \"payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs\" (UID: \"046e8b73-ad3b-4126-85be-742df2a606cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" Oct 30 05:28:09.507438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.504912 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046e8b73-ad3b-4126-85be-742df2a606cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs\" (UID: \"046e8b73-ad3b-4126-85be-742df2a606cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" Oct 30 05:28:09.641378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.638998 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlst\" (UniqueName: \"kubernetes.io/projected/513f322a-2da8-4d24-8ed1-2ee8af8a7991-kube-api-access-5hlst\") pod \"payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9\" (UID: \"513f322a-2da8-4d24-8ed1-2ee8af8a7991\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" Oct 30 05:28:09.641378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.639123 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046e8b73-ad3b-4126-85be-742df2a606cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs\" (UID: \"046e8b73-ad3b-4126-85be-742df2a606cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" Oct 30 05:28:09.641378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.639265 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bz46\" (UniqueName: \"kubernetes.io/projected/046e8b73-ad3b-4126-85be-742df2a606cd-kube-api-access-8bz46\") pod \"payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs\" (UID: \"046e8b73-ad3b-4126-85be-742df2a606cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" Oct 30 05:28:09.641378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.639393 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513f322a-2da8-4d24-8ed1-2ee8af8a7991-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9\" (UID: \"513f322a-2da8-4d24-8ed1-2ee8af8a7991\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" Oct 30 05:28:09.665560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.664520 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046e8b73-ad3b-4126-85be-742df2a606cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs\" (UID: \"046e8b73-ad3b-4126-85be-742df2a606cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" Oct 30 05:28:09.667868 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.667833 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" event={"ID":"027c5a3b-eabe-4622-a1e2-7294bfe4edda","Type":"ContainerStarted","Data":"ffc48516b28ace24a2ef02c5ddc64c8ddb4740e5d66e580072d8dcedd9babec1"} Oct 30 05:28:09.689990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.689923 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bz46\" (UniqueName: \"kubernetes.io/projected/046e8b73-ad3b-4126-85be-742df2a606cd-kube-api-access-8bz46\") pod \"payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs\" (UID: \"046e8b73-ad3b-4126-85be-742df2a606cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" Oct 30 05:28:09.719872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.719816 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" podStartSLOduration=2.719800128 podStartE2EDuration="2.719800128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:09.71918419 +0000 UTC m=+2614.246950977" watchObservedRunningTime="2026-10-30 05:28:09.719800128 +0000 UTC m=+2614.247566920" Oct 30 05:28:09.741686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.741645 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513f322a-2da8-4d24-8ed1-2ee8af8a7991-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9\" (UID: \"513f322a-2da8-4d24-8ed1-2ee8af8a7991\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" Oct 30 05:28:09.741936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.741920 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlst\" (UniqueName: \"kubernetes.io/projected/513f322a-2da8-4d24-8ed1-2ee8af8a7991-kube-api-access-5hlst\") pod \"payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9\" (UID: \"513f322a-2da8-4d24-8ed1-2ee8af8a7991\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" Oct 30 05:28:09.743088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.743058 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513f322a-2da8-4d24-8ed1-2ee8af8a7991-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9\" (UID: \"513f322a-2da8-4d24-8ed1-2ee8af8a7991\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" Oct 30 05:28:09.763175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.763098 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlst\" (UniqueName: \"kubernetes.io/projected/513f322a-2da8-4d24-8ed1-2ee8af8a7991-kube-api-access-5hlst\") pod \"payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9\" (UID: \"513f322a-2da8-4d24-8ed1-2ee8af8a7991\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" Oct 30 05:28:09.864637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:09.864605 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" Oct 30 05:28:10.003846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:10.003808 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" Oct 30 05:28:10.198139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:10.198103 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs"] Oct 30 05:28:10.200723 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:28:10.200691 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046e8b73_ad3b_4126_85be_742df2a606cd.slice/crio-384d77a8b5411f1c7e2ca6caa5cb2587a1ef0a0d3431d02ccdc8ce37211685e6 WatchSource:0}: Error finding container 384d77a8b5411f1c7e2ca6caa5cb2587a1ef0a0d3431d02ccdc8ce37211685e6: Status 404 returned error can't find the container with id 384d77a8b5411f1c7e2ca6caa5cb2587a1ef0a0d3431d02ccdc8ce37211685e6 Oct 30 05:28:10.355940 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:28:10.355903 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513f322a_2da8_4d24_8ed1_2ee8af8a7991.slice/crio-ca3e1b882a747dfa8d0344c03b8fbae836bb389a0ddf26b89204a87266eb4988 WatchSource:0}: Error finding container ca3e1b882a747dfa8d0344c03b8fbae836bb389a0ddf26b89204a87266eb4988: Status 404 returned error can't find the container with id ca3e1b882a747dfa8d0344c03b8fbae836bb389a0ddf26b89204a87266eb4988 Oct 30 05:28:10.358512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:10.358482 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9"] Oct 30 05:28:10.690069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:10.690019 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" event={"ID":"046e8b73-ad3b-4126-85be-742df2a606cd","Type":"ContainerStarted","Data":"4cd9f819255ea7b060e860a3be8f74dc9ae020477a3c06a847a90d9b28b327ac"} Oct 30 05:28:10.690580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:10.690129 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" event={"ID":"046e8b73-ad3b-4126-85be-742df2a606cd","Type":"ContainerStarted","Data":"384d77a8b5411f1c7e2ca6caa5cb2587a1ef0a0d3431d02ccdc8ce37211685e6"} Oct 30 05:28:10.690580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:10.690229 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" Oct 30 05:28:10.692211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:10.692171 6587 generic.go:356] "Generic (PLEG): container finished" podID="027c5a3b-eabe-4622-a1e2-7294bfe4edda" containerID="ffc48516b28ace24a2ef02c5ddc64c8ddb4740e5d66e580072d8dcedd9babec1" exitCode=0 Oct 30 05:28:10.692380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:10.692249 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" event={"ID":"027c5a3b-eabe-4622-a1e2-7294bfe4edda","Type":"ContainerDied","Data":"ffc48516b28ace24a2ef02c5ddc64c8ddb4740e5d66e580072d8dcedd9babec1"} Oct 30 05:28:10.693976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:10.693950 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" event={"ID":"513f322a-2da8-4d24-8ed1-2ee8af8a7991","Type":"ContainerStarted","Data":"fdfff254fd06b517eb9146efb2c802fb30305e6a4afa9d69568d2aa76f0a4994"} Oct 30 05:28:10.694148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:10.693994 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" event={"ID":"513f322a-2da8-4d24-8ed1-2ee8af8a7991","Type":"ContainerStarted","Data":"ca3e1b882a747dfa8d0344c03b8fbae836bb389a0ddf26b89204a87266eb4988"} Oct 30 05:28:10.694148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:10.694052 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" Oct 30 05:28:10.739079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:10.739020 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" podStartSLOduration=1.739003046 podStartE2EDuration="1.739003046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:10.733103327 +0000 UTC m=+2615.260870113" watchObservedRunningTime="2026-10-30 05:28:10.739003046 +0000 UTC m=+2615.266769832" Oct 30 05:28:10.821588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:10.821517 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" podStartSLOduration=1.8214993320000001 podStartE2EDuration="1.821499332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:10.821058569 +0000 UTC m=+2615.348825355" watchObservedRunningTime="2026-10-30 05:28:10.821499332 +0000 UTC m=+2615.349266122" Oct 30 05:28:11.956132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:11.936868 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh"] Oct 30 05:28:11.957561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:11.957474 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh"] Oct 30 05:28:11.957674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:11.957606 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:11.971491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:11.971461 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" Oct 30 05:28:12.022031 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.021998 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-xks-ca\") pod \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\" (UID: \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\") " Oct 30 05:28:12.022246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.022192 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-openshift-ca\") pod \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\" (UID: \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\") " Oct 30 05:28:12.022246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.022237 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027c5a3b-eabe-4622-a1e2-7294bfe4edda-kube-api-access-85f2x\" (UniqueName: \"kubernetes.io/projected/027c5a3b-eabe-4622-a1e2-7294bfe4edda-kube-api-access-85f2x\") pod \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\" (UID: \"027c5a3b-eabe-4622-a1e2-7294bfe4edda\") " Oct 30 05:28:12.022626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.022601 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-openshift-ca" pod "027c5a3b-eabe-4622-a1e2-7294bfe4edda" (UID: "027c5a3b-eabe-4622-a1e2-7294bfe4edda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:28:12.022742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.022718 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.022819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.022767 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.022875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.022850 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.022953 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.022935 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.023099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.023079 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8994c5b4-d21b-4b3f-864a-68fa175ab737-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.023282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.023264 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.023341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.023311 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcltz\" (UniqueName: \"kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-kube-api-access-rcltz\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.023457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.023431 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.023520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.023511 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.023655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.023642 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:12.024547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.024523 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-xks-ca" pod "027c5a3b-eabe-4622-a1e2-7294bfe4edda" (UID: "027c5a3b-eabe-4622-a1e2-7294bfe4edda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:28:12.024751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.024727 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027c5a3b-eabe-4622-a1e2-7294bfe4edda-kube-api-access-85f2x" pod "027c5a3b-eabe-4622-a1e2-7294bfe4edda" (UID: "027c5a3b-eabe-4622-a1e2-7294bfe4edda"). InnerVolumeSpecName "kube-api-access-85f2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:28:12.124981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.124944 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8994c5b4-d21b-4b3f-864a-68fa175ab737-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.125199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.125114 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.125199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.125168 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcltz\" (UniqueName: \"kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-kube-api-access-rcltz\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.125341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.125228 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.125683 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.125344 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.125774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.125569 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.125834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.125597 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.125834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.125653 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.125834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.125809 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8994c5b4-d21b-4b3f-864a-68fa175ab737-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.125834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.125827 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.126047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.125865 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.126047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.125890 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.126047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.125933 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.126047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.126036 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027c5a3b-eabe-4622-a1e2-7294bfe4edda-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:12.126261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.126050 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85f2x\" (UniqueName: \"kubernetes.io/projected/027c5a3b-eabe-4622-a1e2-7294bfe4edda-kube-api-access-85f2x\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:12.126261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.126067 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.128633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.128584 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.128942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.128920 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.148958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.148917 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcltz\" (UniqueName: \"kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-kube-api-access-rcltz\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.160886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.160856 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.282990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.282906 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:12.623818 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:28:12.623780 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8994c5b4_d21b_4b3f_864a_68fa175ab737.slice/crio-387ad3c91d14431e8c2d039b9b9021891045a5576763718ad449a7196d9243b2 WatchSource:0}: Error finding container 387ad3c91d14431e8c2d039b9b9021891045a5576763718ad449a7196d9243b2: Status 404 returned error can't find the container with id 387ad3c91d14431e8c2d039b9b9021891045a5576763718ad449a7196d9243b2 Oct 30 05:28:12.625864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.625834 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh"] Oct 30 05:28:12.628302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.628267 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:12.628414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.628334 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:12.628414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.628381 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:12.714136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.714095 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" event={"ID":"027c5a3b-eabe-4622-a1e2-7294bfe4edda","Type":"ContainerDied","Data":"ef1137fdd87875ef6755f48101871372bb16270775663a24055c06065ed42e97"} Oct 30 05:28:12.714136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.714123 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg" Oct 30 05:28:12.714338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.714151 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef1137fdd87875ef6755f48101871372bb16270775663a24055c06065ed42e97" Oct 30 05:28:12.718200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:12.718169 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" event={"ID":"8994c5b4-d21b-4b3f-864a-68fa175ab737","Type":"ContainerStarted","Data":"387ad3c91d14431e8c2d039b9b9021891045a5576763718ad449a7196d9243b2"} Oct 30 05:28:13.778883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:13.778839 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" event={"ID":"8994c5b4-d21b-4b3f-864a-68fa175ab737","Type":"ContainerStarted","Data":"24bddc0bd1383638807f8a334f2459e789ec8f708e9e706c2ab76aaf9eab0d7d"} Oct 30 05:28:14.283709 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:14.283342 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:14.285345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:14.285303 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body= Oct 30 05:28:14.285511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:14.285391 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" podUID="8994c5b4-d21b-4b3f-864a-68fa175ab737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" Oct 30 05:28:15.283637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:15.283599 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body= Oct 30 05:28:15.284057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:15.283656 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" podUID="8994c5b4-d21b-4b3f-864a-68fa175ab737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" Oct 30 05:28:16.283554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:16.283519 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body= Oct 30 05:28:16.283729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:16.283570 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" podUID="8994c5b4-d21b-4b3f-864a-68fa175ab737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" Oct 30 05:28:17.283424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:17.283318 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body= Oct 30 05:28:17.283424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:17.283398 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" podUID="8994c5b4-d21b-4b3f-864a-68fa175ab737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" Oct 30 05:28:17.846383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:17.841753 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" podStartSLOduration=6.84173462 podStartE2EDuration="6.84173462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:13.871135348 +0000 UTC m=+2618.398902135" watchObservedRunningTime="2026-10-30 05:28:17.84173462 +0000 UTC m=+2622.369501407" Oct 30 05:28:17.861985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:17.861957 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw"] Oct 30 05:28:17.863761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:17.863732 6587 cpu_manager.go:414] "containerMap: removing container" podUID="027c5a3b-eabe-4622-a1e2-7294bfe4edda" containerName="revoke-keys" Oct 30 05:28:17.863761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:17.863759 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027c5a3b-eabe-4622-a1e2-7294bfe4edda" containerName="revoke-keys" Oct 30 05:28:17.864072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:17.864053 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="027c5a3b-eabe-4622-a1e2-7294bfe4edda" containerName="revoke-keys" Oct 30 05:28:17.869403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:17.869381 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" Oct 30 05:28:17.878751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:17.878726 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw"] Oct 30 05:28:17.907932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:17.907898 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:28:17.908072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:17.908010 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:28:17.908072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:17.908046 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:28:17.908211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:17.908112 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:28:18.012996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.012961 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59916050-66bzw\" (UID: \"f1d30d45-ade0-4df6-8fa2-d07894965147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" Oct 30 05:28:18.013167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.013015 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59916050-66bzw\" (UID: \"f1d30d45-ade0-4df6-8fa2-d07894965147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" Oct 30 05:28:18.013214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.013174 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bfzn\" (UniqueName: \"kubernetes.io/projected/f1d30d45-ade0-4df6-8fa2-d07894965147-kube-api-access-8bfzn\") pod \"maas-api-revoke-sub-59916050-66bzw\" (UID: \"f1d30d45-ade0-4df6-8fa2-d07894965147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" Oct 30 05:28:18.115286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.115207 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59916050-66bzw\" (UID: \"f1d30d45-ade0-4df6-8fa2-d07894965147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" Oct 30 05:28:18.115286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.115259 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59916050-66bzw\" (UID: \"f1d30d45-ade0-4df6-8fa2-d07894965147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" Oct 30 05:28:18.115615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.115344 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bfzn\" (UniqueName: \"kubernetes.io/projected/f1d30d45-ade0-4df6-8fa2-d07894965147-kube-api-access-8bfzn\") pod \"maas-api-revoke-sub-59916050-66bzw\" (UID: \"f1d30d45-ade0-4df6-8fa2-d07894965147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" Oct 30 05:28:18.139377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.116379 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59916050-66bzw\" (UID: \"f1d30d45-ade0-4df6-8fa2-d07894965147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" Oct 30 05:28:18.153203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.144018 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bfzn\" (UniqueName: \"kubernetes.io/projected/f1d30d45-ade0-4df6-8fa2-d07894965147-kube-api-access-8bfzn\") pod \"maas-api-revoke-sub-59916050-66bzw\" (UID: \"f1d30d45-ade0-4df6-8fa2-d07894965147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" Oct 30 05:28:18.154633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.154521 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59916050-66bzw\" (UID: \"f1d30d45-ade0-4df6-8fa2-d07894965147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" Oct 30 05:28:18.187440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.187410 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" Oct 30 05:28:18.283908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.283872 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body= Oct 30 05:28:18.284120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.283931 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" podUID="8994c5b4-d21b-4b3f-864a-68fa175ab737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" Oct 30 05:28:18.564699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.564218 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw"] Oct 30 05:28:18.857426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:18.857389 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" event={"ID":"f1d30d45-ade0-4df6-8fa2-d07894965147","Type":"ContainerStarted","Data":"f252a7e9853ab9e824be45eedce3de4e257f87d17731f3c06d3a710f9afa3bf8"} Oct 30 05:28:19.289317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:19.289222 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:19.289678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:19.289653 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:19.290386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:19.290331 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:28:19.868974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:19.868938 6587 generic.go:356] "Generic (PLEG): container finished" podID="f1d30d45-ade0-4df6-8fa2-d07894965147" containerID="32bfd6dead1de4d9af55f224071ae8f15b5df48506e100837687f06307d3e34e" exitCode=0 Oct 30 05:28:19.869457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:19.869032 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" event={"ID":"f1d30d45-ade0-4df6-8fa2-d07894965147","Type":"ContainerDied","Data":"32bfd6dead1de4d9af55f224071ae8f15b5df48506e100837687f06307d3e34e"} Oct 30 05:28:21.103708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.103676 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" Oct 30 05:28:21.211274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.211178 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d30d45-ade0-4df6-8fa2-d07894965147-kube-api-access-8bfzn\" (UniqueName: \"kubernetes.io/projected/f1d30d45-ade0-4df6-8fa2-d07894965147-kube-api-access-8bfzn\") pod \"f1d30d45-ade0-4df6-8fa2-d07894965147\" (UID: \"f1d30d45-ade0-4df6-8fa2-d07894965147\") " Oct 30 05:28:21.211274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.211250 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-openshift-ca\") pod \"f1d30d45-ade0-4df6-8fa2-d07894965147\" (UID: \"f1d30d45-ade0-4df6-8fa2-d07894965147\") " Oct 30 05:28:21.211548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.211526 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-xks-ca\") pod \"f1d30d45-ade0-4df6-8fa2-d07894965147\" (UID: \"f1d30d45-ade0-4df6-8fa2-d07894965147\") " Oct 30 05:28:21.212143 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.212116 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-openshift-ca" pod "f1d30d45-ade0-4df6-8fa2-d07894965147" (UID: "f1d30d45-ade0-4df6-8fa2-d07894965147"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:28:21.212561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.212541 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:21.214213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.214181 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-xks-ca" pod "f1d30d45-ade0-4df6-8fa2-d07894965147" (UID: "f1d30d45-ade0-4df6-8fa2-d07894965147"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:28:21.214213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.214185 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d30d45-ade0-4df6-8fa2-d07894965147-kube-api-access-8bfzn" pod "f1d30d45-ade0-4df6-8fa2-d07894965147" (UID: "f1d30d45-ade0-4df6-8fa2-d07894965147"). InnerVolumeSpecName "kube-api-access-8bfzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:28:21.314653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.314625 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d30d45-ade0-4df6-8fa2-d07894965147-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:21.314653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.314651 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bfzn\" (UniqueName: \"kubernetes.io/projected/f1d30d45-ade0-4df6-8fa2-d07894965147-kube-api-access-8bfzn\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:21.705678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.705646 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" Oct 30 05:28:21.705881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.705729 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" Oct 30 05:28:21.889039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.888963 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" Oct 30 05:28:21.889189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.888957 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw" event={"ID":"f1d30d45-ade0-4df6-8fa2-d07894965147","Type":"ContainerDied","Data":"f252a7e9853ab9e824be45eedce3de4e257f87d17731f3c06d3a710f9afa3bf8"} Oct 30 05:28:21.889189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:21.889097 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f252a7e9853ab9e824be45eedce3de4e257f87d17731f3c06d3a710f9afa3bf8" Oct 30 05:28:23.506672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.506639 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn"] Oct 30 05:28:23.507806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.507786 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f1d30d45-ade0-4df6-8fa2-d07894965147" containerName="revoke-keys" Oct 30 05:28:23.507806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.507807 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d30d45-ade0-4df6-8fa2-d07894965147" containerName="revoke-keys" Oct 30 05:28:23.507976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.507966 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d30d45-ade0-4df6-8fa2-d07894965147" containerName="revoke-keys" Oct 30 05:28:23.512327 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.512308 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.535940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.535909 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-44bc69-openshift-default-dockercfg-nfx8p\"" Oct 30 05:28:23.558736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.558701 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn"] Oct 30 05:28:23.568202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.568166 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-data\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.568432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.568403 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-credential-socket\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.568574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.568488 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istiod-ca-cert\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.568651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.568582 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-socket\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.568715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.568686 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-token\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.568778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.568719 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-envoy\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.568838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.568814 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-podinfo\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.568897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.568842 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-certs\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.568897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.568874 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6698\" (UniqueName: \"kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-kube-api-access-q6698\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.670563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.670533 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-credential-socket\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.670742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.670595 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istiod-ca-cert\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.670742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.670662 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-socket\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.670742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.670727 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-token\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.670886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.670757 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-envoy\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.670886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.670824 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-podinfo\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.670886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.670851 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-certs\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.670886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.670880 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6698\" (UniqueName: \"kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-kube-api-access-q6698\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.671139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.670976 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-credential-socket\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.671139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.671077 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-socket\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.671233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.671199 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-data\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.671291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.671229 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-certs\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.671613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.671594 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istiod-ca-cert\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.671752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.671665 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-data\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.673073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.673046 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-envoy\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.673302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.673283 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-podinfo\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.692510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.692481 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-token\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.693984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.693966 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6698\" (UniqueName: \"kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-kube-api-access-q6698\") pod \"e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:23.831702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:23.831669 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:24.161504 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:28:24.161471 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecec1c2_d569_4d47_af57_6c4398bd1abb.slice/crio-126cf3de91bc3430a9189f9ff433ec14e7cf892fcc429beec439b778b06bcd89 WatchSource:0}: Error finding container 126cf3de91bc3430a9189f9ff433ec14e7cf892fcc429beec439b778b06bcd89: Status 404 returned error can't find the container with id 126cf3de91bc3430a9189f9ff433ec14e7cf892fcc429beec439b778b06bcd89 Oct 30 05:28:24.164839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:24.164813 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn"] Oct 30 05:28:24.165810 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:24.165767 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:24.165916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:24.165861 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:24.165916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:24.165910 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:28:24.990750 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:24.990710 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" event={"ID":"9ecec1c2-d569-4d47-af57-6c4398bd1abb","Type":"ContainerStarted","Data":"98bfa8c53a9d433f976b4973b3d20e02c13f6c80fbf12e8af057df3b5bcfffde"} Oct 30 05:28:24.991382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:24.991332 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" event={"ID":"9ecec1c2-d569-4d47-af57-6c4398bd1abb","Type":"ContainerStarted","Data":"126cf3de91bc3430a9189f9ff433ec14e7cf892fcc429beec439b778b06bcd89"} Oct 30 05:28:25.077522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:25.077176 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" podStartSLOduration=2.077156338 podStartE2EDuration="2.077156338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:25.072921021 +0000 UTC m=+2629.600687807" watchObservedRunningTime="2026-10-30 05:28:25.077156338 +0000 UTC m=+2629.604923127" Oct 30 05:28:25.837382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:25.836264 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:25.852382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:25.851469 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Oct 30 05:28:25.852382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:25.851520 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" podUID="9ecec1c2-d569-4d47-af57-6c4398bd1abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Oct 30 05:28:25.992911 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:28:25.992828 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:28:26.838384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:26.837760 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Oct 30 05:28:26.838384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:26.837822 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" podUID="9ecec1c2-d569-4d47-af57-6c4398bd1abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Oct 30 05:28:27.844382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:27.841748 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Oct 30 05:28:27.844382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:27.841804 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" podUID="9ecec1c2-d569-4d47-af57-6c4398bd1abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Oct 30 05:28:28.832812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:28.832771 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Oct 30 05:28:28.833078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:28.833045 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" podUID="9ecec1c2-d569-4d47-af57-6c4398bd1abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Oct 30 05:28:29.836383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:29.833675 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Oct 30 05:28:29.836383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:29.833736 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" podUID="9ecec1c2-d569-4d47-af57-6c4398bd1abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Oct 30 05:28:31.004139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:31.004079 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:31.004726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:31.004621 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:31.005806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:31.005779 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:28:32.040200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:32.040166 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn"] Oct 30 05:28:32.045077 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:32.045049 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da31fc8b-htszn"] Oct 30 05:28:33.616387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:33.616328 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0805112-162f-41d2-82fc-544db635b227" path="/var/lib/kubelet/pods/b0805112-162f-41d2-82fc-544db635b227/volumes" Oct 30 05:28:41.281271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.281227 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl"] Oct 30 05:28:41.290206 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.289632 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.308637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.308603 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-0dba31\"/\"openshift-service-ca.crt\"" Oct 30 05:28:41.308814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.308678 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-0dba31\"/\"kube-root-ca.crt\"" Oct 30 05:28:41.309579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.309551 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl"] Oct 30 05:28:41.314991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.314965 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-0dba31\"/\"sub-test-model-0dba31-kserve-self-signed-certs\"" Oct 30 05:28:41.315155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.315106 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-0dba31\"/\"default-dockercfg-pqhsj\"" Oct 30 05:28:41.331112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.331077 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79db524a-27ca-40a0-9599-a3a172b15ecb-tls-certs\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.331266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.331205 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-model-cache\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.331266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.331256 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-dshm\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.331434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.331278 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-home\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.331434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.331302 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz857\" (UniqueName: \"kubernetes.io/projected/79db524a-27ca-40a0-9599-a3a172b15ecb-kube-api-access-wz857\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.331694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.331669 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-tmp-dir\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.433058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.433026 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-tmp-dir\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.433264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.433205 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79db524a-27ca-40a0-9599-a3a172b15ecb-tls-certs\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.433377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.433289 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-model-cache\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.433377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.433337 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-dshm\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.433546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.433381 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-home\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.433546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.433416 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz857\" (UniqueName: \"kubernetes.io/projected/79db524a-27ca-40a0-9599-a3a172b15ecb-kube-api-access-wz857\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.433546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.433477 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-tmp-dir\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.433739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.433721 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-model-cache\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.434060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.434038 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-home\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.436061 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.436038 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-dshm\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.436243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.436228 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79db524a-27ca-40a0-9599-a3a172b15ecb-tls-certs\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.455203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.455179 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz857\" (UniqueName: \"kubernetes.io/projected/79db524a-27ca-40a0-9599-a3a172b15ecb-kube-api-access-wz857\") pod \"sub-test-model-0dba31-kserve-6f76b77f95-mxzwl\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.616077 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.613446 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:41.801996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.801958 6587 scope.go:122] "RemoveContainer" containerID="f5aee60dad9ac1c7f5c2fdf8e04ba0a83414e25dba60c0ac0aab3f584dabebfa" Oct 30 05:28:41.818153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.817810 6587 scope.go:122] "RemoveContainer" containerID="62a35cc4baeff17674b4aa5688b23af6c256cdb99c74525acd30e32d8eedf59c" Oct 30 05:28:41.833227 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.833208 6587 scope.go:122] "RemoveContainer" containerID="64b6bdfad00018cd8eb156e1cc406f9d3332a30a11250a252481bd9c898d0395" Oct 30 05:28:41.847468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.845592 6587 scope.go:122] "RemoveContainer" containerID="dfcc719f90826824832851b7244774bbf26af809697d03c0e2ec4689e58656b6" Oct 30 05:28:41.864275 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.864245 6587 scope.go:122] "RemoveContainer" containerID="6fbbbae98753ad6b376c8df26aae858f0e83e77816b4c168bdef9b9005a0b071" Oct 30 05:28:41.878383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:41.878325 6587 scope.go:122] "RemoveContainer" containerID="c073912bcd3f599c7880d29b267db56be47755a5af33256b5f194dfae00cb0b3" Oct 30 05:28:42.025478 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:28:42.024973 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79db524a_27ca_40a0_9599_a3a172b15ecb.slice/crio-3f8bf197aff9fd29708ecb41dc8097e91853e3a6a49fb372a9f60fdba50cb48f WatchSource:0}: Error finding container 3f8bf197aff9fd29708ecb41dc8097e91853e3a6a49fb372a9f60fdba50cb48f: Status 404 returned error can't find the container with id 3f8bf197aff9fd29708ecb41dc8097e91853e3a6a49fb372a9f60fdba50cb48f Oct 30 05:28:42.026927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:42.026898 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl"] Oct 30 05:28:42.239755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:42.239667 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" event={"ID":"79db524a-27ca-40a0-9599-a3a172b15ecb","Type":"ContainerStarted","Data":"729d1aac14e1a36f4994282f755f9a0461c4987cf09813c38b4b63355d05d243"} Oct 30 05:28:42.239755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:42.239708 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" event={"ID":"79db524a-27ca-40a0-9599-a3a172b15ecb","Type":"ContainerStarted","Data":"3f8bf197aff9fd29708ecb41dc8097e91853e3a6a49fb372a9f60fdba50cb48f"} Oct 30 05:28:42.285762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:42.285701 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" podStartSLOduration=1.285682688 podStartE2EDuration="1.285682688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:42.283905143 +0000 UTC m=+2646.811671928" watchObservedRunningTime="2026-10-30 05:28:42.285682688 +0000 UTC m=+2646.813449481" Oct 30 05:28:42.613870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:42.613836 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:43.111145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:43.110958 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l"] Oct 30 05:28:45.032049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:45.032010 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf"] Oct 30 05:28:45.045855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:45.045822 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85608484-9ggnf"] Oct 30 05:28:45.593112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:45.593076 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc0a3b92-9af2-4868-8c56-6822eed68522" path="/var/lib/kubelet/pods/fc0a3b92-9af2-4868-8c56-6822eed68522/volumes" Oct 30 05:28:51.614660 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:51.614628 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:51.633209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:51.633177 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:52.434121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:52.434089 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:28:55.988774 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:28:55.988735 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:28:58.141815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.141762 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" podUID="40b7c36f-e157-4cc4-a04e-962fa1a7cc77" containerName="main" containerID="cri-o://cdd98a7ccde20d19aec50711daee7bcb6864d351ebc62060b367d08ae445c250" gracePeriod=45 Oct 30 05:28:58.487660 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.487624 6587 generic.go:356] "Generic (PLEG): container finished" podID="40b7c36f-e157-4cc4-a04e-962fa1a7cc77" containerID="cdd98a7ccde20d19aec50711daee7bcb6864d351ebc62060b367d08ae445c250" exitCode=0 Oct 30 05:28:58.487845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.487697 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" event={"ID":"40b7c36f-e157-4cc4-a04e-962fa1a7cc77","Type":"ContainerDied","Data":"cdd98a7ccde20d19aec50711daee7bcb6864d351ebc62060b367d08ae445c250"} Oct 30 05:28:58.487845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.487741 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" event={"ID":"40b7c36f-e157-4cc4-a04e-962fa1a7cc77","Type":"ContainerDied","Data":"87a58ed989425512f80cf83358ca0d3ff135b2115c2185b488f416e1305086a6"} Oct 30 05:28:58.487845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.487752 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87a58ed989425512f80cf83358ca0d3ff135b2115c2185b488f416e1305086a6" Oct 30 05:28:58.494799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.494777 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:28:58.610129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.610090 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tls-certs\" (UniqueName: \"kubernetes.io/secret/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tls-certs\") pod \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " Oct 30 05:28:58.610434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.610402 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-kube-api-access-zwgkt\" (UniqueName: \"kubernetes.io/projected/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-kube-api-access-zwgkt\") pod \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " Oct 30 05:28:58.610554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.610462 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-home\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-home\") pod \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " Oct 30 05:28:58.610554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.610533 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-model-cache\") pod \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " Oct 30 05:28:58.610674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.610615 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tmp-dir\") pod \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " Oct 30 05:28:58.610817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.610781 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-home" pod "40b7c36f-e157-4cc4-a04e-962fa1a7cc77" (UID: "40b7c36f-e157-4cc4-a04e-962fa1a7cc77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:58.610891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.610816 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-dshm\") pod \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\" (UID: \"40b7c36f-e157-4cc4-a04e-962fa1a7cc77\") " Oct 30 05:28:58.611009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.610979 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-model-cache" pod "40b7c36f-e157-4cc4-a04e-962fa1a7cc77" (UID: "40b7c36f-e157-4cc4-a04e-962fa1a7cc77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:58.611184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.611012 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tmp-dir" pod "40b7c36f-e157-4cc4-a04e-962fa1a7cc77" (UID: "40b7c36f-e157-4cc4-a04e-962fa1a7cc77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:58.611865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.611841 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:58.611942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.611873 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:58.611942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.611891 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:58.612755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.612726 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tls-certs" pod "40b7c36f-e157-4cc4-a04e-962fa1a7cc77" (UID: "40b7c36f-e157-4cc4-a04e-962fa1a7cc77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:28:58.613284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.613248 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-kube-api-access-zwgkt" pod "40b7c36f-e157-4cc4-a04e-962fa1a7cc77" (UID: "40b7c36f-e157-4cc4-a04e-962fa1a7cc77"). InnerVolumeSpecName "kube-api-access-zwgkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:28:58.613558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.613536 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-dshm" pod "40b7c36f-e157-4cc4-a04e-962fa1a7cc77" (UID: "40b7c36f-e157-4cc4-a04e-962fa1a7cc77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:28:58.714507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.714402 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwgkt\" (UniqueName: \"kubernetes.io/projected/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-kube-api-access-zwgkt\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:58.714507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.714439 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:58.714507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:58.714453 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40b7c36f-e157-4cc4-a04e-962fa1a7cc77-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:28:59.496088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:59.496057 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l" Oct 30 05:28:59.575793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:59.575754 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l"] Oct 30 05:28:59.595968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:28:59.595934 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-262c10/auth-test-model-262c10-kserve-76f9465fcc-gt65l"] Oct 30 05:29:01.585712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:01.585678 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b7c36f-e157-4cc4-a04e-962fa1a7cc77" path="/var/lib/kubelet/pods/40b7c36f-e157-4cc4-a04e-962fa1a7cc77/volumes" Oct 30 05:29:05.532393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:05.531014 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph"] Oct 30 05:29:05.532393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:05.531341 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" podUID="62bcdd5c-8075-434d-89d4-071ea3f6b9d6" containerName="istio-proxy" containerID="cri-o://bc33126225acad9d5475c694a8382f69f54db764a8fe5183d3af40d2c89693d9" gracePeriod=30 Oct 30 05:29:08.404333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.404293 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq"] Oct 30 05:29:08.405215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.405198 6587 cpu_manager.go:414] "containerMap: removing container" podUID="40b7c36f-e157-4cc4-a04e-962fa1a7cc77" containerName="main" Oct 30 05:29:08.405276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.405216 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b7c36f-e157-4cc4-a04e-962fa1a7cc77" containerName="main" Oct 30 05:29:08.405435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.405424 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b7c36f-e157-4cc4-a04e-962fa1a7cc77" containerName="main" Oct 30 05:29:08.409742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.409719 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" Oct 30 05:29:08.458724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.458688 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:29:08.459092 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.458736 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:29:08.459396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.458763 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:29:08.459396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.458799 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:29:08.466086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.466045 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq"] Oct 30 05:29:08.502698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.502653 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4437a4e7-j4pkq\" (UID: \"c237656b-07e0-44e4-acd8-5d3b3d6da565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" Oct 30 05:29:08.503218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.503189 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tzw4\" (UniqueName: \"kubernetes.io/projected/c237656b-07e0-44e4-acd8-5d3b3d6da565-kube-api-access-9tzw4\") pod \"maas-api-revoke-sub-4437a4e7-j4pkq\" (UID: \"c237656b-07e0-44e4-acd8-5d3b3d6da565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" Oct 30 05:29:08.511408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.503735 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4437a4e7-j4pkq\" (UID: \"c237656b-07e0-44e4-acd8-5d3b3d6da565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" Oct 30 05:29:08.613248 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.613211 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tzw4\" (UniqueName: \"kubernetes.io/projected/c237656b-07e0-44e4-acd8-5d3b3d6da565-kube-api-access-9tzw4\") pod \"maas-api-revoke-sub-4437a4e7-j4pkq\" (UID: \"c237656b-07e0-44e4-acd8-5d3b3d6da565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" Oct 30 05:29:08.613545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.613520 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4437a4e7-j4pkq\" (UID: \"c237656b-07e0-44e4-acd8-5d3b3d6da565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" Oct 30 05:29:08.613628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.613562 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4437a4e7-j4pkq\" (UID: \"c237656b-07e0-44e4-acd8-5d3b3d6da565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" Oct 30 05:29:08.614308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.614286 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4437a4e7-j4pkq\" (UID: \"c237656b-07e0-44e4-acd8-5d3b3d6da565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" Oct 30 05:29:08.648342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.648308 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4437a4e7-j4pkq\" (UID: \"c237656b-07e0-44e4-acd8-5d3b3d6da565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" Oct 30 05:29:08.650858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.650828 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tzw4\" (UniqueName: \"kubernetes.io/projected/c237656b-07e0-44e4-acd8-5d3b3d6da565-kube-api-access-9tzw4\") pod \"maas-api-revoke-sub-4437a4e7-j4pkq\" (UID: \"c237656b-07e0-44e4-acd8-5d3b3d6da565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" Oct 30 05:29:08.761458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:08.761373 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" Oct 30 05:29:09.093497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:09.093460 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq"] Oct 30 05:29:09.096428 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:29:09.096353 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc237656b_07e0_44e4_acd8_5d3b3d6da565.slice/crio-7cbdbdaf1b3f069b6e7a4d6f77483cc5c69e8e0c6aa7acc6d826efb25c9e2105 WatchSource:0}: Error finding container 7cbdbdaf1b3f069b6e7a4d6f77483cc5c69e8e0c6aa7acc6d826efb25c9e2105: Status 404 returned error can't find the container with id 7cbdbdaf1b3f069b6e7a4d6f77483cc5c69e8e0c6aa7acc6d826efb25c9e2105 Oct 30 05:29:09.157379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:09.156457 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" event={"ID":"c237656b-07e0-44e4-acd8-5d3b3d6da565","Type":"ContainerStarted","Data":"7cbdbdaf1b3f069b6e7a4d6f77483cc5c69e8e0c6aa7acc6d826efb25c9e2105"} Oct 30 05:29:10.168266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:10.168226 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" event={"ID":"c237656b-07e0-44e4-acd8-5d3b3d6da565","Type":"ContainerStarted","Data":"966ec984ddb09ae81ad4c142bdf84a13685efd00682a0d9ac20b43d4ec040e2b"} Oct 30 05:29:10.225389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:10.215444 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" podStartSLOduration=2.215425928 podStartE2EDuration="2.215425928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:10.213233506 +0000 UTC m=+2674.741000291" watchObservedRunningTime="2026-10-30 05:29:10.215425928 +0000 UTC m=+2674.743192717" Oct 30 05:29:11.193411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.190445 6587 generic.go:356] "Generic (PLEG): container finished" podID="62bcdd5c-8075-434d-89d4-071ea3f6b9d6" containerID="bc33126225acad9d5475c694a8382f69f54db764a8fe5183d3af40d2c89693d9" exitCode=0 Oct 30 05:29:11.193411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.190618 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" event={"ID":"62bcdd5c-8075-434d-89d4-071ea3f6b9d6","Type":"ContainerDied","Data":"bc33126225acad9d5475c694a8382f69f54db764a8fe5183d3af40d2c89693d9"} Oct 30 05:29:11.193411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.192608 6587 generic.go:356] "Generic (PLEG): container finished" podID="c237656b-07e0-44e4-acd8-5d3b3d6da565" containerID="966ec984ddb09ae81ad4c142bdf84a13685efd00682a0d9ac20b43d4ec040e2b" exitCode=0 Oct 30 05:29:11.193411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.192718 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" event={"ID":"c237656b-07e0-44e4-acd8-5d3b3d6da565","Type":"ContainerDied","Data":"966ec984ddb09ae81ad4c142bdf84a13685efd00682a0d9ac20b43d4ec040e2b"} Oct 30 05:29:11.358778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.358715 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:29:11.412176 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.412135 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-token\" (UniqueName: \"kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-token\") pod \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " Oct 30 05:29:11.412443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.412178 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-podinfo\") pod \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " Oct 30 05:29:11.412443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.412303 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-credential-socket\") pod \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " Oct 30 05:29:11.412443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.412416 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-socket\") pod \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " Oct 30 05:29:11.412655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.412590 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-envoy\") pod \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " Oct 30 05:29:11.412655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.412624 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-certs\") pod \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " Oct 30 05:29:11.412774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.412629 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-credential-socket" pod "62bcdd5c-8075-434d-89d4-071ea3f6b9d6" (UID: "62bcdd5c-8075-434d-89d4-071ea3f6b9d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:29:11.412774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.412683 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-data\") pod \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " Oct 30 05:29:11.412774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.412742 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istiod-ca-cert\") pod \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " Oct 30 05:29:11.412956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.412867 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-certs" pod "62bcdd5c-8075-434d-89d4-071ea3f6b9d6" (UID: "62bcdd5c-8075-434d-89d4-071ea3f6b9d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:29:11.412956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.412891 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-kube-api-access-h284r\" (UniqueName: \"kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-kube-api-access-h284r\") pod \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\" (UID: \"62bcdd5c-8075-434d-89d4-071ea3f6b9d6\") " Oct 30 05:29:11.412956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.412900 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-socket" pod "62bcdd5c-8075-434d-89d4-071ea3f6b9d6" (UID: "62bcdd5c-8075-434d-89d4-071ea3f6b9d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:29:11.413129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.412960 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-data" pod "62bcdd5c-8075-434d-89d4-071ea3f6b9d6" (UID: "62bcdd5c-8075-434d-89d4-071ea3f6b9d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:29:11.413236 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.413196 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istiod-ca-cert" pod "62bcdd5c-8075-434d-89d4-071ea3f6b9d6" (UID: "62bcdd5c-8075-434d-89d4-071ea3f6b9d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:29:11.413809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.413726 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:11.413809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.413747 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:11.413809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.413760 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:11.413809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.413775 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:11.413809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.413787 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:11.414882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.414835 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-podinfo" pod "62bcdd5c-8075-434d-89d4-071ea3f6b9d6" (UID: "62bcdd5c-8075-434d-89d4-071ea3f6b9d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:29:11.457382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.420773 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-kube-api-access-h284r" pod "62bcdd5c-8075-434d-89d4-071ea3f6b9d6" (UID: "62bcdd5c-8075-434d-89d4-071ea3f6b9d6"). InnerVolumeSpecName "kube-api-access-h284r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:29:11.457382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.420925 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-token" pod "62bcdd5c-8075-434d-89d4-071ea3f6b9d6" (UID: "62bcdd5c-8075-434d-89d4-071ea3f6b9d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:29:11.483200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.480618 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-envoy" pod "62bcdd5c-8075-434d-89d4-071ea3f6b9d6" (UID: "62bcdd5c-8075-434d-89d4-071ea3f6b9d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:29:11.517835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.516045 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:11.517835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.516084 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h284r\" (UniqueName: \"kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-kube-api-access-h284r\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:11.517835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.516099 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:11.517835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:11.516114 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62bcdd5c-8075-434d-89d4-071ea3f6b9d6-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:12.203202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.203175 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" Oct 30 05:29:12.203669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.203210 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph" event={"ID":"62bcdd5c-8075-434d-89d4-071ea3f6b9d6","Type":"ContainerDied","Data":"900f6435277f7bb327b729635f9bb3d6466bcff587433e3ff882baf14f444aec"} Oct 30 05:29:12.203669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.203273 6587 scope.go:122] "RemoveContainer" containerID="bc33126225acad9d5475c694a8382f69f54db764a8fe5183d3af40d2c89693d9" Oct 30 05:29:12.293380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.289408 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph"] Oct 30 05:29:12.308603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.308566 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph"] Oct 30 05:29:12.488993 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.488955 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" Oct 30 05:29:12.536996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.536695 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-openshift-ca\") pod \"c237656b-07e0-44e4-acd8-5d3b3d6da565\" (UID: \"c237656b-07e0-44e4-acd8-5d3b3d6da565\") " Oct 30 05:29:12.536996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.536879 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c237656b-07e0-44e4-acd8-5d3b3d6da565-kube-api-access-9tzw4\" (UniqueName: \"kubernetes.io/projected/c237656b-07e0-44e4-acd8-5d3b3d6da565-kube-api-access-9tzw4\") pod \"c237656b-07e0-44e4-acd8-5d3b3d6da565\" (UID: \"c237656b-07e0-44e4-acd8-5d3b3d6da565\") " Oct 30 05:29:12.537268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.537033 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-xks-ca\") pod \"c237656b-07e0-44e4-acd8-5d3b3d6da565\" (UID: \"c237656b-07e0-44e4-acd8-5d3b3d6da565\") " Oct 30 05:29:12.545495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.539620 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-openshift-ca" pod "c237656b-07e0-44e4-acd8-5d3b3d6da565" (UID: "c237656b-07e0-44e4-acd8-5d3b3d6da565"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:29:12.545495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.543054 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-xks-ca" pod "c237656b-07e0-44e4-acd8-5d3b3d6da565" (UID: "c237656b-07e0-44e4-acd8-5d3b3d6da565"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:29:12.557615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.557567 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c237656b-07e0-44e4-acd8-5d3b3d6da565-kube-api-access-9tzw4" pod "c237656b-07e0-44e4-acd8-5d3b3d6da565" (UID: "c237656b-07e0-44e4-acd8-5d3b3d6da565"). InnerVolumeSpecName "kube-api-access-9tzw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:29:12.639536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.639502 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:12.639536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.639533 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c237656b-07e0-44e4-acd8-5d3b3d6da565-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:12.639536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:12.639545 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tzw4\" (UniqueName: \"kubernetes.io/projected/c237656b-07e0-44e4-acd8-5d3b3d6da565-kube-api-access-9tzw4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:13.219181 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:13.219138 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" event={"ID":"c237656b-07e0-44e4-acd8-5d3b3d6da565","Type":"ContainerDied","Data":"7cbdbdaf1b3f069b6e7a4d6f77483cc5c69e8e0c6aa7acc6d826efb25c9e2105"} Oct 30 05:29:13.219714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:13.219202 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cbdbdaf1b3f069b6e7a4d6f77483cc5c69e8e0c6aa7acc6d826efb25c9e2105" Oct 30 05:29:13.219714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:13.219265 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq" Oct 30 05:29:13.381052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:13.381011 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw"] Oct 30 05:29:13.642999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:13.642961 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62bcdd5c-8075-434d-89d4-071ea3f6b9d6" path="/var/lib/kubelet/pods/62bcdd5c-8075-434d-89d4-071ea3f6b9d6/volumes" Oct 30 05:29:18.749389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:18.749337 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg"] Oct 30 05:29:18.750320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:18.750300 6587 cpu_manager.go:414] "containerMap: removing container" podUID="62bcdd5c-8075-434d-89d4-071ea3f6b9d6" containerName="istio-proxy" Oct 30 05:29:18.750320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:18.750320 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62bcdd5c-8075-434d-89d4-071ea3f6b9d6" containerName="istio-proxy" Oct 30 05:29:18.750504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:18.750378 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c237656b-07e0-44e4-acd8-5d3b3d6da565" containerName="revoke-keys" Oct 30 05:29:18.750504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:18.750384 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c237656b-07e0-44e4-acd8-5d3b3d6da565" containerName="revoke-keys" Oct 30 05:29:18.750625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:18.750509 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c237656b-07e0-44e4-acd8-5d3b3d6da565" containerName="revoke-keys" Oct 30 05:29:18.750625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:18.750522 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="62bcdd5c-8075-434d-89d4-071ea3f6b9d6" containerName="istio-proxy" Oct 30 05:29:19.011926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.011822 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg"] Oct 30 05:29:19.043799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.043771 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59046973-fnjkg\" (UID: \"2451f7cd-4642-4831-8739-78d9390298e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" Oct 30 05:29:19.043968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.043853 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59046973-fnjkg\" (UID: \"2451f7cd-4642-4831-8739-78d9390298e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" Oct 30 05:29:19.054126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.054102 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q"] Oct 30 05:29:19.132504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.132462 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b259p\" (UniqueName: \"kubernetes.io/projected/2451f7cd-4642-4831-8739-78d9390298e3-kube-api-access-b259p\") pod \"maas-api-revoke-sub-59046973-fnjkg\" (UID: \"2451f7cd-4642-4831-8739-78d9390298e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" Oct 30 05:29:19.132966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.132941 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" Oct 30 05:29:19.148515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.148464 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q"] Oct 30 05:29:19.148749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.148589 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.233383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.222140 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:29:19.233383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.222248 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:29:19.233383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.222376 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:29:19.233383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.222451 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:29:19.249408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.249167 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59046973-fnjkg\" (UID: \"2451f7cd-4642-4831-8739-78d9390298e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" Oct 30 05:29:19.249408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.249218 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.249408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.249309 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.249408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.249383 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.299786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.249456 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b259p\" (UniqueName: \"kubernetes.io/projected/2451f7cd-4642-4831-8739-78d9390298e3-kube-api-access-b259p\") pod \"maas-api-revoke-sub-59046973-fnjkg\" (UID: \"2451f7cd-4642-4831-8739-78d9390298e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" Oct 30 05:29:19.299786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.249534 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l48tn\" (UniqueName: \"kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-kube-api-access-l48tn\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.299786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.249641 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.299786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.249754 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.299786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.249792 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.299786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.249927 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.299786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.249984 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.299786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.250026 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59046973-fnjkg\" (UID: \"2451f7cd-4642-4831-8739-78d9390298e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" Oct 30 05:29:19.299786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.281304 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59046973-fnjkg\" (UID: \"2451f7cd-4642-4831-8739-78d9390298e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" Oct 30 05:29:19.325418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.323915 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59046973-fnjkg\" (UID: \"2451f7cd-4642-4831-8739-78d9390298e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" Oct 30 05:29:19.366883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.365140 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.366883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.365289 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l48tn\" (UniqueName: \"kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-kube-api-access-l48tn\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.366883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.365413 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.366883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.365521 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.366883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.365558 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.366883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.365682 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.366883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.365735 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.366883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.365802 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.366883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.365877 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.366883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.366498 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.366883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.366701 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.366883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.366764 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.367647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.367007 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.390377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.387820 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.408382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.399964 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.462448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.453425 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l48tn\" (UniqueName: \"kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-kube-api-access-l48tn\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.496951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.495772 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.515065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.515029 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.586181 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.515980 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b259p\" (UniqueName: \"kubernetes.io/projected/2451f7cd-4642-4831-8739-78d9390298e3-kube-api-access-b259p\") pod \"maas-api-revoke-sub-59046973-fnjkg\" (UID: \"2451f7cd-4642-4831-8739-78d9390298e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" Oct 30 05:29:19.623077 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.623043 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:19.792575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:19.792540 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" Oct 30 05:29:21.813405 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:29:21.811702 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2451f7cd_4642_4831_8739_78d9390298e3.slice/crio-27b0bdbaf840ef235c7d9d64b9fa7b42d5f69001cdd28057f7c4082ebb0f7238 WatchSource:0}: Error finding container 27b0bdbaf840ef235c7d9d64b9fa7b42d5f69001cdd28057f7c4082ebb0f7238: Status 404 returned error can't find the container with id 27b0bdbaf840ef235c7d9d64b9fa7b42d5f69001cdd28057f7c4082ebb0f7238 Oct 30 05:29:21.815882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:21.815842 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg"] Oct 30 05:29:21.847313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:21.847287 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q"] Oct 30 05:29:21.853898 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:21.853858 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:29:21.854044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:21.853958 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:29:21.854044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:21.853999 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:29:22.303049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:22.303005 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm"] Oct 30 05:29:22.353265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:22.353231 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w"] Oct 30 05:29:22.448534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:22.448417 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" event={"ID":"9c59e0f2-7aca-4982-a50f-9b28382c6e8e","Type":"ContainerStarted","Data":"4e04d7d67806e2be6ce9dcba8b6fc196f8c11e33fa874d9d6c87af421eb9e702"} Oct 30 05:29:22.448534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:22.448496 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" event={"ID":"9c59e0f2-7aca-4982-a50f-9b28382c6e8e","Type":"ContainerStarted","Data":"cbb15c5cfea48749fc629941a4dde9d5670b1599160ac4c5c00656de2ecf8bcf"} Oct 30 05:29:22.456422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:22.454959 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" event={"ID":"2451f7cd-4642-4831-8739-78d9390298e3","Type":"ContainerStarted","Data":"e345a84edf7d30bac0d8dc113fad51286b4de0c2f71bf54757e525d1f49c9b23"} Oct 30 05:29:22.456422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:22.455023 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" event={"ID":"2451f7cd-4642-4831-8739-78d9390298e3","Type":"ContainerStarted","Data":"27b0bdbaf840ef235c7d9d64b9fa7b42d5f69001cdd28057f7c4082ebb0f7238"} Oct 30 05:29:22.522384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:22.522209 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" podStartSLOduration=4.522191496 podStartE2EDuration="4.522191496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:22.512680941 +0000 UTC m=+2687.040447729" watchObservedRunningTime="2026-10-30 05:29:22.522191496 +0000 UTC m=+2687.049958282" Oct 30 05:29:22.562234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:22.562171 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" podStartSLOduration=4.562152054 podStartE2EDuration="4.562152054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:22.558643753 +0000 UTC m=+2687.086410539" watchObservedRunningTime="2026-10-30 05:29:22.562152054 +0000 UTC m=+2687.089918862" Oct 30 05:29:22.625179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:22.625094 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:23.434624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:23.434499 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:29:23.434624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:23.434569 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:29:23.472842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:23.472808 6587 generic.go:356] "Generic (PLEG): container finished" podID="2451f7cd-4642-4831-8739-78d9390298e3" containerID="e345a84edf7d30bac0d8dc113fad51286b4de0c2f71bf54757e525d1f49c9b23" exitCode=0 Oct 30 05:29:23.476015 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:23.473455 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" event={"ID":"2451f7cd-4642-4831-8739-78d9390298e3","Type":"ContainerDied","Data":"e345a84edf7d30bac0d8dc113fad51286b4de0c2f71bf54757e525d1f49c9b23"} Oct 30 05:29:23.655435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:23.643508 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" start-of-body= Oct 30 05:29:23.655435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:23.643568 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" podUID="9c59e0f2-7aca-4982-a50f-9b28382c6e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" Oct 30 05:29:24.629895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:24.627759 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" start-of-body= Oct 30 05:29:24.629895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:24.627818 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" podUID="9c59e0f2-7aca-4982-a50f-9b28382c6e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" Oct 30 05:29:25.160989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.160946 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" Oct 30 05:29:25.277144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.277048 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-xks-ca\") pod \"2451f7cd-4642-4831-8739-78d9390298e3\" (UID: \"2451f7cd-4642-4831-8739-78d9390298e3\") " Oct 30 05:29:25.277144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.277126 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2451f7cd-4642-4831-8739-78d9390298e3-kube-api-access-b259p\" (UniqueName: \"kubernetes.io/projected/2451f7cd-4642-4831-8739-78d9390298e3-kube-api-access-b259p\") pod \"2451f7cd-4642-4831-8739-78d9390298e3\" (UID: \"2451f7cd-4642-4831-8739-78d9390298e3\") " Oct 30 05:29:25.278112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.277408 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-openshift-ca\") pod \"2451f7cd-4642-4831-8739-78d9390298e3\" (UID: \"2451f7cd-4642-4831-8739-78d9390298e3\") " Oct 30 05:29:25.278187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.278103 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-openshift-ca" pod "2451f7cd-4642-4831-8739-78d9390298e3" (UID: "2451f7cd-4642-4831-8739-78d9390298e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:29:25.279293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.279260 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:25.287608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.287310 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-xks-ca" pod "2451f7cd-4642-4831-8739-78d9390298e3" (UID: "2451f7cd-4642-4831-8739-78d9390298e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:29:25.299332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.299283 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2451f7cd-4642-4831-8739-78d9390298e3-kube-api-access-b259p" pod "2451f7cd-4642-4831-8739-78d9390298e3" (UID: "2451f7cd-4642-4831-8739-78d9390298e3"). InnerVolumeSpecName "kube-api-access-b259p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:29:25.381205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.381165 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2451f7cd-4642-4831-8739-78d9390298e3-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:25.381205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.381197 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b259p\" (UniqueName: \"kubernetes.io/projected/2451f7cd-4642-4831-8739-78d9390298e3-kube-api-access-b259p\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:25.509723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.509694 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" Oct 30 05:29:25.509962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.509730 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg" event={"ID":"2451f7cd-4642-4831-8739-78d9390298e3","Type":"ContainerDied","Data":"27b0bdbaf840ef235c7d9d64b9fa7b42d5f69001cdd28057f7c4082ebb0f7238"} Oct 30 05:29:25.509962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.509819 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27b0bdbaf840ef235c7d9d64b9fa7b42d5f69001cdd28057f7c4082ebb0f7238" Oct 30 05:29:25.659380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.650767 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" start-of-body= Oct 30 05:29:25.659380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:25.650821 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" podUID="9c59e0f2-7aca-4982-a50f-9b28382c6e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" Oct 30 05:29:26.009904 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:29:26.009869 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:29:26.553871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:26.553835 6587 generic.go:356] "Generic (PLEG): container finished" podID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerID="74a7eb4b2804ef7810b12a38693e254f7ba3c99d867fec953b2e2ac263e973ca" exitCode=0 Oct 30 05:29:26.554084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:26.553877 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" event={"ID":"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce","Type":"ContainerDied","Data":"74a7eb4b2804ef7810b12a38693e254f7ba3c99d867fec953b2e2ac263e973ca"} Oct 30 05:29:26.554579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:26.554558 6587 scope.go:122] "RemoveContainer" containerID="74a7eb4b2804ef7810b12a38693e254f7ba3c99d867fec953b2e2ac263e973ca" Oct 30 05:29:26.556960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:26.556912 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:29:26.557095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:26.556987 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:29:26.557095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:26.557017 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:29:26.638452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:26.637532 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" start-of-body= Oct 30 05:29:26.638452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:26.637589 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" podUID="9c59e0f2-7aca-4982-a50f-9b28382c6e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" Oct 30 05:29:27.303690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:27.303648 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" podUID="d67d5b5e-c62d-4024-b9ab-3a6b501acf89" containerName="payload-processing" containerID="cri-o://0b3162da12f9550ab3d4fe7f169dda0994fc37dc5b427db46d0bd9a4ba6ab5de" gracePeriod=25 Oct 30 05:29:27.357398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:27.357331 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" podUID="de5b2919-e724-4011-824d-b6f83bd67b83" containerName="payload-pre-processing" containerID="cri-o://c406e462d79852fbaeaeccc518ca00e0fbec51988f36d241c4f4bed627061930" gracePeriod=25 Oct 30 05:29:27.640695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:27.636608 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" start-of-body= Oct 30 05:29:27.640695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:27.636682 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" podUID="9c59e0f2-7aca-4982-a50f-9b28382c6e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" Oct 30 05:29:27.686575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:27.686527 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" event={"ID":"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce","Type":"ContainerStarted","Data":"1d0990593626ef7e662f82833f26945287eb71cf8aa26217fe72f1b6dd78ba65"} Oct 30 05:29:27.693075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:27.693037 6587 generic.go:356] "Generic (PLEG): container finished" podID="d67d5b5e-c62d-4024-b9ab-3a6b501acf89" containerID="0b3162da12f9550ab3d4fe7f169dda0994fc37dc5b427db46d0bd9a4ba6ab5de" exitCode=0 Oct 30 05:29:27.694062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:27.693177 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" event={"ID":"d67d5b5e-c62d-4024-b9ab-3a6b501acf89","Type":"ContainerDied","Data":"0b3162da12f9550ab3d4fe7f169dda0994fc37dc5b427db46d0bd9a4ba6ab5de"} Oct 30 05:29:27.697067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:27.697031 6587 generic.go:356] "Generic (PLEG): container finished" podID="de5b2919-e724-4011-824d-b6f83bd67b83" containerID="c406e462d79852fbaeaeccc518ca00e0fbec51988f36d241c4f4bed627061930" exitCode=0 Oct 30 05:29:27.697265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:27.697063 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" event={"ID":"de5b2919-e724-4011-824d-b6f83bd67b83","Type":"ContainerDied","Data":"c406e462d79852fbaeaeccc518ca00e0fbec51988f36d241c4f4bed627061930"} Oct 30 05:29:28.111904 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.111878 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" Oct 30 05:29:28.308559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.308525 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5b2919-e724-4011-824d-b6f83bd67b83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5b2919-e724-4011-824d-b6f83bd67b83-plugins-config-volume\") pod \"de5b2919-e724-4011-824d-b6f83bd67b83\" (UID: \"de5b2919-e724-4011-824d-b6f83bd67b83\") " Oct 30 05:29:28.308924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.308580 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5b2919-e724-4011-824d-b6f83bd67b83-kube-api-access-wf8rc\" (UniqueName: \"kubernetes.io/projected/de5b2919-e724-4011-824d-b6f83bd67b83-kube-api-access-wf8rc\") pod \"de5b2919-e724-4011-824d-b6f83bd67b83\" (UID: \"de5b2919-e724-4011-824d-b6f83bd67b83\") " Oct 30 05:29:28.308974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.308931 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5b2919-e724-4011-824d-b6f83bd67b83-plugins-config-volume" pod "de5b2919-e724-4011-824d-b6f83bd67b83" (UID: "de5b2919-e724-4011-824d-b6f83bd67b83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:29:28.310514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.310293 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5b2919-e724-4011-824d-b6f83bd67b83-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:28.311703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.311630 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5b2919-e724-4011-824d-b6f83bd67b83-kube-api-access-wf8rc" pod "de5b2919-e724-4011-824d-b6f83bd67b83" (UID: "de5b2919-e724-4011-824d-b6f83bd67b83"). InnerVolumeSpecName "kube-api-access-wf8rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:29:28.342383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.336711 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:29:28.366387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.352495 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 05:29:28.366387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.352560 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 05:29:28.384567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.384544 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" Oct 30 05:29:28.411822 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.411791 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-plugins-config-volume\") pod \"d67d5b5e-c62d-4024-b9ab-3a6b501acf89\" (UID: \"d67d5b5e-c62d-4024-b9ab-3a6b501acf89\") " Oct 30 05:29:28.412239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.412217 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-plugins-config-volume" pod "d67d5b5e-c62d-4024-b9ab-3a6b501acf89" (UID: "d67d5b5e-c62d-4024-b9ab-3a6b501acf89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:29:28.412317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.412254 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-kube-api-access-kq44v\" (UniqueName: \"kubernetes.io/projected/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-kube-api-access-kq44v\") pod \"d67d5b5e-c62d-4024-b9ab-3a6b501acf89\" (UID: \"d67d5b5e-c62d-4024-b9ab-3a6b501acf89\") " Oct 30 05:29:28.413144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.413110 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf8rc\" (UniqueName: \"kubernetes.io/projected/de5b2919-e724-4011-824d-b6f83bd67b83-kube-api-access-wf8rc\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:28.413144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.413141 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:28.414474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.414452 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-kube-api-access-kq44v" pod "d67d5b5e-c62d-4024-b9ab-3a6b501acf89" (UID: "d67d5b5e-c62d-4024-b9ab-3a6b501acf89"). InnerVolumeSpecName "kube-api-access-kq44v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:29:28.482487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.482433 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" podUID="fffc12ee-42fe-4e32-afb7-b8b35d6c8694" containerName="main" containerID="cri-o://fe3525c7118c3189f7faa18e7ed22dccc3e6e013e3f8c3f00983d8afee13a374" gracePeriod=45 Oct 30 05:29:28.514847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.514813 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq44v\" (UniqueName: \"kubernetes.io/projected/d67d5b5e-c62d-4024-b9ab-3a6b501acf89-kube-api-access-kq44v\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:28.612334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.612296 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" podUID="fffc12ee-42fe-4e32-afb7-b8b35d6c8694" containerName="main" probeResult="failure" output="Get \"https://10.134.1.30:8000/ready\": dial tcp 10.134.1.30:8000: connect: connection refused" Oct 30 05:29:28.661690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.657675 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" start-of-body= Oct 30 05:29:28.661690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.657740 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" podUID="9c59e0f2-7aca-4982-a50f-9b28382c6e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.61:15021/healthz/ready\": dial tcp 10.134.1.61:15021: connect: connection refused" Oct 30 05:29:28.711230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.711197 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" Oct 30 05:29:28.711640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.711225 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w" event={"ID":"de5b2919-e724-4011-824d-b6f83bd67b83","Type":"ContainerDied","Data":"9f7448579f9fb104f0c6a5f3901821889725550aa53acccd4a0aeb8f6eb16d65"} Oct 30 05:29:28.711947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.711932 6587 scope.go:122] "RemoveContainer" containerID="c406e462d79852fbaeaeccc518ca00e0fbec51988f36d241c4f4bed627061930" Oct 30 05:29:28.713998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.713882 6587 generic.go:356] "Generic (PLEG): container finished" podID="fffc12ee-42fe-4e32-afb7-b8b35d6c8694" containerID="fe3525c7118c3189f7faa18e7ed22dccc3e6e013e3f8c3f00983d8afee13a374" exitCode=0 Oct 30 05:29:28.714167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.714146 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" event={"ID":"fffc12ee-42fe-4e32-afb7-b8b35d6c8694","Type":"ContainerDied","Data":"fe3525c7118c3189f7faa18e7ed22dccc3e6e013e3f8c3f00983d8afee13a374"} Oct 30 05:29:28.735226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.723675 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" Oct 30 05:29:28.735226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.724588 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm" event={"ID":"d67d5b5e-c62d-4024-b9ab-3a6b501acf89","Type":"ContainerDied","Data":"5f36c51723cd0bc675e73ca1287845c5e736015c9373476ddcfd9ca7c0533a7c"} Oct 30 05:29:28.802147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.802125 6587 scope.go:122] "RemoveContainer" containerID="0b3162da12f9550ab3d4fe7f169dda0994fc37dc5b427db46d0bd9a4ba6ab5de" Oct 30 05:29:28.899272 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.899184 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w"] Oct 30 05:29:28.904202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.904171 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w"] Oct 30 05:29:28.997112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:28.997049 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm"] Oct 30 05:29:29.002434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.002395 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm"] Oct 30 05:29:29.054968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.054940 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:29:29.092872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.092839 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-dshm\") pod \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " Oct 30 05:29:29.093067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.092888 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-kube-api-access-9brlr\" (UniqueName: \"kubernetes.io/projected/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-kube-api-access-9brlr\") pod \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " Oct 30 05:29:29.093191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.093165 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-model-cache\") pod \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " Oct 30 05:29:29.093343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.093217 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tmp-dir\") pod \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " Oct 30 05:29:29.093448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.093339 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-home\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-home\") pod \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " Oct 30 05:29:29.093448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.093406 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tls-certs\" (UniqueName: \"kubernetes.io/secret/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tls-certs\") pod \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\" (UID: \"fffc12ee-42fe-4e32-afb7-b8b35d6c8694\") " Oct 30 05:29:29.093448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.093435 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-model-cache" pod "fffc12ee-42fe-4e32-afb7-b8b35d6c8694" (UID: "fffc12ee-42fe-4e32-afb7-b8b35d6c8694"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:29:29.093625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.093594 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-home" pod "fffc12ee-42fe-4e32-afb7-b8b35d6c8694" (UID: "fffc12ee-42fe-4e32-afb7-b8b35d6c8694"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:29:29.093783 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.093761 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tmp-dir" pod "fffc12ee-42fe-4e32-afb7-b8b35d6c8694" (UID: "fffc12ee-42fe-4e32-afb7-b8b35d6c8694"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:29:29.094276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.094257 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:29.094380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.094278 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:29.094380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.094291 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:29.095938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.095913 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tls-certs" pod "fffc12ee-42fe-4e32-afb7-b8b35d6c8694" (UID: "fffc12ee-42fe-4e32-afb7-b8b35d6c8694"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:29:29.096045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.095985 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-dshm" pod "fffc12ee-42fe-4e32-afb7-b8b35d6c8694" (UID: "fffc12ee-42fe-4e32-afb7-b8b35d6c8694"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:29:29.096139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.096121 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-kube-api-access-9brlr" pod "fffc12ee-42fe-4e32-afb7-b8b35d6c8694" (UID: "fffc12ee-42fe-4e32-afb7-b8b35d6c8694"). InnerVolumeSpecName "kube-api-access-9brlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:29:29.196331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.196240 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:29.196331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.196280 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:29.196331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.196289 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brlr\" (UniqueName: \"kubernetes.io/projected/fffc12ee-42fe-4e32-afb7-b8b35d6c8694-kube-api-access-9brlr\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:29:29.347386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.345770 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 05:29:29.347386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.345832 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 05:29:29.592434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.592397 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67d5b5e-c62d-4024-b9ab-3a6b501acf89" path="/var/lib/kubelet/pods/d67d5b5e-c62d-4024-b9ab-3a6b501acf89/volumes" Oct 30 05:29:29.594419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.593140 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5b2919-e724-4011-824d-b6f83bd67b83" path="/var/lib/kubelet/pods/de5b2919-e724-4011-824d-b6f83bd67b83/volumes" Oct 30 05:29:29.652629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.650809 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:29.652629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.652559 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:29.653419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.653390 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:29:29.776092 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.776063 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" Oct 30 05:29:29.776092 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.776077 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw" event={"ID":"fffc12ee-42fe-4e32-afb7-b8b35d6c8694","Type":"ContainerDied","Data":"4c349c5556cdab5b3bc363d304ce5e9d8ec0495e5533f4f53417c16c1c655044"} Oct 30 05:29:29.776433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.776128 6587 scope.go:122] "RemoveContainer" containerID="fe3525c7118c3189f7faa18e7ed22dccc3e6e013e3f8c3f00983d8afee13a374" Oct 30 05:29:29.905426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.904780 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw"] Oct 30 05:29:29.910785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:29.910756 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-674082/auth-test-model-674082-kserve-5d5d58d7c-96wmw"] Oct 30 05:29:30.339383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:30.336714 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 05:29:30.339383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:30.336785 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 05:29:31.340432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:31.338771 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 05:29:31.340432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:31.338839 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 05:29:31.605402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:31.605314 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fffc12ee-42fe-4e32-afb7-b8b35d6c8694" path="/var/lib/kubelet/pods/fffc12ee-42fe-4e32-afb7-b8b35d6c8694/volumes" Oct 30 05:29:32.336383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:32.335771 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 05:29:32.336383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:32.335833 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 05:29:33.340766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:33.340730 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:29:33.341280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:33.341229 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:29:33.342013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:33.341994 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:29:36.200128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:36.200095 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:29:36.200830 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:36.200332 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:29:41.992056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:41.992012 6587 scope.go:122] "RemoveContainer" containerID="e161730fb007f0e91f8b3a6b3155db32200818a715c195d494d8c6d41417544e" Oct 30 05:29:42.005790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:42.005768 6587 scope.go:122] "RemoveContainer" containerID="24d520ed0e9677ce6902df1b5ff6087f8ce6dd67305be264f45159cff5678581" Oct 30 05:29:44.004627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.004588 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh"] Oct 30 05:29:44.005503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.005485 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fffc12ee-42fe-4e32-afb7-b8b35d6c8694" containerName="main" Oct 30 05:29:44.005559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.005504 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffc12ee-42fe-4e32-afb7-b8b35d6c8694" containerName="main" Oct 30 05:29:44.005559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.005530 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d67d5b5e-c62d-4024-b9ab-3a6b501acf89" containerName="payload-processing" Oct 30 05:29:44.005559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.005535 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67d5b5e-c62d-4024-b9ab-3a6b501acf89" containerName="payload-processing" Oct 30 05:29:44.005559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.005545 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2451f7cd-4642-4831-8739-78d9390298e3" containerName="revoke-keys" Oct 30 05:29:44.005559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.005551 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2451f7cd-4642-4831-8739-78d9390298e3" containerName="revoke-keys" Oct 30 05:29:44.005725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.005562 6587 cpu_manager.go:414] "containerMap: removing container" podUID="de5b2919-e724-4011-824d-b6f83bd67b83" containerName="payload-pre-processing" Oct 30 05:29:44.005725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.005567 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5b2919-e724-4011-824d-b6f83bd67b83" containerName="payload-pre-processing" Oct 30 05:29:44.005795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.005733 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffc12ee-42fe-4e32-afb7-b8b35d6c8694" containerName="main" Oct 30 05:29:44.005795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.005743 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67d5b5e-c62d-4024-b9ab-3a6b501acf89" containerName="payload-processing" Oct 30 05:29:44.005795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.005751 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2451f7cd-4642-4831-8739-78d9390298e3" containerName="revoke-keys" Oct 30 05:29:44.005795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.005757 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5b2919-e724-4011-824d-b6f83bd67b83" containerName="payload-pre-processing" Oct 30 05:29:44.011603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.011577 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.049684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.046859 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-262c10\"/\"openshift-service-ca.crt\"" Oct 30 05:29:44.050224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.049964 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-262c10\"/\"kube-root-ca.crt\"" Oct 30 05:29:44.050224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.050108 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-262c10\"/\"default-dockercfg-ssckh\"" Oct 30 05:29:44.050224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.050195 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-262c10\"/\"rate-test-model-262c10-kserve-self-signed-certs\"" Oct 30 05:29:44.062396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.054832 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh"] Oct 30 05:29:44.064813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.064138 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7kg2\" (UniqueName: \"kubernetes.io/projected/8e74f7bb-be64-439c-9991-9050a769b5cf-kube-api-access-d7kg2\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.064813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.064196 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-dshm\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.064813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.064292 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-home\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.064813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.064322 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e74f7bb-be64-439c-9991-9050a769b5cf-tls-certs\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.064813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.064375 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-tmp-dir\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.064813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.064523 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-model-cache\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.166181 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.166135 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-model-cache\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.166468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.166448 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kg2\" (UniqueName: \"kubernetes.io/projected/8e74f7bb-be64-439c-9991-9050a769b5cf-kube-api-access-d7kg2\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.166543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.166484 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-dshm\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.166589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.166549 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-home\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.166589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.166571 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e74f7bb-be64-439c-9991-9050a769b5cf-tls-certs\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.166680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.166582 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-model-cache\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.166680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.166620 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-tmp-dir\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.167410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.166906 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-home\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.167410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.166946 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-tmp-dir\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.168957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.168926 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-dshm\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.169069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.169029 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e74f7bb-be64-439c-9991-9050a769b5cf-tls-certs\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.188263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.188237 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7kg2\" (UniqueName: \"kubernetes.io/projected/8e74f7bb-be64-439c-9991-9050a769b5cf-kube-api-access-d7kg2\") pod \"rate-test-model-262c10-kserve-648694d885-p2rhh\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.330782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.330733 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:44.701791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.701716 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh"] Oct 30 05:29:44.934889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.934852 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" event={"ID":"8e74f7bb-be64-439c-9991-9050a769b5cf","Type":"ContainerStarted","Data":"df53b99829fb7613f1bd6e788cb7946b6a9ef63e3c52ab51aee83c435186f71c"} Oct 30 05:29:44.935093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.934906 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" event={"ID":"8e74f7bb-be64-439c-9991-9050a769b5cf","Type":"ContainerStarted","Data":"c3224561727d1989e2bad3389048b7c0e2b2b679caa41ae993774de34383c9e7"} Oct 30 05:29:44.982286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:44.982182 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" podStartSLOduration=1.982157191 podStartE2EDuration="1.982157191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:44.979824264 +0000 UTC m=+2709.507591051" watchObservedRunningTime="2026-10-30 05:29:44.982157191 +0000 UTC m=+2709.509923976" Oct 30 05:29:45.331728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:45.331691 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:46.463284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.463249 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k"] Oct 30 05:29:46.468648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.468615 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.498276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.498243 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k"] Oct 30 05:29:46.498667 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.498642 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-35a94e\"/\"openshift-service-ca.crt\"" Oct 30 05:29:46.498805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.498674 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-35a94e\"/\"auth-test-model-35a94e-kserve-self-signed-certs\"" Oct 30 05:29:46.499514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.498655 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-35a94e\"/\"kube-root-ca.crt\"" Oct 30 05:29:46.505075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.505040 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-35a94e\"/\"default-dockercfg-8qgvt\"" Oct 30 05:29:46.514887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.514857 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvl74\" (UniqueName: \"kubernetes.io/projected/ae125d46-b3a6-4d61-b977-cecd363489be-kube-api-access-rvl74\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.515704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.514928 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae125d46-b3a6-4d61-b977-cecd363489be-tls-certs\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.515704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.515056 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-tmp-dir\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.515704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.515094 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-home\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.515704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.515351 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-model-cache\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.515704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.515447 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-dshm\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.616818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.616777 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvl74\" (UniqueName: \"kubernetes.io/projected/ae125d46-b3a6-4d61-b977-cecd363489be-kube-api-access-rvl74\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.617027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.616843 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae125d46-b3a6-4d61-b977-cecd363489be-tls-certs\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.617027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.616957 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-tmp-dir\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.617027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.617007 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-home\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.617770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.617331 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-model-cache\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.617770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.617404 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-dshm\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.617770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.617651 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-home\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.618018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.617930 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-model-cache\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.618115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.618089 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-tmp-dir\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.619862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.619836 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-dshm\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.620335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.620315 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae125d46-b3a6-4d61-b977-cecd363489be-tls-certs\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.639519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.639462 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvl74\" (UniqueName: \"kubernetes.io/projected/ae125d46-b3a6-4d61-b977-cecd363489be-kube-api-access-rvl74\") pod \"auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:46.812381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:46.811691 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:47.223307 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:29:47.223271 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae125d46_b3a6_4d61_b977_cecd363489be.slice/crio-75959c1acfc84358d405b3b1cffa769a3d9da4464869d433846b49550c69663d WatchSource:0}: Error finding container 75959c1acfc84358d405b3b1cffa769a3d9da4464869d433846b49550c69663d: Status 404 returned error can't find the container with id 75959c1acfc84358d405b3b1cffa769a3d9da4464869d433846b49550c69663d Oct 30 05:29:47.223825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:47.223792 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k"] Oct 30 05:29:47.980427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:47.980384 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" event={"ID":"ae125d46-b3a6-4d61-b977-cecd363489be","Type":"ContainerStarted","Data":"30e6532df37a2a495beb674bd7c42fc427e481dd7a86805b3b87852a19c60cd2"} Oct 30 05:29:47.980427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:47.980436 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" event={"ID":"ae125d46-b3a6-4d61-b977-cecd363489be","Type":"ContainerStarted","Data":"75959c1acfc84358d405b3b1cffa769a3d9da4464869d433846b49550c69663d"} Oct 30 05:29:48.030769 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:48.030699 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" podStartSLOduration=2.030677165 podStartE2EDuration="2.030677165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:48.023827576 +0000 UTC m=+2712.551594362" watchObservedRunningTime="2026-10-30 05:29:48.030677165 +0000 UTC m=+2712.558443952" Oct 30 05:29:48.812155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:48.812120 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:54.331956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:54.331897 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:54.350166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:54.350133 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:55.088472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.088439 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:29:55.122311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.122279 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs"] Oct 30 05:29:55.169697 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.169664 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9"] Oct 30 05:29:55.299691 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.299648 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz"] Oct 30 05:29:55.306958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.306922 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.342472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.328254 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-02cffa-openshift-default-dockercfg-wdscb\"" Oct 30 05:29:55.342472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.336867 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz"] Oct 30 05:29:55.479388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.476895 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.479388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.477031 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.479388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.477098 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.479388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.477161 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.479388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.477207 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.479388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.477262 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9207797-cae2-4c27-9e1b-bdce5405161c-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.479388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.477332 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.479388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.477514 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p62g\" (UniqueName: \"kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-kube-api-access-7p62g\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.479388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.477562 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.620618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.618830 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.620618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.618925 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.620618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.618990 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.620618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.619034 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.620618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.619079 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9207797-cae2-4c27-9e1b-bdce5405161c-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.620618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.619145 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.620618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.619314 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p62g\" (UniqueName: \"kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-kube-api-access-7p62g\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.620618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.619375 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.620618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.619501 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.629389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.622730 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.664549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.657526 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.664549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.657870 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.664549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.658510 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9207797-cae2-4c27-9e1b-bdce5405161c-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.664549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.659006 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.768393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.764381 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p62g\" (UniqueName: \"kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-kube-api-access-7p62g\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.802042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.801740 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.835662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.834076 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.838692 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.838281 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.931247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:55.931149 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:55.994157 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:29:55.993308 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:29:56.553989 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:29:56.553940 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9207797_cae2_4c27_9e1b_bdce5405161c.slice/crio-b9993c185b5ad20dc22bdc9809af6b0dd5b71efb072c44fd9ed1e2473ea79cb0 WatchSource:0}: Error finding container b9993c185b5ad20dc22bdc9809af6b0dd5b71efb072c44fd9ed1e2473ea79cb0: Status 404 returned error can't find the container with id b9993c185b5ad20dc22bdc9809af6b0dd5b71efb072c44fd9ed1e2473ea79cb0 Oct 30 05:29:56.557266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:56.557225 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:29:56.557414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:56.557300 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:29:56.557414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:56.557327 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:29:56.557500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:56.557472 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz"] Oct 30 05:29:56.812318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:56.812227 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:56.830946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:56.830915 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:57.284428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:57.284293 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" event={"ID":"c9207797-cae2-4c27-9e1b-bdce5405161c","Type":"ContainerStarted","Data":"0ac27b7a3b27da22512bbab85a70a89e84cfc05fd0dbce34b7256a6f4fde8346"} Oct 30 05:29:57.284428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:57.284373 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" event={"ID":"c9207797-cae2-4c27-9e1b-bdce5405161c","Type":"ContainerStarted","Data":"b9993c185b5ad20dc22bdc9809af6b0dd5b71efb072c44fd9ed1e2473ea79cb0"} Oct 30 05:29:57.295883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:57.295857 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:29:57.383438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:57.383328 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" podStartSLOduration=2.383312341 podStartE2EDuration="2.383312341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:57.335159903 +0000 UTC m=+2721.862926690" watchObservedRunningTime="2026-10-30 05:29:57.383312341 +0000 UTC m=+2721.911079129" Oct 30 05:29:57.931868 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:57.931829 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:57.939289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:57.939258 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:58.299010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:58.298980 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:29:58.299850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:29:58.299828 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:30:00.125384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.125206 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" podUID="046e8b73-ad3b-4126-85be-742df2a606cd" containerName="payload-processing" containerID="cri-o://4cd9f819255ea7b060e860a3be8f74dc9ae020477a3c06a847a90d9b28b327ac" gracePeriod=25 Oct 30 05:30:00.173389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.172388 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" podUID="513f322a-2da8-4d24-8ed1-2ee8af8a7991" containerName="payload-pre-processing" containerID="cri-o://fdfff254fd06b517eb9146efb2c802fb30305e6a4afa9d69568d2aa76f0a4994" gracePeriod=25 Oct 30 05:30:00.469513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.469470 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2"] Oct 30 05:30:00.596879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.596834 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb"] Oct 30 05:30:00.599569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.599537 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" Oct 30 05:30:00.605931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.605909 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g"] Oct 30 05:30:00.606290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.606260 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" Oct 30 05:30:00.616379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.613214 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq"] Oct 30 05:30:00.616379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.614150 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" Oct 30 05:30:00.619430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.619410 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx"] Oct 30 05:30:00.639619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.619629 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" Oct 30 05:30:00.639803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.621104 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:30:00.639803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.621833 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:30:00.639803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.621904 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:30:00.841386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.840135 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8b3d3c-c341-4dab-9c82-f85d8e203f54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2\" (UID: \"da8b3d3c-c341-4dab-9c82-f85d8e203f54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" Oct 30 05:30:00.841386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.840211 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0de3126-e112-4a37-aa81-0f312100c7b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g\" (UID: \"d0de3126-e112-4a37-aa81-0f312100c7b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" Oct 30 05:30:00.841386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.840246 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksntd\" (UniqueName: \"kubernetes.io/projected/d0de3126-e112-4a37-aa81-0f312100c7b6-kube-api-access-ksntd\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g\" (UID: \"d0de3126-e112-4a37-aa81-0f312100c7b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" Oct 30 05:30:00.841386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.840303 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hltv\" (UniqueName: \"kubernetes.io/projected/da8b3d3c-c341-4dab-9c82-f85d8e203f54-kube-api-access-5hltv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2\" (UID: \"da8b3d3c-c341-4dab-9c82-f85d8e203f54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" Oct 30 05:30:00.841386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.840473 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7kf5\" (UniqueName: \"kubernetes.io/projected/bb155f3e-3d01-431a-84a5-307d1109c9fa-kube-api-access-k7kf5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq\" (UID: \"bb155f3e-3d01-431a-84a5-307d1109c9fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" Oct 30 05:30:00.841386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.840517 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aabb8f8-1678-4f27-945e-40e30674980f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb\" (UID: \"5aabb8f8-1678-4f27-945e-40e30674980f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" Oct 30 05:30:00.841386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.840639 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbgqp\" (UniqueName: \"kubernetes.io/projected/5aabb8f8-1678-4f27-945e-40e30674980f-kube-api-access-hbgqp\") pod \"maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb\" (UID: \"5aabb8f8-1678-4f27-945e-40e30674980f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" Oct 30 05:30:00.841386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.840783 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb155f3e-3d01-431a-84a5-307d1109c9fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq\" (UID: \"bb155f3e-3d01-431a-84a5-307d1109c9fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" Oct 30 05:30:00.982659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.981896 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb155f3e-3d01-431a-84a5-307d1109c9fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq\" (UID: \"bb155f3e-3d01-431a-84a5-307d1109c9fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" Oct 30 05:30:00.982659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.982170 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8b3d3c-c341-4dab-9c82-f85d8e203f54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2\" (UID: \"da8b3d3c-c341-4dab-9c82-f85d8e203f54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" Oct 30 05:30:00.982659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.982245 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0de3126-e112-4a37-aa81-0f312100c7b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g\" (UID: \"d0de3126-e112-4a37-aa81-0f312100c7b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" Oct 30 05:30:00.982659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.982282 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksntd\" (UniqueName: \"kubernetes.io/projected/d0de3126-e112-4a37-aa81-0f312100c7b6-kube-api-access-ksntd\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g\" (UID: \"d0de3126-e112-4a37-aa81-0f312100c7b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" Oct 30 05:30:00.982659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.982331 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hltv\" (UniqueName: \"kubernetes.io/projected/da8b3d3c-c341-4dab-9c82-f85d8e203f54-kube-api-access-5hltv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2\" (UID: \"da8b3d3c-c341-4dab-9c82-f85d8e203f54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" Oct 30 05:30:00.982659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.982500 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7kf5\" (UniqueName: \"kubernetes.io/projected/bb155f3e-3d01-431a-84a5-307d1109c9fa-kube-api-access-k7kf5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq\" (UID: \"bb155f3e-3d01-431a-84a5-307d1109c9fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" Oct 30 05:30:00.982659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.982540 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aabb8f8-1678-4f27-945e-40e30674980f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb\" (UID: \"5aabb8f8-1678-4f27-945e-40e30674980f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" Oct 30 05:30:00.985812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.983135 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbgqp\" (UniqueName: \"kubernetes.io/projected/5aabb8f8-1678-4f27-945e-40e30674980f-kube-api-access-hbgqp\") pod \"maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb\" (UID: \"5aabb8f8-1678-4f27-945e-40e30674980f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" Oct 30 05:30:00.985812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.983262 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb155f3e-3d01-431a-84a5-307d1109c9fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq\" (UID: \"bb155f3e-3d01-431a-84a5-307d1109c9fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" Oct 30 05:30:00.985812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.984131 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0de3126-e112-4a37-aa81-0f312100c7b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g\" (UID: \"d0de3126-e112-4a37-aa81-0f312100c7b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" Oct 30 05:30:00.985812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.984775 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aabb8f8-1678-4f27-945e-40e30674980f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb\" (UID: \"5aabb8f8-1678-4f27-945e-40e30674980f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" Oct 30 05:30:00.985812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:00.985009 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8b3d3c-c341-4dab-9c82-f85d8e203f54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2\" (UID: \"da8b3d3c-c341-4dab-9c82-f85d8e203f54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" Oct 30 05:30:01.031664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.029585 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h"] Oct 30 05:30:01.064423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.037906 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" Oct 30 05:30:01.064423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.062984 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7kf5\" (UniqueName: \"kubernetes.io/projected/bb155f3e-3d01-431a-84a5-307d1109c9fa-kube-api-access-k7kf5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq\" (UID: \"bb155f3e-3d01-431a-84a5-307d1109c9fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" Oct 30 05:30:01.065872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.065843 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbgqp\" (UniqueName: \"kubernetes.io/projected/5aabb8f8-1678-4f27-945e-40e30674980f-kube-api-access-hbgqp\") pod \"maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb\" (UID: \"5aabb8f8-1678-4f27-945e-40e30674980f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" Oct 30 05:30:01.069578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.069551 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq"] Oct 30 05:30:01.073289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.073267 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" Oct 30 05:30:01.088385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.085593 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mmzn\" (UniqueName: \"kubernetes.io/projected/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-kube-api-access-8mmzn\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h\" (UID: \"10cb04e6-7c18-43e9-ac64-fdcbbac05b78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" Oct 30 05:30:01.088385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.087515 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w"] Oct 30 05:30:01.088385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.087609 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" Oct 30 05:30:01.105239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.092158 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx\" (UID: \"9f5876d8-6d01-4a99-950d-8af2c95c3bf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" Oct 30 05:30:01.105239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.092340 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwgtn\" (UniqueName: \"kubernetes.io/projected/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-kube-api-access-lwgtn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx\" (UID: \"9f5876d8-6d01-4a99-950d-8af2c95c3bf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" Oct 30 05:30:01.105239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.093578 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h\" (UID: \"10cb04e6-7c18-43e9-ac64-fdcbbac05b78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" Oct 30 05:30:01.105239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.094187 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2"] Oct 30 05:30:01.119338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.105663 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g"] Oct 30 05:30:01.119338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.105713 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr"] Oct 30 05:30:01.119338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.108974 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" Oct 30 05:30:01.119338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.112854 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hltv\" (UniqueName: \"kubernetes.io/projected/da8b3d3c-c341-4dab-9c82-f85d8e203f54-kube-api-access-5hltv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2\" (UID: \"da8b3d3c-c341-4dab-9c82-f85d8e203f54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" Oct 30 05:30:01.143226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.121064 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx"] Oct 30 05:30:01.143857 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.143836 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq"] Oct 30 05:30:01.144036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.144024 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb"] Oct 30 05:30:01.144170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.144159 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7"] Oct 30 05:30:01.144448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.121161 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" Oct 30 05:30:01.144640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.121770 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksntd\" (UniqueName: \"kubernetes.io/projected/d0de3126-e112-4a37-aa81-0f312100c7b6-kube-api-access-ksntd\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g\" (UID: \"d0de3126-e112-4a37-aa81-0f312100c7b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" Oct 30 05:30:01.144895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.141691 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" Oct 30 05:30:01.179370 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.176723 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" Oct 30 05:30:01.209351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.206534 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq"] Oct 30 05:30:01.254438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.236302 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" Oct 30 05:30:01.254438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.241731 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mmzn\" (UniqueName: \"kubernetes.io/projected/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-kube-api-access-8mmzn\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h\" (UID: \"10cb04e6-7c18-43e9-ac64-fdcbbac05b78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" Oct 30 05:30:01.254438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.242129 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx\" (UID: \"9f5876d8-6d01-4a99-950d-8af2c95c3bf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" Oct 30 05:30:01.254438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.242266 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd4fcaf-282e-4541-8da1-872b78f3a564-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq\" (UID: \"bdd4fcaf-282e-4541-8da1-872b78f3a564\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" Oct 30 05:30:01.254438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.242350 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwgtn\" (UniqueName: \"kubernetes.io/projected/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-kube-api-access-lwgtn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx\" (UID: \"9f5876d8-6d01-4a99-950d-8af2c95c3bf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" Oct 30 05:30:01.254438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.242434 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mlmb\" (UniqueName: \"kubernetes.io/projected/bdd4fcaf-282e-4541-8da1-872b78f3a564-kube-api-access-6mlmb\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq\" (UID: \"bdd4fcaf-282e-4541-8da1-872b78f3a564\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" Oct 30 05:30:01.254438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.242479 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h\" (UID: \"10cb04e6-7c18-43e9-ac64-fdcbbac05b78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" Oct 30 05:30:01.254438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.244082 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h\" (UID: \"10cb04e6-7c18-43e9-ac64-fdcbbac05b78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" Oct 30 05:30:01.254438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.244741 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx\" (UID: \"9f5876d8-6d01-4a99-950d-8af2c95c3bf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" Oct 30 05:30:01.537467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.537432 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h"] Oct 30 05:30:01.537759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.537741 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq"] Oct 30 05:30:01.537908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:01.537895 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc"] Oct 30 05:30:02.068791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.068760 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwgtn\" (UniqueName: \"kubernetes.io/projected/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-kube-api-access-lwgtn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx\" (UID: \"9f5876d8-6d01-4a99-950d-8af2c95c3bf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" Oct 30 05:30:02.090467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.074789 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mmzn\" (UniqueName: \"kubernetes.io/projected/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-kube-api-access-8mmzn\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h\" (UID: \"10cb04e6-7c18-43e9-ac64-fdcbbac05b78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" Oct 30 05:30:02.090467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.075027 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" Oct 30 05:30:02.138496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.138454 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" Oct 30 05:30:02.166096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.154446 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" Oct 30 05:30:02.166096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.154884 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" Oct 30 05:30:02.166096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.155055 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" Oct 30 05:30:02.166096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.163035 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd4fcaf-282e-4541-8da1-872b78f3a564-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq\" (UID: \"bdd4fcaf-282e-4541-8da1-872b78f3a564\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" Oct 30 05:30:02.181598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.181550 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd4fcaf-282e-4541-8da1-872b78f3a564-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq\" (UID: \"bdd4fcaf-282e-4541-8da1-872b78f3a564\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" Oct 30 05:30:02.191798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.189246 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg6wg\" (UniqueName: \"kubernetes.io/projected/32594502-7640-4be4-86c2-8a21c066f46c-kube-api-access-kg6wg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr\" (UID: \"32594502-7640-4be4-86c2-8a21c066f46c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" Oct 30 05:30:02.191798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.191530 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mlmb\" (UniqueName: \"kubernetes.io/projected/bdd4fcaf-282e-4541-8da1-872b78f3a564-kube-api-access-6mlmb\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq\" (UID: \"bdd4fcaf-282e-4541-8da1-872b78f3a564\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" Oct 30 05:30:02.191798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.191725 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7\" (UID: \"f8a5dcff-eb8e-418d-b0ea-26155015b2aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" Oct 30 05:30:02.199621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.192081 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32594502-7640-4be4-86c2-8a21c066f46c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr\" (UID: \"32594502-7640-4be4-86c2-8a21c066f46c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" Oct 30 05:30:02.206447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.206413 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w\" (UID: \"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" Oct 30 05:30:02.207223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.206548 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6sqt\" (UniqueName: \"kubernetes.io/projected/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-kube-api-access-k6sqt\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7\" (UID: \"f8a5dcff-eb8e-418d-b0ea-26155015b2aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" Oct 30 05:30:02.207223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.206652 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4hxd\" (UniqueName: \"kubernetes.io/projected/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-kube-api-access-n4hxd\") pod \"maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w\" (UID: \"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" Oct 30 05:30:02.231995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.231899 6587 generic.go:356] "Generic (PLEG): container finished" podID="513f322a-2da8-4d24-8ed1-2ee8af8a7991" containerID="fdfff254fd06b517eb9146efb2c802fb30305e6a4afa9d69568d2aa76f0a4994" exitCode=0 Oct 30 05:30:02.237126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.237096 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mlmb\" (UniqueName: \"kubernetes.io/projected/bdd4fcaf-282e-4541-8da1-872b78f3a564-kube-api-access-6mlmb\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq\" (UID: \"bdd4fcaf-282e-4541-8da1-872b78f3a564\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" Oct 30 05:30:02.377097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.313792 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg6wg\" (UniqueName: \"kubernetes.io/projected/32594502-7640-4be4-86c2-8a21c066f46c-kube-api-access-kg6wg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr\" (UID: \"32594502-7640-4be4-86c2-8a21c066f46c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" Oct 30 05:30:02.377097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.313891 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7\" (UID: \"f8a5dcff-eb8e-418d-b0ea-26155015b2aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" Oct 30 05:30:02.377097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.314014 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32594502-7640-4be4-86c2-8a21c066f46c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr\" (UID: \"32594502-7640-4be4-86c2-8a21c066f46c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" Oct 30 05:30:02.377097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.314093 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w\" (UID: \"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" Oct 30 05:30:02.377097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.314166 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6sqt\" (UniqueName: \"kubernetes.io/projected/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-kube-api-access-k6sqt\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7\" (UID: \"f8a5dcff-eb8e-418d-b0ea-26155015b2aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" Oct 30 05:30:02.377097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.314236 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hxd\" (UniqueName: \"kubernetes.io/projected/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-kube-api-access-n4hxd\") pod \"maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w\" (UID: \"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" Oct 30 05:30:02.391750 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.391570 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7\" (UID: \"f8a5dcff-eb8e-418d-b0ea-26155015b2aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" Oct 30 05:30:02.393307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.393133 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32594502-7640-4be4-86c2-8a21c066f46c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr\" (UID: \"32594502-7640-4be4-86c2-8a21c066f46c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" Oct 30 05:30:02.394721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.393730 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w\" (UID: \"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" Oct 30 05:30:02.398037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.397184 6587 generic.go:356] "Generic (PLEG): container finished" podID="046e8b73-ad3b-4126-85be-742df2a606cd" containerID="4cd9f819255ea7b060e860a3be8f74dc9ae020477a3c06a847a90d9b28b327ac" exitCode=0 Oct 30 05:30:02.398885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.398858 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg6wg\" (UniqueName: \"kubernetes.io/projected/32594502-7640-4be4-86c2-8a21c066f46c-kube-api-access-kg6wg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr\" (UID: \"32594502-7640-4be4-86c2-8a21c066f46c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" Oct 30 05:30:02.407741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.404191 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hxd\" (UniqueName: \"kubernetes.io/projected/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-kube-api-access-n4hxd\") pod \"maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w\" (UID: \"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" Oct 30 05:30:02.415284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.415242 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" Oct 30 05:30:02.593947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.593916 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r"] Oct 30 05:30:02.601148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.601016 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6sqt\" (UniqueName: \"kubernetes.io/projected/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-kube-api-access-k6sqt\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7\" (UID: \"f8a5dcff-eb8e-418d-b0ea-26155015b2aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" Oct 30 05:30:02.647863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.624624 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" event={"ID":"513f322a-2da8-4d24-8ed1-2ee8af8a7991","Type":"ContainerDied","Data":"fdfff254fd06b517eb9146efb2c802fb30305e6a4afa9d69568d2aa76f0a4994"} Oct 30 05:30:02.647863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.624694 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws"] Oct 30 05:30:02.648092 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.647899 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" Oct 30 05:30:02.675024 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.674987 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" event={"ID":"513f322a-2da8-4d24-8ed1-2ee8af8a7991","Type":"ContainerDied","Data":"ca3e1b882a747dfa8d0344c03b8fbae836bb389a0ddf26b89204a87266eb4988"} Oct 30 05:30:02.675263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.675246 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca3e1b882a747dfa8d0344c03b8fbae836bb389a0ddf26b89204a87266eb4988" Oct 30 05:30:02.675392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.675374 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" event={"ID":"046e8b73-ad3b-4126-85be-742df2a606cd","Type":"ContainerDied","Data":"4cd9f819255ea7b060e860a3be8f74dc9ae020477a3c06a847a90d9b28b327ac"} Oct 30 05:30:02.675511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.675496 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6"] Oct 30 05:30:02.676276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.675121 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" Oct 30 05:30:02.696314 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.696282 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" Oct 30 05:30:02.705322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.699584 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" Oct 30 05:30:02.705322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.699912 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" Oct 30 05:30:02.705322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.700199 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" Oct 30 05:30:02.714769 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.714723 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" Oct 30 05:30:02.767486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.695311 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" event={"ID":"046e8b73-ad3b-4126-85be-742df2a606cd","Type":"ContainerDied","Data":"384d77a8b5411f1c7e2ca6caa5cb2587a1ef0a0d3431d02ccdc8ce37211685e6"} Oct 30 05:30:02.767486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.764695 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="384d77a8b5411f1c7e2ca6caa5cb2587a1ef0a0d3431d02ccdc8ce37211685e6" Oct 30 05:30:02.767486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.764729 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw"] Oct 30 05:30:02.785017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.780522 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5"] Oct 30 05:30:02.785017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.781152 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" Oct 30 05:30:02.791573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.788280 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w"] Oct 30 05:30:02.791573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.788329 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7"] Oct 30 05:30:02.791573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.788370 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq"] Oct 30 05:30:02.791573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.788396 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr"] Oct 30 05:30:02.791573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.788432 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" Oct 30 05:30:02.791573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.788475 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r"] Oct 30 05:30:02.791573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.788507 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc"] Oct 30 05:30:02.791573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.788600 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw"] Oct 30 05:30:02.791573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.788628 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5"] Oct 30 05:30:02.791573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.788681 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6"] Oct 30 05:30:02.791573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.788704 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws"] Oct 30 05:30:02.796562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.793472 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54"] Oct 30 05:30:02.799759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.799729 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" Oct 30 05:30:02.801001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.800975 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e56c03f-xss54"] Oct 30 05:30:02.805721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.805372 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq\" (UID: \"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" Oct 30 05:30:02.805721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.805445 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl6lm\" (UniqueName: \"kubernetes.io/projected/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-kube-api-access-jl6lm\") pod \"maas-api-key-cleanup-29888970-qrbbc\" (UID: \"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" Oct 30 05:30:02.805721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.805528 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888970-qrbbc\" (UID: \"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" Oct 30 05:30:02.806816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.805941 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttdcn\" (UniqueName: \"kubernetes.io/projected/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-kube-api-access-ttdcn\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq\" (UID: \"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" Oct 30 05:30:02.883432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.870597 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" Oct 30 05:30:02.911407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.908752 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/513f322a-2da8-4d24-8ed1-2ee8af8a7991-kube-api-access-5hlst\" (UniqueName: \"kubernetes.io/projected/513f322a-2da8-4d24-8ed1-2ee8af8a7991-kube-api-access-5hlst\") pod \"513f322a-2da8-4d24-8ed1-2ee8af8a7991\" (UID: \"513f322a-2da8-4d24-8ed1-2ee8af8a7991\") " Oct 30 05:30:02.911407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.909179 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/513f322a-2da8-4d24-8ed1-2ee8af8a7991-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513f322a-2da8-4d24-8ed1-2ee8af8a7991-plugins-config-volume\") pod \"513f322a-2da8-4d24-8ed1-2ee8af8a7991\" (UID: \"513f322a-2da8-4d24-8ed1-2ee8af8a7991\") " Oct 30 05:30:02.911407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.909577 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gmhj\" (UniqueName: \"kubernetes.io/projected/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-kube-api-access-8gmhj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw\" (UID: \"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" Oct 30 05:30:02.911407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.909716 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e85bdf-2d09-4979-8729-6e47998265c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6\" (UID: \"a8e85bdf-2d09-4979-8729-6e47998265c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" Oct 30 05:30:02.911407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.909781 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr997\" (UniqueName: \"kubernetes.io/projected/247f6667-54e3-4e2c-98e7-9d5a85641098-kube-api-access-nr997\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5\" (UID: \"247f6667-54e3-4e2c-98e7-9d5a85641098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" Oct 30 05:30:02.911407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.909819 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttdcn\" (UniqueName: \"kubernetes.io/projected/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-kube-api-access-ttdcn\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq\" (UID: \"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" Oct 30 05:30:02.911407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.909898 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4z89\" (UniqueName: \"kubernetes.io/projected/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-kube-api-access-b4z89\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws\" (UID: \"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" Oct 30 05:30:02.911407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.910002 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247f6667-54e3-4e2c-98e7-9d5a85641098-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5\" (UID: \"247f6667-54e3-4e2c-98e7-9d5a85641098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" Oct 30 05:30:02.911407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.910072 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da52a806-feb6-4661-b0bc-89296f15a88e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r\" (UID: \"da52a806-feb6-4661-b0bc-89296f15a88e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" Oct 30 05:30:02.911407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.910104 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws\" (UID: \"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" Oct 30 05:30:02.911407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.910164 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq\" (UID: \"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" Oct 30 05:30:02.911407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.910223 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbnxk\" (UniqueName: \"kubernetes.io/projected/a8e85bdf-2d09-4979-8729-6e47998265c9-kube-api-access-vbnxk\") pod \"maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6\" (UID: \"a8e85bdf-2d09-4979-8729-6e47998265c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" Oct 30 05:30:02.911407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.910267 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl6lm\" (UniqueName: \"kubernetes.io/projected/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-kube-api-access-jl6lm\") pod \"maas-api-key-cleanup-29888970-qrbbc\" (UID: \"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" Oct 30 05:30:03.054731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.991379 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" podUID="513f322a-2da8-4d24-8ed1-2ee8af8a7991" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.54:50052\" within 1s: context deadline exceeded" Oct 30 05:30:03.054731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.004423 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq\" (UID: \"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" Oct 30 05:30:03.054731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.005659 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513f322a-2da8-4d24-8ed1-2ee8af8a7991-plugins-config-volume" pod "513f322a-2da8-4d24-8ed1-2ee8af8a7991" (UID: "513f322a-2da8-4d24-8ed1-2ee8af8a7991"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:03.054731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.047246 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttdcn\" (UniqueName: \"kubernetes.io/projected/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-kube-api-access-ttdcn\") pod \"maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq\" (UID: \"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" Oct 30 05:30:03.190631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.189922 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl6lm\" (UniqueName: \"kubernetes.io/projected/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-kube-api-access-jl6lm\") pod \"maas-api-key-cleanup-29888970-qrbbc\" (UID: \"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" Oct 30 05:30:03.191428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.191399 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888970-qrbbc\" (UID: \"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" Oct 30 05:30:03.191554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:02.910347 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888970-qrbbc\" (UID: \"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" Oct 30 05:30:03.191621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.191549 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n5qm\" (UniqueName: \"kubernetes.io/projected/da52a806-feb6-4661-b0bc-89296f15a88e-kube-api-access-4n5qm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r\" (UID: \"da52a806-feb6-4661-b0bc-89296f15a88e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" Oct 30 05:30:03.191621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.191600 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw\" (UID: \"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" Oct 30 05:30:03.280488 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:30:03.242780 6587 logging.go:55] [core] [Channel #12680 SubChannel #12681]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.54:50052", ServerName: "10.134.1.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.54:50052: operation was canceled" Oct 30 05:30:03.280488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.254752 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" Oct 30 05:30:03.280488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.275739 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513f322a-2da8-4d24-8ed1-2ee8af8a7991-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:03.324455 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:30:03.324413 6587 logging.go:55] [core] [Channel #12682 SubChannel #12683]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.53:50052", ServerName: "10.134.1.53:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.53:50052: operation was canceled" Oct 30 05:30:03.324703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.205677 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" podUID="046e8b73-ad3b-4126-85be-742df2a606cd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.53:50052\" within 1s: context deadline exceeded" Oct 30 05:30:03.339215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.338155 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513f322a-2da8-4d24-8ed1-2ee8af8a7991-kube-api-access-5hlst" pod "513f322a-2da8-4d24-8ed1-2ee8af8a7991" (UID: "513f322a-2da8-4d24-8ed1-2ee8af8a7991"). InnerVolumeSpecName "kube-api-access-5hlst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:03.350299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.350261 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" Oct 30 05:30:03.384871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.384839 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046e8b73-ad3b-4126-85be-742df2a606cd-kube-api-access-8bz46\" (UniqueName: \"kubernetes.io/projected/046e8b73-ad3b-4126-85be-742df2a606cd-kube-api-access-8bz46\") pod \"046e8b73-ad3b-4126-85be-742df2a606cd\" (UID: \"046e8b73-ad3b-4126-85be-742df2a606cd\") " Oct 30 05:30:03.385078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.385061 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/046e8b73-ad3b-4126-85be-742df2a606cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046e8b73-ad3b-4126-85be-742df2a606cd-plugins-config-volume\") pod \"046e8b73-ad3b-4126-85be-742df2a606cd\" (UID: \"046e8b73-ad3b-4126-85be-742df2a606cd\") " Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.386570 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046e8b73-ad3b-4126-85be-742df2a606cd-plugins-config-volume" pod "046e8b73-ad3b-4126-85be-742df2a606cd" (UID: "046e8b73-ad3b-4126-85be-742df2a606cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.386680 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmhj\" (UniqueName: \"kubernetes.io/projected/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-kube-api-access-8gmhj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw\" (UID: \"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.386807 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e85bdf-2d09-4979-8729-6e47998265c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6\" (UID: \"a8e85bdf-2d09-4979-8729-6e47998265c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.386870 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr997\" (UniqueName: \"kubernetes.io/projected/247f6667-54e3-4e2c-98e7-9d5a85641098-kube-api-access-nr997\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5\" (UID: \"247f6667-54e3-4e2c-98e7-9d5a85641098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.386969 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4z89\" (UniqueName: \"kubernetes.io/projected/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-kube-api-access-b4z89\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws\" (UID: \"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.387056 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247f6667-54e3-4e2c-98e7-9d5a85641098-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5\" (UID: \"247f6667-54e3-4e2c-98e7-9d5a85641098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.387124 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da52a806-feb6-4661-b0bc-89296f15a88e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r\" (UID: \"da52a806-feb6-4661-b0bc-89296f15a88e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.387156 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws\" (UID: \"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.387232 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbnxk\" (UniqueName: \"kubernetes.io/projected/a8e85bdf-2d09-4979-8729-6e47998265c9-kube-api-access-vbnxk\") pod \"maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6\" (UID: \"a8e85bdf-2d09-4979-8729-6e47998265c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.387428 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n5qm\" (UniqueName: \"kubernetes.io/projected/da52a806-feb6-4661-b0bc-89296f15a88e-kube-api-access-4n5qm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r\" (UID: \"da52a806-feb6-4661-b0bc-89296f15a88e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.387455 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw\" (UID: \"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.387619 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046e8b73-ad3b-4126-85be-742df2a606cd-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.387640 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hlst\" (UniqueName: \"kubernetes.io/projected/513f322a-2da8-4d24-8ed1-2ee8af8a7991-kube-api-access-5hlst\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.388474 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw\" (UID: \"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.389054 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e85bdf-2d09-4979-8729-6e47998265c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6\" (UID: \"a8e85bdf-2d09-4979-8729-6e47998265c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" Oct 30 05:30:03.389380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.389199 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws\" (UID: \"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" Oct 30 05:30:03.400401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.391198 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247f6667-54e3-4e2c-98e7-9d5a85641098-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5\" (UID: \"247f6667-54e3-4e2c-98e7-9d5a85641098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" Oct 30 05:30:03.400401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.399345 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da52a806-feb6-4661-b0bc-89296f15a88e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r\" (UID: \"da52a806-feb6-4661-b0bc-89296f15a88e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" Oct 30 05:30:03.472219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.471675 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmhj\" (UniqueName: \"kubernetes.io/projected/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-kube-api-access-8gmhj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw\" (UID: \"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" Oct 30 05:30:03.485284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.484962 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4z89\" (UniqueName: \"kubernetes.io/projected/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-kube-api-access-b4z89\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws\" (UID: \"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" Oct 30 05:30:03.505772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.503192 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n5qm\" (UniqueName: \"kubernetes.io/projected/da52a806-feb6-4661-b0bc-89296f15a88e-kube-api-access-4n5qm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r\" (UID: \"da52a806-feb6-4661-b0bc-89296f15a88e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" Oct 30 05:30:03.505772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.503863 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr997\" (UniqueName: \"kubernetes.io/projected/247f6667-54e3-4e2c-98e7-9d5a85641098-kube-api-access-nr997\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5\" (UID: \"247f6667-54e3-4e2c-98e7-9d5a85641098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" Oct 30 05:30:03.505772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.504788 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbnxk\" (UniqueName: \"kubernetes.io/projected/a8e85bdf-2d09-4979-8729-6e47998265c9-kube-api-access-vbnxk\") pod \"maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6\" (UID: \"a8e85bdf-2d09-4979-8729-6e47998265c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" Oct 30 05:30:03.559226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.542588 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv"] Oct 30 05:30:03.559226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.544266 6587 cpu_manager.go:414] "containerMap: removing container" podUID="046e8b73-ad3b-4126-85be-742df2a606cd" containerName="payload-processing" Oct 30 05:30:03.559226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.544292 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046e8b73-ad3b-4126-85be-742df2a606cd" containerName="payload-processing" Oct 30 05:30:03.559226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.544314 6587 cpu_manager.go:414] "containerMap: removing container" podUID="513f322a-2da8-4d24-8ed1-2ee8af8a7991" containerName="payload-pre-processing" Oct 30 05:30:03.559226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.544322 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513f322a-2da8-4d24-8ed1-2ee8af8a7991" containerName="payload-pre-processing" Oct 30 05:30:03.559226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.544590 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="513f322a-2da8-4d24-8ed1-2ee8af8a7991" containerName="payload-pre-processing" Oct 30 05:30:03.559226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.544624 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="046e8b73-ad3b-4126-85be-742df2a606cd" containerName="payload-processing" Oct 30 05:30:03.559226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.558717 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046e8b73-ad3b-4126-85be-742df2a606cd-kube-api-access-8bz46" pod "046e8b73-ad3b-4126-85be-742df2a606cd" (UID: "046e8b73-ad3b-4126-85be-742df2a606cd"). InnerVolumeSpecName "kube-api-access-8bz46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:03.563095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.559920 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046e8b73-ad3b-4126-85be-742df2a606cd-kube-api-access-8bz46\" (UniqueName: \"kubernetes.io/projected/046e8b73-ad3b-4126-85be-742df2a606cd-kube-api-access-8bz46\") pod \"046e8b73-ad3b-4126-85be-742df2a606cd\" (UID: \"046e8b73-ad3b-4126-85be-742df2a606cd\") " Oct 30 05:30:03.564613 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:30:03.564082 6587 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/046e8b73-ad3b-4126-85be-742df2a606cd/volumes/kubernetes.io~projected/kube-api-access-8bz46 Oct 30 05:30:03.564613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.564156 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046e8b73-ad3b-4126-85be-742df2a606cd-kube-api-access-8bz46" pod "046e8b73-ad3b-4126-85be-742df2a606cd" (UID: "046e8b73-ad3b-4126-85be-742df2a606cd"). InnerVolumeSpecName "kube-api-access-8bz46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:03.573493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.571975 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs" Oct 30 05:30:03.573493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.572125 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9" Oct 30 05:30:03.711005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.575104 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv"] Oct 30 05:30:03.711005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.583802 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" Oct 30 05:30:03.711005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.637801 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-02cffa\"" Oct 30 05:30:03.711005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.637968 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-02cffa-dockercfg-7c5mg\"" Oct 30 05:30:03.875062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.875028 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" Oct 30 05:30:03.884440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.883429 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bz46\" (UniqueName: \"kubernetes.io/projected/046e8b73-ad3b-4126-85be-742df2a606cd-kube-api-access-8bz46\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:03.893668 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.887279 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" Oct 30 05:30:03.958916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.958881 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" Oct 30 05:30:03.961797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.961729 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be2e889-114d-4222-b92e-ed319dab5220" path="/var/lib/kubelet/pods/0be2e889-114d-4222-b92e-ed319dab5220/volumes" Oct 30 05:30:03.965125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:03.965107 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr"] Oct 30 05:30:04.059742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.059688 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22cqm\" (UniqueName: \"kubernetes.io/projected/b966db7f-5185-48c0-80f4-4584735d9c89-kube-api-access-22cqm\") pod \"payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv\" (UID: \"b966db7f-5185-48c0-80f4-4584735d9c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" Oct 30 05:30:04.067484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.067446 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b966db7f-5185-48c0-80f4-4584735d9c89-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv\" (UID: \"b966db7f-5185-48c0-80f4-4584735d9c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" Oct 30 05:30:04.079150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.079121 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" Oct 30 05:30:04.100421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.100071 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr"] Oct 30 05:30:04.100421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.100203 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" Oct 30 05:30:04.121597 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:30:04.121554 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb155f3e_3d01_431a_84a5_307d1109c9fa.slice/crio-2fd520249f7a92f5e544f4fc3a3a38e3a28601db8f36d80b0dd5f682ff2f9a0c WatchSource:0}: Error finding container 2fd520249f7a92f5e544f4fc3a3a38e3a28601db8f36d80b0dd5f682ff2f9a0c: Status 404 returned error can't find the container with id 2fd520249f7a92f5e544f4fc3a3a38e3a28601db8f36d80b0dd5f682ff2f9a0c Oct 30 05:30:04.177460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.174998 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22cqm\" (UniqueName: \"kubernetes.io/projected/b966db7f-5185-48c0-80f4-4584735d9c89-kube-api-access-22cqm\") pod \"payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv\" (UID: \"b966db7f-5185-48c0-80f4-4584735d9c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" Oct 30 05:30:04.177460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.175246 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b966db7f-5185-48c0-80f4-4584735d9c89-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv\" (UID: \"b966db7f-5185-48c0-80f4-4584735d9c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" Oct 30 05:30:04.177460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.176518 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b966db7f-5185-48c0-80f4-4584735d9c89-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv\" (UID: \"b966db7f-5185-48c0-80f4-4584735d9c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" Oct 30 05:30:04.235900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.215320 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" Oct 30 05:30:04.287445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.280923 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13e466d-8257-409c-8d60-e8d7ec1a70f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr\" (UID: \"d13e466d-8257-409c-8d60-e8d7ec1a70f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" Oct 30 05:30:04.287966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.287935 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff6jm\" (UniqueName: \"kubernetes.io/projected/d13e466d-8257-409c-8d60-e8d7ec1a70f7-kube-api-access-ff6jm\") pod \"payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr\" (UID: \"d13e466d-8257-409c-8d60-e8d7ec1a70f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" Oct 30 05:30:04.378559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.376993 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22cqm\" (UniqueName: \"kubernetes.io/projected/b966db7f-5185-48c0-80f4-4584735d9c89-kube-api-access-22cqm\") pod \"payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv\" (UID: \"b966db7f-5185-48c0-80f4-4584735d9c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" Oct 30 05:30:04.394922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.390674 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13e466d-8257-409c-8d60-e8d7ec1a70f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr\" (UID: \"d13e466d-8257-409c-8d60-e8d7ec1a70f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" Oct 30 05:30:04.394922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.391015 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff6jm\" (UniqueName: \"kubernetes.io/projected/d13e466d-8257-409c-8d60-e8d7ec1a70f7-kube-api-access-ff6jm\") pod \"payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr\" (UID: \"d13e466d-8257-409c-8d60-e8d7ec1a70f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" Oct 30 05:30:04.394922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.392421 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13e466d-8257-409c-8d60-e8d7ec1a70f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr\" (UID: \"d13e466d-8257-409c-8d60-e8d7ec1a70f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" Oct 30 05:30:04.442883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.434707 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff6jm\" (UniqueName: \"kubernetes.io/projected/d13e466d-8257-409c-8d60-e8d7ec1a70f7-kube-api-access-ff6jm\") pod \"payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr\" (UID: \"d13e466d-8257-409c-8d60-e8d7ec1a70f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" Oct 30 05:30:04.476207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.476143 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq"] Oct 30 05:30:04.485454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.485048 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf"] Oct 30 05:30:04.495509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.495483 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.503830 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.503782 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh"] Oct 30 05:30:04.560352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.560327 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf"] Oct 30 05:30:04.561528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.561507 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh"] Oct 30 05:30:04.573633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.563784 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.590632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.590593 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9"] Oct 30 05:30:04.597687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.596390 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.597687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.596447 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.597687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.596474 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbm28\" (UniqueName: \"kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-kube-api-access-zbm28\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.597687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.596567 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54424663-3cda-49f5-a298-88beadbf9343-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.597687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.596621 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.597687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.596703 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54424663-3cda-49f5-a298-88beadbf9343-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.597687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.596746 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.597687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.596878 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.597687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.597065 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.598853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.597905 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9"] Oct 30 05:30:04.604801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.604778 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" Oct 30 05:30:04.656477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.613887 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" event={"ID":"bb155f3e-3d01-431a-84a5-307d1109c9fa","Type":"ContainerStarted","Data":"2fd520249f7a92f5e544f4fc3a3a38e3a28601db8f36d80b0dd5f682ff2f9a0c"} Oct 30 05:30:04.656477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.621409 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs"] Oct 30 05:30:04.662698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.662443 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" Oct 30 05:30:04.674049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.673768 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs"] Oct 30 05:30:04.698871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.698833 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.699632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.698890 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.699632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.698927 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.699632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.698954 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.699632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.698994 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.699632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.699225 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.699632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.699393 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fs5l\" (UniqueName: \"kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-kube-api-access-7fs5l\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.699632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.699457 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.699632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.699520 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.699632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.699613 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.700909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.699689 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.700909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.699725 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.700909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.699749 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbm28\" (UniqueName: \"kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-kube-api-access-zbm28\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.700909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.699809 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.700909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.699843 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54424663-3cda-49f5-a298-88beadbf9343-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.700909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.699892 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.700909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.699972 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54424663-3cda-49f5-a298-88beadbf9343-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.700909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.700016 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.700909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.700331 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.700909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.700456 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.700909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.700489 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.700909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.700756 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.701705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.701683 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54424663-3cda-49f5-a298-88beadbf9343-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.743021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.735922 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54424663-3cda-49f5-a298-88beadbf9343-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.743021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.736140 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.764776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.764723 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbm28\" (UniqueName: \"kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-kube-api-access-zbm28\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.766564 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.766520 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.804002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.803957 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.804262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.804240 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.804449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.804427 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.804965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.804574 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.804965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.804643 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.804965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.804744 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fs5l\" (UniqueName: \"kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-kube-api-access-7fs5l\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.804965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.804838 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.804965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.804855 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.806123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.805562 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.806123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.805599 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.806123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.805744 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.806123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.806043 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.806912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.806824 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.806912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.806883 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.807756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.807697 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.810534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.810511 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.862932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.837875 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fs5l\" (UniqueName: \"kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-kube-api-access-7fs5l\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.862932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.852811 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.903946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.902994 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:04.909551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.909331 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:04.960426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.960397 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb"] Oct 30 05:30:04.967168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.967139 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h"] Oct 30 05:30:04.994900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:04.994839 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx"] Oct 30 05:30:05.066292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.066204 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2"] Oct 30 05:30:05.074599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.073625 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g"] Oct 30 05:30:05.864831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.864787 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="046e8b73-ad3b-4126-85be-742df2a606cd" path="/var/lib/kubelet/pods/046e8b73-ad3b-4126-85be-742df2a606cd/volumes" Oct 30 05:30:05.866218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.866189 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="513f322a-2da8-4d24-8ed1-2ee8af8a7991" path="/var/lib/kubelet/pods/513f322a-2da8-4d24-8ed1-2ee8af8a7991/volumes" Oct 30 05:30:05.866747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.866724 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw"] Oct 30 05:30:05.866952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.866917 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" event={"ID":"10cb04e6-7c18-43e9-ac64-fdcbbac05b78","Type":"ContainerStarted","Data":"0420f646ce965de8748182e89fff0fc0e410d20b25553682b9ff8024e5e11e72"} Oct 30 05:30:05.883041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.882986 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq"] Oct 30 05:30:05.883307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.883259 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" event={"ID":"da8b3d3c-c341-4dab-9c82-f85d8e203f54","Type":"ContainerStarted","Data":"69be8b1403065bc9dbbb86a5d535eca111eaa15f01a1b6b08496da7ea6166b74"} Oct 30 05:30:05.890378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.890340 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5"] Oct 30 05:30:05.892666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.892633 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" event={"ID":"bb155f3e-3d01-431a-84a5-307d1109c9fa","Type":"ContainerStarted","Data":"1da33fdbaba71e0337932542b6c6a4bfe0e359fdb961220c9eb8ad9bb82aa871"} Oct 30 05:30:05.896435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.896412 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6"] Oct 30 05:30:05.906448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.906419 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w"] Oct 30 05:30:05.912095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.912067 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" event={"ID":"5aabb8f8-1678-4f27-945e-40e30674980f","Type":"ContainerStarted","Data":"14483992a50737b59c4c380d2347d7bc5de071069d157c364d1555c9aa4b4250"} Oct 30 05:30:05.914671 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.914256 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws"] Oct 30 05:30:05.968234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.968075 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r"] Oct 30 05:30:05.972954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.972007 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" event={"ID":"d0de3126-e112-4a37-aa81-0f312100c7b6","Type":"ContainerStarted","Data":"0df97b9994260f473c4d6e9787c8d2ca506f8b5ddf4a39e4e9d68b536a32b3fb"} Oct 30 05:30:05.975707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.974057 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc"] Oct 30 05:30:05.980615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.980464 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" event={"ID":"9f5876d8-6d01-4a99-950d-8af2c95c3bf8","Type":"ContainerStarted","Data":"3e91fa3c52b55f2558e55d1aee77c4d3a9021015f917ea6331bc4e80af4088f5"} Oct 30 05:30:05.982418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.982394 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq"] Oct 30 05:30:05.993660 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.991716 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr"] Oct 30 05:30:05.998962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:05.998931 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7"] Oct 30 05:30:06.004176 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:06.004152 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh"] Oct 30 05:30:06.009620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:06.009567 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" podStartSLOduration=6.009551675 podStartE2EDuration="6.009551675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:05.972049888 +0000 UTC m=+2730.499816675" watchObservedRunningTime="2026-10-30 05:30:06.009551675 +0000 UTC m=+2730.537318462" Oct 30 05:30:06.067412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:06.066952 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr"] Oct 30 05:30:06.083985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:06.083955 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv"] Oct 30 05:30:06.094566 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:06.094213 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" podStartSLOduration=6.094183684 podStartE2EDuration="6.094183684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:06.029033266 +0000 UTC m=+2730.556800053" watchObservedRunningTime="2026-10-30 05:30:06.094183684 +0000 UTC m=+2730.621950514" Oct 30 05:30:06.147383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:06.116351 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:06.160634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:06.147708 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:06.171134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:06.171095 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:06.171651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:06.116596 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf"] Oct 30 05:30:06.357114 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:30:06.355423 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54424663_3cda_49f5_a298_88beadbf9343.slice/crio-d5c0cb254fa906b751467e32d14f3546bb215a3357ea78c72dc7c704bc7848c7 WatchSource:0}: Error finding container d5c0cb254fa906b751467e32d14f3546bb215a3357ea78c72dc7c704bc7848c7: Status 404 returned error can't find the container with id d5c0cb254fa906b751467e32d14f3546bb215a3357ea78c72dc7c704bc7848c7 Oct 30 05:30:06.463592 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:06.463026 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:06.463592 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:06.463115 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:06.463592 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:06.463158 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:06.998992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:06.998941 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" event={"ID":"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf","Type":"ContainerStarted","Data":"9a098357b056579cfecac4dbe583d28b68540ff823c57bd142945446f89c2848"} Oct 30 05:30:07.000331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.000308 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" event={"ID":"54424663-3cda-49f5-a298-88beadbf9343","Type":"ContainerStarted","Data":"d5c0cb254fa906b751467e32d14f3546bb215a3357ea78c72dc7c704bc7848c7"} Oct 30 05:30:07.001727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.001705 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" event={"ID":"d13e466d-8257-409c-8d60-e8d7ec1a70f7","Type":"ContainerStarted","Data":"3d0783d263400c19f23a6b543b1dd9d961b74da0fd845735c899755a134f6cce"} Oct 30 05:30:07.003085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.003063 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" event={"ID":"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e","Type":"ContainerStarted","Data":"3c0ed43d076b7583c507c3c32160b82b384c95693f4398372f0435e458d56a7c"} Oct 30 05:30:07.004285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.004264 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" event={"ID":"f8a5dcff-eb8e-418d-b0ea-26155015b2aa","Type":"ContainerStarted","Data":"31dd4609dad232629ac8f1be9cfb38bc3ec8a69b31ec73b5d9475818b4e8cbfd"} Oct 30 05:30:07.005704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.005679 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" event={"ID":"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2","Type":"ContainerStarted","Data":"674b4d3ef6582d9f805db06082d9ea5e3f021458c2b1534ae0839b0a6a98148f"} Oct 30 05:30:07.007014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.006992 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" event={"ID":"da52a806-feb6-4661-b0bc-89296f15a88e","Type":"ContainerStarted","Data":"791de5eaddb3f2129edb8a00436c81dc92d72d7bd91a671a8351daeff6806d6f"} Oct 30 05:30:07.008404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.008347 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" event={"ID":"b966db7f-5185-48c0-80f4-4584735d9c89","Type":"ContainerStarted","Data":"e014ea6db21119204e9272d71f369b3d7705e1f14bd465a2de7e47b09b81d80c"} Oct 30 05:30:07.009969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.009945 6587 generic.go:356] "Generic (PLEG): container finished" podID="d0de3126-e112-4a37-aa81-0f312100c7b6" containerID="8a5545331072074a97772de8b99606b72701b0db712f92d88dd2d68a224a646b" exitCode=60 Oct 30 05:30:07.010068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.010018 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" event={"ID":"d0de3126-e112-4a37-aa81-0f312100c7b6","Type":"ContainerDied","Data":"8a5545331072074a97772de8b99606b72701b0db712f92d88dd2d68a224a646b"} Oct 30 05:30:07.010779 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.010761 6587 scope.go:122] "RemoveContainer" containerID="8a5545331072074a97772de8b99606b72701b0db712f92d88dd2d68a224a646b" Oct 30 05:30:07.011836 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.011817 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" event={"ID":"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1","Type":"ContainerStarted","Data":"0ab9c44629e5982b0bedf554ccddecc6e21450ace16116261fa22b84c558efa6"} Oct 30 05:30:07.011926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.011851 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" event={"ID":"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1","Type":"ContainerStarted","Data":"4275ff1fbe2492fbca0ce8768ecec0eb076e4fe7461f4bd4594802fe6628eafa"} Oct 30 05:30:07.014866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.014749 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" event={"ID":"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff","Type":"ContainerStarted","Data":"f084a6f4216f8e990fae9bd6d525b18ef4e24a6aeb0d26ef4deb0d843b75c06f"} Oct 30 05:30:07.031636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.025261 6587 generic.go:356] "Generic (PLEG): container finished" podID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" containerID="bb9994942f2f39c2443eed4cafe8dcbd94daf274153f603ee10002d2cef4800c" exitCode=60 Oct 30 05:30:07.031636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.025350 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" event={"ID":"9f5876d8-6d01-4a99-950d-8af2c95c3bf8","Type":"ContainerDied","Data":"bb9994942f2f39c2443eed4cafe8dcbd94daf274153f603ee10002d2cef4800c"} Oct 30 05:30:07.031636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.025962 6587 scope.go:122] "RemoveContainer" containerID="bb9994942f2f39c2443eed4cafe8dcbd94daf274153f603ee10002d2cef4800c" Oct 30 05:30:07.045617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.044434 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" event={"ID":"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8","Type":"ContainerStarted","Data":"c79e6e17e58402c426f26c942f8f2ca6ed81245e7f52da650050a8f589480db3"} Oct 30 05:30:07.045617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.044495 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" event={"ID":"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8","Type":"ContainerStarted","Data":"78d09e7c0d5db8a5dcb18466827a964048a51b281d07d511960b54cec31f7a9c"} Oct 30 05:30:07.049390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.047184 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" event={"ID":"a8e85bdf-2d09-4979-8729-6e47998265c9","Type":"ContainerStarted","Data":"c5c2600a5eb34aeeb9b37f044180427be50ce1cb8eb88ee9ec965d8c5f2ac3ec"} Oct 30 05:30:07.049390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.049178 6587 generic.go:356] "Generic (PLEG): container finished" podID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" containerID="a62af9af3db19023bbde463edc4fa3f2c8f0eb2a3f68521cedec76a0410870c8" exitCode=60 Oct 30 05:30:07.049390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.049290 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" event={"ID":"10cb04e6-7c18-43e9-ac64-fdcbbac05b78","Type":"ContainerDied","Data":"a62af9af3db19023bbde463edc4fa3f2c8f0eb2a3f68521cedec76a0410870c8"} Oct 30 05:30:07.050400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.049775 6587 scope.go:122] "RemoveContainer" containerID="a62af9af3db19023bbde463edc4fa3f2c8f0eb2a3f68521cedec76a0410870c8" Oct 30 05:30:07.053687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.053664 6587 generic.go:356] "Generic (PLEG): container finished" podID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" containerID="6be209b5bc69b055f5608a1ce4082f51dec0dc06e7df3742c96a13ecdd44a96f" exitCode=60 Oct 30 05:30:07.053842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.053784 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" event={"ID":"da8b3d3c-c341-4dab-9c82-f85d8e203f54","Type":"ContainerDied","Data":"6be209b5bc69b055f5608a1ce4082f51dec0dc06e7df3742c96a13ecdd44a96f"} Oct 30 05:30:07.054237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.054216 6587 scope.go:122] "RemoveContainer" containerID="6be209b5bc69b055f5608a1ce4082f51dec0dc06e7df3742c96a13ecdd44a96f" Oct 30 05:30:07.061842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.061810 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" event={"ID":"247f6667-54e3-4e2c-98e7-9d5a85641098","Type":"ContainerStarted","Data":"05fd9fb83af9343af743fb4933176666137746dfbb122a2e1a70028442ba5175"} Oct 30 05:30:07.074979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.074950 6587 generic.go:356] "Generic (PLEG): container finished" podID="bb155f3e-3d01-431a-84a5-307d1109c9fa" containerID="1da33fdbaba71e0337932542b6c6a4bfe0e359fdb961220c9eb8ad9bb82aa871" exitCode=60 Oct 30 05:30:07.075289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.075246 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" event={"ID":"bb155f3e-3d01-431a-84a5-307d1109c9fa","Type":"ContainerDied","Data":"1da33fdbaba71e0337932542b6c6a4bfe0e359fdb961220c9eb8ad9bb82aa871"} Oct 30 05:30:07.075800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.075778 6587 scope.go:122] "RemoveContainer" containerID="1da33fdbaba71e0337932542b6c6a4bfe0e359fdb961220c9eb8ad9bb82aa871" Oct 30 05:30:07.080885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.080862 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" event={"ID":"bdd4fcaf-282e-4541-8da1-872b78f3a564","Type":"ContainerStarted","Data":"f7c5a3c2882cb0879fdc0c8f8bc491282fffb2c45f53d7c00f724eedb15a91c5"} Oct 30 05:30:07.095426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.095400 6587 generic.go:356] "Generic (PLEG): container finished" podID="5aabb8f8-1678-4f27-945e-40e30674980f" containerID="c5211df730d04fe7a6051fc64e6922f417b8f4ba6f94a8085f343c9d0063f897" exitCode=60 Oct 30 05:30:07.095964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.095617 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" event={"ID":"5aabb8f8-1678-4f27-945e-40e30674980f","Type":"ContainerDied","Data":"c5211df730d04fe7a6051fc64e6922f417b8f4ba6f94a8085f343c9d0063f897"} Oct 30 05:30:07.096184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.096107 6587 scope.go:122] "RemoveContainer" containerID="c5211df730d04fe7a6051fc64e6922f417b8f4ba6f94a8085f343c9d0063f897" Oct 30 05:30:07.105960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.105905 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" event={"ID":"32594502-7640-4be4-86c2-8a21c066f46c","Type":"ContainerStarted","Data":"6c458bb57785c4cf95159a647ba159f31d316715f2ebef4247d5c0afce9867ee"} Oct 30 05:30:07.214487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.214388 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" podStartSLOduration=7.214353748 podStartE2EDuration="7.214353748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:07.212041566 +0000 UTC m=+2731.739808351" watchObservedRunningTime="2026-10-30 05:30:07.214353748 +0000 UTC m=+2731.742120545" Oct 30 05:30:07.288407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:07.288153 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" podStartSLOduration=7.288139106 podStartE2EDuration="7.288139106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:07.282777419 +0000 UTC m=+2731.810544205" watchObservedRunningTime="2026-10-30 05:30:07.288139106 +0000 UTC m=+2731.815905892" Oct 30 05:30:08.131214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.130707 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" event={"ID":"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf","Type":"ContainerStarted","Data":"954b2850487489ca5c770cb0efd8c655bc6fb5f36b7d9b6e149d5c95f3ecd6b5"} Oct 30 05:30:08.134588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.134551 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" event={"ID":"54424663-3cda-49f5-a298-88beadbf9343","Type":"ContainerStarted","Data":"da1715bd43f17a5c90bba01971076e9c84130f2c0847739b93da8bbafb6bd3a9"} Oct 30 05:30:08.137669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.137631 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" event={"ID":"d13e466d-8257-409c-8d60-e8d7ec1a70f7","Type":"ContainerStarted","Data":"5efacb5c4507c357f0805ffc5636bc77382a545fbd1da59aba4b35df9bd4ab57"} Oct 30 05:30:08.138509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.138485 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" Oct 30 05:30:08.150883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.150584 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" event={"ID":"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e","Type":"ContainerStarted","Data":"0aa79e5ea09a94e586ce85d03019264024e1c4281340c0335d3076c9db5913d8"} Oct 30 05:30:08.154694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.154664 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" event={"ID":"f8a5dcff-eb8e-418d-b0ea-26155015b2aa","Type":"ContainerStarted","Data":"90e61220ab6edf3fc76321bd6217881358331a6b3d7f59516bb51cd918abfb6c"} Oct 30 05:30:08.158423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.158391 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" event={"ID":"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2","Type":"ContainerStarted","Data":"5b3a1f15cf74c92157fa633cb2228ae6132b3443191d4df2355bd29a27b6d8ed"} Oct 30 05:30:08.161554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.161525 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" event={"ID":"da52a806-feb6-4661-b0bc-89296f15a88e","Type":"ContainerStarted","Data":"8457914ad9666639101d96b5c53d129fab2aeac5fa97e11a4112c5149059c0f7"} Oct 30 05:30:08.165465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.164791 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" event={"ID":"b966db7f-5185-48c0-80f4-4584735d9c89","Type":"ContainerStarted","Data":"c3db0ab38785dbb9eecc80b6bfb58df7babb27951dd3bbe95e30977d82f6f286"} Oct 30 05:30:08.165465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.165269 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" Oct 30 05:30:08.168419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.167734 6587 generic.go:356] "Generic (PLEG): container finished" podID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" containerID="0ab9c44629e5982b0bedf554ccddecc6e21450ace16116261fa22b84c558efa6" exitCode=60 Oct 30 05:30:08.168419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.167849 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" event={"ID":"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1","Type":"ContainerDied","Data":"0ab9c44629e5982b0bedf554ccddecc6e21450ace16116261fa22b84c558efa6"} Oct 30 05:30:08.168419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.168106 6587 scope.go:122] "RemoveContainer" containerID="0ab9c44629e5982b0bedf554ccddecc6e21450ace16116261fa22b84c558efa6" Oct 30 05:30:08.181573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.181544 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" event={"ID":"9f5876d8-6d01-4a99-950d-8af2c95c3bf8","Type":"ContainerStarted","Data":"81a184c323a7a8dfe0de6fd3625a538b0f7b553043bc4573c799b19a0cffc409"} Oct 30 05:30:08.188540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.188513 6587 generic.go:356] "Generic (PLEG): container finished" podID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" containerID="c79e6e17e58402c426f26c942f8f2ca6ed81245e7f52da650050a8f589480db3" exitCode=60 Oct 30 05:30:08.188816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.188794 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" event={"ID":"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8","Type":"ContainerDied","Data":"c79e6e17e58402c426f26c942f8f2ca6ed81245e7f52da650050a8f589480db3"} Oct 30 05:30:08.189243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.188832 6587 scope.go:122] "RemoveContainer" containerID="c79e6e17e58402c426f26c942f8f2ca6ed81245e7f52da650050a8f589480db3" Oct 30 05:30:08.189651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.189604 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" podStartSLOduration=4.189586377 podStartE2EDuration="4.189586377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:08.184006461 +0000 UTC m=+2732.711773248" watchObservedRunningTime="2026-10-30 05:30:08.189586377 +0000 UTC m=+2732.717353164" Oct 30 05:30:08.190796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.190766 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" event={"ID":"a8e85bdf-2d09-4979-8729-6e47998265c9","Type":"ContainerStarted","Data":"63c9016b70556fa39ca40da75cc9072c01c277b4fb197770bb54dbb2fe370eb8"} Oct 30 05:30:08.194964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.194945 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" event={"ID":"bdd4fcaf-282e-4541-8da1-872b78f3a564","Type":"ContainerStarted","Data":"78d98eabc9f1754c0b18ac55f41309f43bad97b1088189813efdcb9a54ac0b4e"} Oct 30 05:30:08.278977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.278872 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" podStartSLOduration=4.278852121 podStartE2EDuration="4.278852121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:08.275793809 +0000 UTC m=+2732.803560594" watchObservedRunningTime="2026-10-30 05:30:08.278852121 +0000 UTC m=+2732.806618910" Oct 30 05:30:08.345433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.344929 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" podStartSLOduration=8.344911111 podStartE2EDuration="8.344911111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:08.333855896 +0000 UTC m=+2732.861622683" watchObservedRunningTime="2026-10-30 05:30:08.344911111 +0000 UTC m=+2732.872677897" Oct 30 05:30:08.443234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.441655 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" podStartSLOduration=6.441634572 podStartE2EDuration="6.441634572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:08.437788733 +0000 UTC m=+2732.965555520" watchObservedRunningTime="2026-10-30 05:30:08.441634572 +0000 UTC m=+2732.969401351" Oct 30 05:30:08.532234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.532139 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" podStartSLOduration=8.532120156 podStartE2EDuration="8.532120156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:08.484334684 +0000 UTC m=+2733.012101471" watchObservedRunningTime="2026-10-30 05:30:08.532120156 +0000 UTC m=+2733.059886947" Oct 30 05:30:08.537114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.536877 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" podStartSLOduration=6.536866546 podStartE2EDuration="6.536866546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:08.527011034 +0000 UTC m=+2733.054777821" watchObservedRunningTime="2026-10-30 05:30:08.536866546 +0000 UTC m=+2733.064633334" Oct 30 05:30:08.578300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.578254 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" podStartSLOduration=8.578236764 podStartE2EDuration="8.578236764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:08.568831782 +0000 UTC m=+2733.096598569" watchObservedRunningTime="2026-10-30 05:30:08.578236764 +0000 UTC m=+2733.106003549" Oct 30 05:30:08.618120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.616040 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" podStartSLOduration=8.61601836 podStartE2EDuration="8.61601836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:08.612764906 +0000 UTC m=+2733.140531692" watchObservedRunningTime="2026-10-30 05:30:08.61601836 +0000 UTC m=+2733.143785147" Oct 30 05:30:08.760519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.760445 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" podStartSLOduration=8.76041301 podStartE2EDuration="8.76041301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:08.681669258 +0000 UTC m=+2733.209436045" watchObservedRunningTime="2026-10-30 05:30:08.76041301 +0000 UTC m=+2733.288179797" Oct 30 05:30:08.789635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.788012 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" podStartSLOduration=8.787994701 podStartE2EDuration="8.787994701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:08.734794554 +0000 UTC m=+2733.262561340" watchObservedRunningTime="2026-10-30 05:30:08.787994701 +0000 UTC m=+2733.315761489" Oct 30 05:30:08.905086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.905050 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:08.906686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.906590 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 05:30:08.906686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.906643 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" podUID="54424663-3cda-49f5-a298-88beadbf9343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 05:30:08.909852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.909832 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:08.911006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.910986 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 05:30:08.911080 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:08.911024 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" podUID="5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 05:30:09.145415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.144962 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds_66c8e0c3-624f-44f9-b797-b7837a2deb53/payload-processing/0.log" Oct 30 05:30:09.278860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.278647 6587 generic.go:356] "Generic (PLEG): container finished" podID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" containerID="8836640343dd4c93bc39d2b080383c8ada2747e41ef12f346a7acd8ae56dfc6f" exitCode=60 Oct 30 05:30:09.279275 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.279252 6587 scope.go:122] "RemoveContainer" containerID="8836640343dd4c93bc39d2b080383c8ada2747e41ef12f346a7acd8ae56dfc6f" Oct 30 05:30:09.283029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.279565 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" event={"ID":"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff","Type":"ContainerDied","Data":"8836640343dd4c93bc39d2b080383c8ada2747e41ef12f346a7acd8ae56dfc6f"} Oct 30 05:30:09.290785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.290754 6587 generic.go:356] "Generic (PLEG): container finished" podID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" containerID="81a184c323a7a8dfe0de6fd3625a538b0f7b553043bc4573c799b19a0cffc409" exitCode=60 Oct 30 05:30:09.291651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.291186 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" event={"ID":"9f5876d8-6d01-4a99-950d-8af2c95c3bf8","Type":"ContainerDied","Data":"81a184c323a7a8dfe0de6fd3625a538b0f7b553043bc4573c799b19a0cffc409"} Oct 30 05:30:09.291651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.291535 6587 scope.go:122] "RemoveContainer" containerID="bb9994942f2f39c2443eed4cafe8dcbd94daf274153f603ee10002d2cef4800c" Oct 30 05:30:09.292389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.291881 6587 scope.go:122] "RemoveContainer" containerID="81a184c323a7a8dfe0de6fd3625a538b0f7b553043bc4573c799b19a0cffc409" Oct 30 05:30:09.292571 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:09.292394 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx_odh-ai-gateway-infra(9f5876d8-6d01-4a99-950d-8af2c95c3bf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" podUID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" Oct 30 05:30:09.297243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.295605 6587 generic.go:356] "Generic (PLEG): container finished" podID="a8e85bdf-2d09-4979-8729-6e47998265c9" containerID="63c9016b70556fa39ca40da75cc9072c01c277b4fb197770bb54dbb2fe370eb8" exitCode=60 Oct 30 05:30:09.297243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.295695 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" event={"ID":"a8e85bdf-2d09-4979-8729-6e47998265c9","Type":"ContainerDied","Data":"63c9016b70556fa39ca40da75cc9072c01c277b4fb197770bb54dbb2fe370eb8"} Oct 30 05:30:09.297243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.296173 6587 scope.go:122] "RemoveContainer" containerID="63c9016b70556fa39ca40da75cc9072c01c277b4fb197770bb54dbb2fe370eb8" Oct 30 05:30:09.300489 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.300465 6587 generic.go:356] "Generic (PLEG): container finished" podID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" containerID="c448be4d7f40fb822feda966b302b539ab90f7d40da539c0f1ff8d6001e4122e" exitCode=60 Oct 30 05:30:09.300595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.300580 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" event={"ID":"10cb04e6-7c18-43e9-ac64-fdcbbac05b78","Type":"ContainerDied","Data":"c448be4d7f40fb822feda966b302b539ab90f7d40da539c0f1ff8d6001e4122e"} Oct 30 05:30:09.301092 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.301073 6587 scope.go:122] "RemoveContainer" containerID="c448be4d7f40fb822feda966b302b539ab90f7d40da539c0f1ff8d6001e4122e" Oct 30 05:30:09.301547 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:09.301514 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h_odh-ai-gateway-infra(10cb04e6-7c18-43e9-ac64-fdcbbac05b78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" podUID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" Oct 30 05:30:09.360471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.360279 6587 generic.go:356] "Generic (PLEG): container finished" podID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" containerID="ea4de1936b55ebca8aa73a4e228f9efd89998a9acff6a6e256d6e6b09fc24abb" exitCode=60 Oct 30 05:30:09.360931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.360854 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" event={"ID":"da8b3d3c-c341-4dab-9c82-f85d8e203f54","Type":"ContainerDied","Data":"ea4de1936b55ebca8aa73a4e228f9efd89998a9acff6a6e256d6e6b09fc24abb"} Oct 30 05:30:09.361927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.361816 6587 scope.go:122] "RemoveContainer" containerID="ea4de1936b55ebca8aa73a4e228f9efd89998a9acff6a6e256d6e6b09fc24abb" Oct 30 05:30:09.362933 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:09.362877 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2_odh-ai-gateway-infra(da8b3d3c-c341-4dab-9c82-f85d8e203f54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" podUID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" Oct 30 05:30:09.447955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.417714 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vqsmz_a8739f61-fe1e-4c58-b535-4815d55e05f1/payload-processing/0.log" Oct 30 05:30:09.514488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.514464 6587 scope.go:122] "RemoveContainer" containerID="0512908cf7d60b69ae0b7af0f29e7a19fbfb2df0f704181fd3d2859012142797" Oct 30 05:30:09.515089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.515049 6587 generic.go:356] "Generic (PLEG): container finished" podID="247f6667-54e3-4e2c-98e7-9d5a85641098" containerID="0512908cf7d60b69ae0b7af0f29e7a19fbfb2df0f704181fd3d2859012142797" exitCode=60 Oct 30 05:30:09.515144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.515090 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" event={"ID":"247f6667-54e3-4e2c-98e7-9d5a85641098","Type":"ContainerDied","Data":"0512908cf7d60b69ae0b7af0f29e7a19fbfb2df0f704181fd3d2859012142797"} Oct 30 05:30:09.559385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.557245 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" event={"ID":"bb155f3e-3d01-431a-84a5-307d1109c9fa","Type":"ContainerStarted","Data":"13e47ec469fa4296347394e0a6cc573144685106da734f2e28bb86797ade77a9"} Oct 30 05:30:09.559385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.557861 6587 scope.go:122] "RemoveContainer" containerID="13e47ec469fa4296347394e0a6cc573144685106da734f2e28bb86797ade77a9" Oct 30 05:30:09.559385 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:09.558413 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq_odh-ai-gateway-infra(bb155f3e-3d01-431a-84a5-307d1109c9fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" Oct 30 05:30:09.576560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.575617 6587 generic.go:356] "Generic (PLEG): container finished" podID="bdd4fcaf-282e-4541-8da1-872b78f3a564" containerID="78d98eabc9f1754c0b18ac55f41309f43bad97b1088189813efdcb9a54ac0b4e" exitCode=60 Oct 30 05:30:09.576560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.576216 6587 scope.go:122] "RemoveContainer" containerID="78d98eabc9f1754c0b18ac55f41309f43bad97b1088189813efdcb9a54ac0b4e" Oct 30 05:30:09.693665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.693624 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vqsmz_a8739f61-fe1e-4c58-b535-4815d55e05f1/payload-processing/0.log" Oct 30 05:30:09.701663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.701006 6587 generic.go:356] "Generic (PLEG): container finished" podID="5aabb8f8-1678-4f27-945e-40e30674980f" containerID="60b6b759b97d98e10e06c929a37eef3407a88ec45a0e312d34dbcf593e321448" exitCode=60 Oct 30 05:30:09.705980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.705952 6587 scope.go:122] "RemoveContainer" containerID="60b6b759b97d98e10e06c929a37eef3407a88ec45a0e312d34dbcf593e321448" Oct 30 05:30:09.707855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.707038 6587 generic.go:356] "Generic (PLEG): container finished" podID="32594502-7640-4be4-86c2-8a21c066f46c" containerID="df9cc060d94c839d478bc76c9c64508d1c1f105cbc8efab49d81b2687b3f0e6a" exitCode=60 Oct 30 05:30:09.707855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.707615 6587 scope.go:122] "RemoveContainer" containerID="df9cc060d94c839d478bc76c9c64508d1c1f105cbc8efab49d81b2687b3f0e6a" Oct 30 05:30:09.708408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.708386 6587 scope.go:122] "RemoveContainer" containerID="a62af9af3db19023bbde463edc4fa3f2c8f0eb2a3f68521cedec76a0410870c8" Oct 30 05:30:09.709196 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:09.709174 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb_odh-ai-gateway-infra(5aabb8f8-1678-4f27-945e-40e30674980f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" podUID="5aabb8f8-1678-4f27-945e-40e30674980f" Oct 30 05:30:09.800426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.800198 6587 generic.go:356] "Generic (PLEG): container finished" podID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" containerID="0aa79e5ea09a94e586ce85d03019264024e1c4281340c0335d3076c9db5913d8" exitCode=60 Oct 30 05:30:09.802145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.801678 6587 scope.go:122] "RemoveContainer" containerID="0aa79e5ea09a94e586ce85d03019264024e1c4281340c0335d3076c9db5913d8" Oct 30 05:30:09.805719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.805691 6587 generic.go:356] "Generic (PLEG): container finished" podID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" containerID="90e61220ab6edf3fc76321bd6217881358331a6b3d7f59516bb51cd918abfb6c" exitCode=60 Oct 30 05:30:09.808924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.807829 6587 scope.go:122] "RemoveContainer" containerID="90e61220ab6edf3fc76321bd6217881358331a6b3d7f59516bb51cd918abfb6c" Oct 30 05:30:09.808924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.808740 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" event={"ID":"bdd4fcaf-282e-4541-8da1-872b78f3a564","Type":"ContainerDied","Data":"78d98eabc9f1754c0b18ac55f41309f43bad97b1088189813efdcb9a54ac0b4e"} Oct 30 05:30:09.808924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.808787 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" event={"ID":"5aabb8f8-1678-4f27-945e-40e30674980f","Type":"ContainerDied","Data":"60b6b759b97d98e10e06c929a37eef3407a88ec45a0e312d34dbcf593e321448"} Oct 30 05:30:09.808924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.808822 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" event={"ID":"32594502-7640-4be4-86c2-8a21c066f46c","Type":"ContainerDied","Data":"df9cc060d94c839d478bc76c9c64508d1c1f105cbc8efab49d81b2687b3f0e6a"} Oct 30 05:30:09.808924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.808848 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" event={"ID":"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e","Type":"ContainerDied","Data":"0aa79e5ea09a94e586ce85d03019264024e1c4281340c0335d3076c9db5913d8"} Oct 30 05:30:09.808924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.808873 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" event={"ID":"f8a5dcff-eb8e-418d-b0ea-26155015b2aa","Type":"ContainerDied","Data":"90e61220ab6edf3fc76321bd6217881358331a6b3d7f59516bb51cd918abfb6c"} Oct 30 05:30:09.810576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.810557 6587 generic.go:356] "Generic (PLEG): container finished" podID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" containerID="5b3a1f15cf74c92157fa633cb2228ae6132b3443191d4df2355bd29a27b6d8ed" exitCode=60 Oct 30 05:30:09.810773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.810752 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" event={"ID":"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2","Type":"ContainerDied","Data":"5b3a1f15cf74c92157fa633cb2228ae6132b3443191d4df2355bd29a27b6d8ed"} Oct 30 05:30:09.883018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.810924 6587 scope.go:122] "RemoveContainer" containerID="5b3a1f15cf74c92157fa633cb2228ae6132b3443191d4df2355bd29a27b6d8ed" Oct 30 05:30:09.913726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.898143 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h_82a13271-a884-4256-8ae5-f24d9a61c7c6/payload-processing/0.log" Oct 30 05:30:09.913726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.898378 6587 generic.go:356] "Generic (PLEG): container finished" podID="da52a806-feb6-4661-b0bc-89296f15a88e" containerID="8457914ad9666639101d96b5c53d129fab2aeac5fa97e11a4112c5149059c0f7" exitCode=60 Oct 30 05:30:09.913726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.898516 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" event={"ID":"da52a806-feb6-4661-b0bc-89296f15a88e","Type":"ContainerDied","Data":"8457914ad9666639101d96b5c53d129fab2aeac5fa97e11a4112c5149059c0f7"} Oct 30 05:30:09.913726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.898901 6587 scope.go:122] "RemoveContainer" containerID="8457914ad9666639101d96b5c53d129fab2aeac5fa97e11a4112c5149059c0f7" Oct 30 05:30:09.913726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.905309 6587 generic.go:356] "Generic (PLEG): container finished" podID="d0de3126-e112-4a37-aa81-0f312100c7b6" containerID="67af5a5a7c30555c708e73ccce5734eb8a38051e333838e43aad3991535fb98b" exitCode=60 Oct 30 05:30:09.913726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.905493 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" event={"ID":"d0de3126-e112-4a37-aa81-0f312100c7b6","Type":"ContainerDied","Data":"67af5a5a7c30555c708e73ccce5734eb8a38051e333838e43aad3991535fb98b"} Oct 30 05:30:09.913726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.907371 6587 scope.go:122] "RemoveContainer" containerID="67af5a5a7c30555c708e73ccce5734eb8a38051e333838e43aad3991535fb98b" Oct 30 05:30:09.913726 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:09.907660 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g_odh-ai-gateway-infra(d0de3126-e112-4a37-aa81-0f312100c7b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" podUID="d0de3126-e112-4a37-aa81-0f312100c7b6" Oct 30 05:30:09.913726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.907696 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 05:30:09.913726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.907734 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" podUID="54424663-3cda-49f5-a298-88beadbf9343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 05:30:09.913726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.912946 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 05:30:09.913726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:09.912985 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" podUID="5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 05:30:10.107333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:10.107098 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h_82a13271-a884-4256-8ae5-f24d9a61c7c6/payload-processing/0.log" Oct 30 05:30:10.299595 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:10.299367 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb18203a_9a1a_4c6d_9cc2_b5041b3fc6a1.slice/crio-a8fb07f081d08fcdc173de8362fcf523a219fc28567a379034ba4db2f93481c7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb18203a_9a1a_4c6d_9cc2_b5041b3fc6a1.slice/crio-conmon-a8fb07f081d08fcdc173de8362fcf523a219fc28567a379034ba4db2f93481c7.scope\": RecentStats: unable to find data in memory cache]" Oct 30 05:30:10.393805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:10.393771 6587 scope.go:122] "RemoveContainer" containerID="6be209b5bc69b055f5608a1ce4082f51dec0dc06e7df3742c96a13ecdd44a96f" Oct 30 05:30:10.490546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:10.490429 6587 scope.go:122] "RemoveContainer" containerID="c5211df730d04fe7a6051fc64e6922f417b8f4ba6f94a8085f343c9d0063f897" Oct 30 05:30:10.540552 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:10.540514 6587 scope.go:122] "RemoveContainer" containerID="8a5545331072074a97772de8b99606b72701b0db712f92d88dd2d68a224a646b" Oct 30 05:30:10.909258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:10.909156 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 05:30:10.909411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:10.909222 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" podUID="54424663-3cda-49f5-a298-88beadbf9343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 05:30:10.910318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:10.909963 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 05:30:10.910318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:10.910039 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" podUID="5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 05:30:10.910318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:10.910112 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v"] Oct 30 05:30:11.012725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.012688 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v"] Oct 30 05:30:11.012828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.012814 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" Oct 30 05:30:11.112846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.061197 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:30:11.112846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.108298 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g"] Oct 30 05:30:11.114652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.114619 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcbe01d-f7d2g"] Oct 30 05:30:11.200383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.185019 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" event={"ID":"a8e85bdf-2d09-4979-8729-6e47998265c9","Type":"ContainerStarted","Data":"f13bb37578f0ebec286232d1b1821f6a5bf1960ba67955bb1eebdc6ceceebe4c"} Oct 30 05:30:11.208308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.208278 6587 scope.go:122] "RemoveContainer" containerID="c448be4d7f40fb822feda966b302b539ab90f7d40da539c0f1ff8d6001e4122e" Oct 30 05:30:11.214971 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:11.214936 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h_odh-ai-gateway-infra(10cb04e6-7c18-43e9-ac64-fdcbbac05b78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" podUID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" Oct 30 05:30:11.225635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.220351 6587 generic.go:356] "Generic (PLEG): container finished" podID="bb155f3e-3d01-431a-84a5-307d1109c9fa" containerID="13e47ec469fa4296347394e0a6cc573144685106da734f2e28bb86797ade77a9" exitCode=60 Oct 30 05:30:11.225635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.220491 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" event={"ID":"bb155f3e-3d01-431a-84a5-307d1109c9fa","Type":"ContainerDied","Data":"13e47ec469fa4296347394e0a6cc573144685106da734f2e28bb86797ade77a9"} Oct 30 05:30:11.225635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.220545 6587 scope.go:122] "RemoveContainer" containerID="1da33fdbaba71e0337932542b6c6a4bfe0e359fdb961220c9eb8ad9bb82aa871" Oct 30 05:30:11.225635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.225075 6587 scope.go:122] "RemoveContainer" containerID="13e47ec469fa4296347394e0a6cc573144685106da734f2e28bb86797ade77a9" Oct 30 05:30:11.225910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.225777 6587 scope.go:122] "RemoveContainer" containerID="ea4de1936b55ebca8aa73a4e228f9efd89998a9acff6a6e256d6e6b09fc24abb" Oct 30 05:30:11.228266 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:11.226152 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2_odh-ai-gateway-infra(da8b3d3c-c341-4dab-9c82-f85d8e203f54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" podUID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" Oct 30 05:30:11.228266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.226301 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vld\" (UniqueName: \"kubernetes.io/projected/fc5c804d-fffb-4fcc-8686-080b8f61703c-kube-api-access-c4vld\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v\" (UID: \"fc5c804d-fffb-4fcc-8686-080b8f61703c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" Oct 30 05:30:11.228266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.226495 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v\" (UID: \"fc5c804d-fffb-4fcc-8686-080b8f61703c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" Oct 30 05:30:11.228266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.226834 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v\" (UID: \"fc5c804d-fffb-4fcc-8686-080b8f61703c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" Oct 30 05:30:11.239192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.239159 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" event={"ID":"f8a5dcff-eb8e-418d-b0ea-26155015b2aa","Type":"ContainerStarted","Data":"d98dd3d6e2b05121525d45b8195c09849450fd7eab109f3419dd7b39593529b5"} Oct 30 05:30:11.253749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.253725 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" event={"ID":"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1","Type":"ContainerDied","Data":"a8fb07f081d08fcdc173de8362fcf523a219fc28567a379034ba4db2f93481c7"} Oct 30 05:30:11.253905 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:11.252125 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq_odh-ai-gateway-infra(bb155f3e-3d01-431a-84a5-307d1109c9fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" Oct 30 05:30:11.254059 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.253035 6587 scope.go:122] "RemoveContainer" containerID="60b6b759b97d98e10e06c929a37eef3407a88ec45a0e312d34dbcf593e321448" Oct 30 05:30:11.254523 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:11.254496 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb_odh-ai-gateway-infra(5aabb8f8-1678-4f27-945e-40e30674980f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" podUID="5aabb8f8-1678-4f27-945e-40e30674980f" Oct 30 05:30:11.254681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.244580 6587 generic.go:356] "Generic (PLEG): container finished" podID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" containerID="a8fb07f081d08fcdc173de8362fcf523a219fc28567a379034ba4db2f93481c7" exitCode=60 Oct 30 05:30:11.254945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.254930 6587 scope.go:122] "RemoveContainer" containerID="67af5a5a7c30555c708e73ccce5734eb8a38051e333838e43aad3991535fb98b" Oct 30 05:30:11.255350 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:11.255327 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g_odh-ai-gateway-infra(d0de3126-e112-4a37-aa81-0f312100c7b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" podUID="d0de3126-e112-4a37-aa81-0f312100c7b6" Oct 30 05:30:11.255517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.255503 6587 scope.go:122] "RemoveContainer" containerID="a8fb07f081d08fcdc173de8362fcf523a219fc28567a379034ba4db2f93481c7" Oct 30 05:30:11.255900 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:11.255878 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888970-qrbbc_odh-ai-gateway-infra(cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" podUID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" Oct 30 05:30:11.258998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.258976 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" event={"ID":"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff","Type":"ContainerStarted","Data":"0b4b2b163281afbd9cdcffa9157b643bfba4d7dd4e3e2cada46bac9be5c013ac"} Oct 30 05:30:11.312427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.312398 6587 scope.go:122] "RemoveContainer" containerID="81a184c323a7a8dfe0de6fd3625a538b0f7b553043bc4573c799b19a0cffc409" Oct 30 05:30:11.312991 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:11.312956 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx_odh-ai-gateway-infra(9f5876d8-6d01-4a99-950d-8af2c95c3bf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" podUID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" Oct 30 05:30:11.357382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.357195 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v\" (UID: \"fc5c804d-fffb-4fcc-8686-080b8f61703c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" Oct 30 05:30:11.371471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.357523 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vld\" (UniqueName: \"kubernetes.io/projected/fc5c804d-fffb-4fcc-8686-080b8f61703c-kube-api-access-c4vld\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v\" (UID: \"fc5c804d-fffb-4fcc-8686-080b8f61703c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" Oct 30 05:30:11.371471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.357744 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v\" (UID: \"fc5c804d-fffb-4fcc-8686-080b8f61703c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" Oct 30 05:30:11.374476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.372250 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v\" (UID: \"fc5c804d-fffb-4fcc-8686-080b8f61703c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" Oct 30 05:30:11.391024 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.387587 6587 scope.go:122] "RemoveContainer" containerID="0ab9c44629e5982b0bedf554ccddecc6e21450ace16116261fa22b84c558efa6" Oct 30 05:30:11.402891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.401149 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v\" (UID: \"fc5c804d-fffb-4fcc-8686-080b8f61703c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" Oct 30 05:30:11.402891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.401368 6587 generic.go:356] "Generic (PLEG): container finished" podID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" containerID="7c31976cc4288ae5f0cdf876edc5b88b414518a4b846789f4dc472a5684f5178" exitCode=60 Oct 30 05:30:11.402891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.401712 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vld\" (UniqueName: \"kubernetes.io/projected/fc5c804d-fffb-4fcc-8686-080b8f61703c-kube-api-access-c4vld\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v\" (UID: \"fc5c804d-fffb-4fcc-8686-080b8f61703c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" Oct 30 05:30:11.402891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.401760 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" event={"ID":"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8","Type":"ContainerDied","Data":"7c31976cc4288ae5f0cdf876edc5b88b414518a4b846789f4dc472a5684f5178"} Oct 30 05:30:11.402891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.402602 6587 scope.go:122] "RemoveContainer" containerID="7c31976cc4288ae5f0cdf876edc5b88b414518a4b846789f4dc472a5684f5178" Oct 30 05:30:11.403286 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:11.402954 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw_odh-ai-gateway-infra(4427d4fe-f1ad-4635-94ad-b06b9d2f81c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" podUID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" Oct 30 05:30:11.542383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.517148 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" Oct 30 05:30:11.608275 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.606948 6587 scope.go:122] "RemoveContainer" containerID="c79e6e17e58402c426f26c942f8f2ca6ed81245e7f52da650050a8f589480db3" Oct 30 05:30:12.005894 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.998629 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 05:30:12.005894 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:11.998697 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" podUID="54424663-3cda-49f5-a298-88beadbf9343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 05:30:12.005894 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.001420 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 05:30:12.005894 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.001462 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" podUID="5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 05:30:12.009427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.009205 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58097ffd-8c5b-4a63-a0b7-7cfe31728523" path="/var/lib/kubelet/pods/58097ffd-8c5b-4a63-a0b7-7cfe31728523/volumes" Oct 30 05:30:12.075620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.075482 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" podStartSLOduration=12.075462851 podStartE2EDuration="12.075462851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:11.99980211 +0000 UTC m=+2736.527568899" watchObservedRunningTime="2026-10-30 05:30:12.075462851 +0000 UTC m=+2736.603229637" Oct 30 05:30:12.464436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.440986 6587 scope.go:122] "RemoveContainer" containerID="a8fb07f081d08fcdc173de8362fcf523a219fc28567a379034ba4db2f93481c7" Oct 30 05:30:12.489298 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:12.489229 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888970-qrbbc_odh-ai-gateway-infra(cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" podUID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" Oct 30 05:30:12.505018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.503637 6587 generic.go:356] "Generic (PLEG): container finished" podID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" containerID="0b4b2b163281afbd9cdcffa9157b643bfba4d7dd4e3e2cada46bac9be5c013ac" exitCode=60 Oct 30 05:30:12.505018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.503789 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" event={"ID":"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff","Type":"ContainerDied","Data":"0b4b2b163281afbd9cdcffa9157b643bfba4d7dd4e3e2cada46bac9be5c013ac"} Oct 30 05:30:12.505018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.503851 6587 scope.go:122] "RemoveContainer" containerID="8836640343dd4c93bc39d2b080383c8ada2747e41ef12f346a7acd8ae56dfc6f" Oct 30 05:30:12.505018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.504418 6587 scope.go:122] "RemoveContainer" containerID="0b4b2b163281afbd9cdcffa9157b643bfba4d7dd4e3e2cada46bac9be5c013ac" Oct 30 05:30:12.505018 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:12.504950 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws_odh-ai-gateway-infra(a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" podUID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" Oct 30 05:30:12.509161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.508902 6587 scope.go:122] "RemoveContainer" containerID="7c31976cc4288ae5f0cdf876edc5b88b414518a4b846789f4dc472a5684f5178" Oct 30 05:30:12.509919 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:12.509753 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw_odh-ai-gateway-infra(4427d4fe-f1ad-4635-94ad-b06b9d2f81c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" podUID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" Oct 30 05:30:12.543383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.542331 6587 generic.go:356] "Generic (PLEG): container finished" podID="a8e85bdf-2d09-4979-8729-6e47998265c9" containerID="f13bb37578f0ebec286232d1b1821f6a5bf1960ba67955bb1eebdc6ceceebe4c" exitCode=60 Oct 30 05:30:12.543383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.542562 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" event={"ID":"a8e85bdf-2d09-4979-8729-6e47998265c9","Type":"ContainerDied","Data":"f13bb37578f0ebec286232d1b1821f6a5bf1960ba67955bb1eebdc6ceceebe4c"} Oct 30 05:30:12.543383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.543138 6587 scope.go:122] "RemoveContainer" containerID="f13bb37578f0ebec286232d1b1821f6a5bf1960ba67955bb1eebdc6ceceebe4c" Oct 30 05:30:12.565571 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:12.543733 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6_odh-ai-gateway-infra(a8e85bdf-2d09-4979-8729-6e47998265c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" podUID="a8e85bdf-2d09-4979-8729-6e47998265c9" Oct 30 05:30:12.592510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.592461 6587 generic.go:356] "Generic (PLEG): container finished" podID="247f6667-54e3-4e2c-98e7-9d5a85641098" containerID="d3159df56c691297a72da69a45070a624be9ccf1168d27089ba7ae862e0ff755" exitCode=60 Oct 30 05:30:12.592847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.592826 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" event={"ID":"247f6667-54e3-4e2c-98e7-9d5a85641098","Type":"ContainerDied","Data":"d3159df56c691297a72da69a45070a624be9ccf1168d27089ba7ae862e0ff755"} Oct 30 05:30:12.593918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.593895 6587 scope.go:122] "RemoveContainer" containerID="d3159df56c691297a72da69a45070a624be9ccf1168d27089ba7ae862e0ff755" Oct 30 05:30:12.594591 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:12.594542 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5_odh-ai-gateway-infra(247f6667-54e3-4e2c-98e7-9d5a85641098)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" podUID="247f6667-54e3-4e2c-98e7-9d5a85641098" Oct 30 05:30:12.596162 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.596141 6587 scope.go:122] "RemoveContainer" containerID="13e47ec469fa4296347394e0a6cc573144685106da734f2e28bb86797ade77a9" Oct 30 05:30:12.596516 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:12.596493 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq_odh-ai-gateway-infra(bb155f3e-3d01-431a-84a5-307d1109c9fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" Oct 30 05:30:12.598307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.598291 6587 scope.go:122] "RemoveContainer" containerID="63c9016b70556fa39ca40da75cc9072c01c277b4fb197770bb54dbb2fe370eb8" Oct 30 05:30:12.599482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.599460 6587 generic.go:356] "Generic (PLEG): container finished" podID="bdd4fcaf-282e-4541-8da1-872b78f3a564" containerID="590df08b6555df99bb439076ebdd17547c9d29472d53d52115795fbcc8038ea4" exitCode=60 Oct 30 05:30:12.599574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.599545 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" event={"ID":"bdd4fcaf-282e-4541-8da1-872b78f3a564","Type":"ContainerDied","Data":"590df08b6555df99bb439076ebdd17547c9d29472d53d52115795fbcc8038ea4"} Oct 30 05:30:12.600100 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.600079 6587 scope.go:122] "RemoveContainer" containerID="590df08b6555df99bb439076ebdd17547c9d29472d53d52115795fbcc8038ea4" Oct 30 05:30:12.600440 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:12.600418 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq_odh-ai-gateway-infra(bdd4fcaf-282e-4541-8da1-872b78f3a564)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" podUID="bdd4fcaf-282e-4541-8da1-872b78f3a564" Oct 30 05:30:12.606244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.606224 6587 generic.go:356] "Generic (PLEG): container finished" podID="32594502-7640-4be4-86c2-8a21c066f46c" containerID="f9246019a66748a7aa7fa069caff20210e419a6de517568b087a2bb878d740ee" exitCode=60 Oct 30 05:30:12.606413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.606301 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" event={"ID":"32594502-7640-4be4-86c2-8a21c066f46c","Type":"ContainerDied","Data":"f9246019a66748a7aa7fa069caff20210e419a6de517568b087a2bb878d740ee"} Oct 30 05:30:12.606611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.606546 6587 scope.go:122] "RemoveContainer" containerID="f9246019a66748a7aa7fa069caff20210e419a6de517568b087a2bb878d740ee" Oct 30 05:30:12.606970 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:12.606945 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr_odh-ai-gateway-infra(32594502-7640-4be4-86c2-8a21c066f46c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" podUID="32594502-7640-4be4-86c2-8a21c066f46c" Oct 30 05:30:12.609034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.609014 6587 generic.go:356] "Generic (PLEG): container finished" podID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" containerID="c751fb4c2d505558460f1d46bad1ce33f94ae5fea988529db409925cdc32075b" exitCode=60 Oct 30 05:30:12.609136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.609081 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" event={"ID":"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e","Type":"ContainerDied","Data":"c751fb4c2d505558460f1d46bad1ce33f94ae5fea988529db409925cdc32075b"} Oct 30 05:30:12.609416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.609397 6587 scope.go:122] "RemoveContainer" containerID="c751fb4c2d505558460f1d46bad1ce33f94ae5fea988529db409925cdc32075b" Oct 30 05:30:12.609771 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:12.609745 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq_odh-ai-gateway-infra(b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" podUID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" Oct 30 05:30:12.611100 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.611079 6587 generic.go:356] "Generic (PLEG): container finished" podID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" containerID="d98dd3d6e2b05121525d45b8195c09849450fd7eab109f3419dd7b39593529b5" exitCode=60 Oct 30 05:30:12.611186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.611161 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" event={"ID":"f8a5dcff-eb8e-418d-b0ea-26155015b2aa","Type":"ContainerDied","Data":"d98dd3d6e2b05121525d45b8195c09849450fd7eab109f3419dd7b39593529b5"} Oct 30 05:30:12.611490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.611466 6587 scope.go:122] "RemoveContainer" containerID="d98dd3d6e2b05121525d45b8195c09849450fd7eab109f3419dd7b39593529b5" Oct 30 05:30:12.611812 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:12.611788 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7_odh-ai-gateway-infra(f8a5dcff-eb8e-418d-b0ea-26155015b2aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" podUID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" Oct 30 05:30:12.697901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.697871 6587 scope.go:122] "RemoveContainer" containerID="0512908cf7d60b69ae0b7af0f29e7a19fbfb2df0f704181fd3d2859012142797" Oct 30 05:30:12.701399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.700908 6587 generic.go:356] "Generic (PLEG): container finished" podID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" containerID="9dfe1afa8e3b84c5752b126f61150f3d5948c6fe2f71ac360b37c64d892e385c" exitCode=60 Oct 30 05:30:12.701399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.701074 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" event={"ID":"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2","Type":"ContainerDied","Data":"9dfe1afa8e3b84c5752b126f61150f3d5948c6fe2f71ac360b37c64d892e385c"} Oct 30 05:30:12.701774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.701735 6587 scope.go:122] "RemoveContainer" containerID="9dfe1afa8e3b84c5752b126f61150f3d5948c6fe2f71ac360b37c64d892e385c" Oct 30 05:30:12.702664 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:12.702625 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w_odh-ai-gateway-infra(f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" Oct 30 05:30:12.759384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.759173 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v"] Oct 30 05:30:12.799553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.799516 6587 generic.go:356] "Generic (PLEG): container finished" podID="da52a806-feb6-4661-b0bc-89296f15a88e" containerID="7c1a93f168be30e2b5923a428e9a6750b27e09eafaef2334004e508a83513dc5" exitCode=60 Oct 30 05:30:12.799855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.799829 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" event={"ID":"da52a806-feb6-4661-b0bc-89296f15a88e","Type":"ContainerDied","Data":"7c1a93f168be30e2b5923a428e9a6750b27e09eafaef2334004e508a83513dc5"} Oct 30 05:30:12.801601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.801573 6587 scope.go:122] "RemoveContainer" containerID="7c1a93f168be30e2b5923a428e9a6750b27e09eafaef2334004e508a83513dc5" Oct 30 05:30:12.803100 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:12.802342 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r_odh-ai-gateway-infra(da52a806-feb6-4661-b0bc-89296f15a88e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" podUID="da52a806-feb6-4661-b0bc-89296f15a88e" Oct 30 05:30:12.863383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.839045 6587 scope.go:122] "RemoveContainer" containerID="78d98eabc9f1754c0b18ac55f41309f43bad97b1088189813efdcb9a54ac0b4e" Oct 30 05:30:12.906823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.906431 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 05:30:12.906823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.906485 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" podUID="54424663-3cda-49f5-a298-88beadbf9343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 05:30:12.963914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.949477 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 05:30:12.963914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:12.949538 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" podUID="5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 05:30:13.075884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:13.043599 6587 scope.go:122] "RemoveContainer" containerID="df9cc060d94c839d478bc76c9c64508d1c1f105cbc8efab49d81b2687b3f0e6a" Oct 30 05:30:13.784447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:13.759647 6587 scope.go:122] "RemoveContainer" containerID="0aa79e5ea09a94e586ce85d03019264024e1c4281340c0335d3076c9db5913d8" Oct 30 05:30:13.906491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:13.906244 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 05:30:13.906491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:13.906300 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" podUID="54424663-3cda-49f5-a298-88beadbf9343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 05:30:13.942384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:13.917337 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 05:30:13.942384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:13.917405 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" podUID="5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 05:30:14.009916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.009703 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" event={"ID":"fc5c804d-fffb-4fcc-8686-080b8f61703c","Type":"ContainerStarted","Data":"e63f2aad23c74d9d3e5a2a552023f9858aa20d8d7208bdf2ce8f39495a33d093"} Oct 30 05:30:14.021644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.021620 6587 scope.go:122] "RemoveContainer" containerID="90e61220ab6edf3fc76321bd6217881358331a6b3d7f59516bb51cd918abfb6c" Oct 30 05:30:14.072382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.071158 6587 scope.go:122] "RemoveContainer" containerID="0b4b2b163281afbd9cdcffa9157b643bfba4d7dd4e3e2cada46bac9be5c013ac" Oct 30 05:30:14.072382 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:14.071767 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws_odh-ai-gateway-infra(a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" podUID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" Oct 30 05:30:14.089457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.089417 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5"] Oct 30 05:30:14.092796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.092762 6587 scope.go:122] "RemoveContainer" containerID="f13bb37578f0ebec286232d1b1821f6a5bf1960ba67955bb1eebdc6ceceebe4c" Oct 30 05:30:14.093316 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:14.093287 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6_odh-ai-gateway-infra(a8e85bdf-2d09-4979-8729-6e47998265c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" podUID="a8e85bdf-2d09-4979-8729-6e47998265c9" Oct 30 05:30:14.101151 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.101025 6587 scope.go:122] "RemoveContainer" containerID="d3159df56c691297a72da69a45070a624be9ccf1168d27089ba7ae862e0ff755" Oct 30 05:30:14.101676 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:14.101502 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5_odh-ai-gateway-infra(247f6667-54e3-4e2c-98e7-9d5a85641098)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" podUID="247f6667-54e3-4e2c-98e7-9d5a85641098" Oct 30 05:30:14.105687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.105661 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5"] Oct 30 05:30:14.105838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.105821 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" Oct 30 05:30:14.107353 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.107333 6587 scope.go:122] "RemoveContainer" containerID="590df08b6555df99bb439076ebdd17547c9d29472d53d52115795fbcc8038ea4" Oct 30 05:30:14.107875 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:14.107846 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq_odh-ai-gateway-infra(bdd4fcaf-282e-4541-8da1-872b78f3a564)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" podUID="bdd4fcaf-282e-4541-8da1-872b78f3a564" Oct 30 05:30:14.110710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.110690 6587 scope.go:122] "RemoveContainer" containerID="f9246019a66748a7aa7fa069caff20210e419a6de517568b087a2bb878d740ee" Oct 30 05:30:14.111169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.111151 6587 scope.go:122] "RemoveContainer" containerID="9dfe1afa8e3b84c5752b126f61150f3d5948c6fe2f71ac360b37c64d892e385c" Oct 30 05:30:14.111320 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:14.111188 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr_odh-ai-gateway-infra(32594502-7640-4be4-86c2-8a21c066f46c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" podUID="32594502-7640-4be4-86c2-8a21c066f46c" Oct 30 05:30:14.111458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.111342 6587 scope.go:122] "RemoveContainer" containerID="7c1a93f168be30e2b5923a428e9a6750b27e09eafaef2334004e508a83513dc5" Oct 30 05:30:14.111582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.111564 6587 scope.go:122] "RemoveContainer" containerID="d98dd3d6e2b05121525d45b8195c09849450fd7eab109f3419dd7b39593529b5" Oct 30 05:30:14.111983 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:14.111960 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7_odh-ai-gateway-infra(f8a5dcff-eb8e-418d-b0ea-26155015b2aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" podUID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" Oct 30 05:30:14.112104 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:14.111636 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r_odh-ai-gateway-infra(da52a806-feb6-4661-b0bc-89296f15a88e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" podUID="da52a806-feb6-4661-b0bc-89296f15a88e" Oct 30 05:30:14.112202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.112128 6587 scope.go:122] "RemoveContainer" containerID="c751fb4c2d505558460f1d46bad1ce33f94ae5fea988529db409925cdc32075b" Oct 30 05:30:14.112610 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:14.112585 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq_odh-ai-gateway-infra(b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" podUID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" Oct 30 05:30:14.112882 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:14.112823 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w_odh-ai-gateway-infra(f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" Oct 30 05:30:14.142379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.141339 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ec92c7\"" Oct 30 05:30:14.185935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.183635 6587 scope.go:122] "RemoveContainer" containerID="5b3a1f15cf74c92157fa633cb2228ae6132b3443191d4df2355bd29a27b6d8ed" Oct 30 05:30:14.185935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.184540 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ec92c7-dockercfg-w9d9z\"" Oct 30 05:30:14.202218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.201220 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7dfe16d-d341-4c77-9279-0e95437cdf49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5\" (UID: \"f7dfe16d-d341-4c77-9279-0e95437cdf49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" Oct 30 05:30:14.202218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.201435 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckr8r\" (UniqueName: \"kubernetes.io/projected/f7dfe16d-d341-4c77-9279-0e95437cdf49-kube-api-access-ckr8r\") pod \"payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5\" (UID: \"f7dfe16d-d341-4c77-9279-0e95437cdf49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" Oct 30 05:30:14.335187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.310906 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7dfe16d-d341-4c77-9279-0e95437cdf49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5\" (UID: \"f7dfe16d-d341-4c77-9279-0e95437cdf49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" Oct 30 05:30:14.335187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.311023 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckr8r\" (UniqueName: \"kubernetes.io/projected/f7dfe16d-d341-4c77-9279-0e95437cdf49-kube-api-access-ckr8r\") pod \"payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5\" (UID: \"f7dfe16d-d341-4c77-9279-0e95437cdf49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" Oct 30 05:30:14.335187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.323017 6587 scope.go:122] "RemoveContainer" containerID="8457914ad9666639101d96b5c53d129fab2aeac5fa97e11a4112c5149059c0f7" Oct 30 05:30:14.400384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.368716 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckr8r\" (UniqueName: \"kubernetes.io/projected/f7dfe16d-d341-4c77-9279-0e95437cdf49-kube-api-access-ckr8r\") pod \"payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5\" (UID: \"f7dfe16d-d341-4c77-9279-0e95437cdf49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" Oct 30 05:30:14.400384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.374048 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7dfe16d-d341-4c77-9279-0e95437cdf49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5\" (UID: \"f7dfe16d-d341-4c77-9279-0e95437cdf49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" Oct 30 05:30:14.509417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.509383 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" Oct 30 05:30:14.906393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.905814 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 05:30:14.906393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.905872 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" podUID="54424663-3cda-49f5-a298-88beadbf9343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 05:30:14.910347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.910247 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 05:30:14.910347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:14.910297 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" podUID="5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 05:30:15.133274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:15.133229 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5"] Oct 30 05:30:15.180083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:15.180049 6587 generic.go:356] "Generic (PLEG): container finished" podID="fc5c804d-fffb-4fcc-8686-080b8f61703c" containerID="99fbffb904e126155b5d9cbfab7d40f2ab1d6d78a5ec2da918bafdfb76f2fccc" exitCode=0 Oct 30 05:30:15.180221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:15.180206 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" event={"ID":"fc5c804d-fffb-4fcc-8686-080b8f61703c","Type":"ContainerDied","Data":"99fbffb904e126155b5d9cbfab7d40f2ab1d6d78a5ec2da918bafdfb76f2fccc"} Oct 30 05:30:15.181933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:15.181908 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" event={"ID":"f7dfe16d-d341-4c77-9279-0e95437cdf49","Type":"ContainerStarted","Data":"f767a90230efe556a95d01804bef0a909982622134866c6b28c81f10c37d4d96"} Oct 30 05:30:15.910620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:15.910570 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:15.911462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:15.911438 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:15.912679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:15.912656 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:30:15.916565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:15.916543 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:15.916860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:15.916828 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:15.917793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:15.917775 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:30:16.240768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:16.240542 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" event={"ID":"f7dfe16d-d341-4c77-9279-0e95437cdf49","Type":"ContainerStarted","Data":"5c9eb5fa118e0bb8322ccfe0bedbf52ee51c5dfac8f1ada24d9237aeaf2fdeb0"} Oct 30 05:30:16.243184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:16.243105 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" Oct 30 05:30:16.298759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:16.298701 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" podStartSLOduration=3.298683276 podStartE2EDuration="3.298683276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:16.29722052 +0000 UTC m=+2740.824987307" watchObservedRunningTime="2026-10-30 05:30:16.298683276 +0000 UTC m=+2740.826450063" Oct 30 05:30:16.734580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:16.734545 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" Oct 30 05:30:16.859855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:16.859823 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5c804d-fffb-4fcc-8686-080b8f61703c-kube-api-access-c4vld\" (UniqueName: \"kubernetes.io/projected/fc5c804d-fffb-4fcc-8686-080b8f61703c-kube-api-access-c4vld\") pod \"fc5c804d-fffb-4fcc-8686-080b8f61703c\" (UID: \"fc5c804d-fffb-4fcc-8686-080b8f61703c\") " Oct 30 05:30:16.861146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:16.859941 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-xks-ca\") pod \"fc5c804d-fffb-4fcc-8686-080b8f61703c\" (UID: \"fc5c804d-fffb-4fcc-8686-080b8f61703c\") " Oct 30 05:30:16.861146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:16.860013 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-openshift-ca\") pod \"fc5c804d-fffb-4fcc-8686-080b8f61703c\" (UID: \"fc5c804d-fffb-4fcc-8686-080b8f61703c\") " Oct 30 05:30:16.861146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:16.860459 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-openshift-ca" pod "fc5c804d-fffb-4fcc-8686-080b8f61703c" (UID: "fc5c804d-fffb-4fcc-8686-080b8f61703c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:16.861146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:16.860877 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:17.025751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.025666 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-xks-ca" pod "fc5c804d-fffb-4fcc-8686-080b8f61703c" (UID: "fc5c804d-fffb-4fcc-8686-080b8f61703c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:30:17.026117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.025754 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5c804d-fffb-4fcc-8686-080b8f61703c-kube-api-access-c4vld" pod "fc5c804d-fffb-4fcc-8686-080b8f61703c" (UID: "fc5c804d-fffb-4fcc-8686-080b8f61703c"). InnerVolumeSpecName "kube-api-access-c4vld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:17.065384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.065322 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4vld\" (UniqueName: \"kubernetes.io/projected/fc5c804d-fffb-4fcc-8686-080b8f61703c-kube-api-access-c4vld\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:17.065384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.065377 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5c804d-fffb-4fcc-8686-080b8f61703c-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:17.238796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.238769 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" Oct 30 05:30:17.239205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.238844 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v" event={"ID":"fc5c804d-fffb-4fcc-8686-080b8f61703c","Type":"ContainerDied","Data":"e63f2aad23c74d9d3e5a2a552023f9858aa20d8d7208bdf2ce8f39495a33d093"} Oct 30 05:30:17.239205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.239027 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e63f2aad23c74d9d3e5a2a552023f9858aa20d8d7208bdf2ce8f39495a33d093" Oct 30 05:30:17.922552 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.922514 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2"] Oct 30 05:30:17.924239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.924206 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fc5c804d-fffb-4fcc-8686-080b8f61703c" containerName="revoke-keys" Oct 30 05:30:17.924239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.924237 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5c804d-fffb-4fcc-8686-080b8f61703c" containerName="revoke-keys" Oct 30 05:30:17.924589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.924569 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5c804d-fffb-4fcc-8686-080b8f61703c" containerName="revoke-keys" Oct 30 05:30:17.938290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.938260 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.011675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.971160 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2"] Oct 30 05:30:18.011675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.982814 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.011675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.982961 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.011675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.982993 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.011675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.983022 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmfxm\" (UniqueName: \"kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-kube-api-access-cmfxm\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.011675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.983166 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.011675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.983236 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.011675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.983296 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.011675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.983443 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.011675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:17.983478 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.085027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.084981 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.085027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.085038 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.085326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.085080 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmfxm\" (UniqueName: \"kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-kube-api-access-cmfxm\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.085326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.085157 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.085326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.085227 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.085326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.085302 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.085659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.085467 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.085659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.085502 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.085659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.085475 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.085659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.085609 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.085901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.085884 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.086125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.086104 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.086125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.086110 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.091336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.086754 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.092788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.092764 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.093114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.093095 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.108091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.108057 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.108236 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.108175 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmfxm\" (UniqueName: \"kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-kube-api-access-cmfxm\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.313298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.313271 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:18.740639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.740534 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:18.799718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.799681 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2"] Oct 30 05:30:18.819323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.819265 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:18.819534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:18.819406 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:19.265808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:19.265718 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td"] Oct 30 05:30:19.270242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:19.270205 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" event={"ID":"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee","Type":"ContainerStarted","Data":"4e8358ce19edfc71ed77677c149dda992ea9fe91ef2c2430199344c4077761e8"} Oct 30 05:30:19.270561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:19.270534 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" event={"ID":"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee","Type":"ContainerStarted","Data":"c42efb55e6e7251d26e88229459ccd32ad84bd1e5fcada47f0dbd730506627fa"} Oct 30 05:30:19.281772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:19.281732 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" podUID="d11c4e59-293e-41b7-9f29-157a813083ad" containerName="istio-proxy" containerID="cri-o://80974c739aa434b1c7fcefa1218abc9c5b02e559ee6419f006cbb332ad5f9a06" gracePeriod=30 Oct 30 05:30:19.313510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:19.313483 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:19.383994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:19.383911 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" podStartSLOduration=2.383889147 podStartE2EDuration="2.383889147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:19.367544386 +0000 UTC m=+2743.895311174" watchObservedRunningTime="2026-10-30 05:30:19.383889147 +0000 UTC m=+2743.911655933" Oct 30 05:30:19.909342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:19.909313 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" Oct 30 05:30:20.319380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:20.315338 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" start-of-body= Oct 30 05:30:20.339708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:20.339660 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" podUID="5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" Oct 30 05:30:20.582434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:20.578572 6587 scope.go:122] "RemoveContainer" containerID="a8fb07f081d08fcdc173de8362fcf523a219fc28567a379034ba4db2f93481c7" Oct 30 05:30:20.596809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:20.586105 6587 scope.go:122] "RemoveContainer" containerID="13e47ec469fa4296347394e0a6cc573144685106da734f2e28bb86797ade77a9" Oct 30 05:30:20.615955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:20.602245 6587 scope.go:122] "RemoveContainer" containerID="67af5a5a7c30555c708e73ccce5734eb8a38051e333838e43aad3991535fb98b" Oct 30 05:30:20.615955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:20.603075 6587 scope.go:122] "RemoveContainer" containerID="c448be4d7f40fb822feda966b302b539ab90f7d40da539c0f1ff8d6001e4122e" Oct 30 05:30:20.663537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:20.617678 6587 scope.go:122] "RemoveContainer" containerID="ea4de1936b55ebca8aa73a4e228f9efd89998a9acff6a6e256d6e6b09fc24abb" Oct 30 05:30:21.313730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:21.313695 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" start-of-body= Oct 30 05:30:21.313958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:21.313751 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" podUID="5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" Oct 30 05:30:21.540543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:21.405982 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" Oct 30 05:30:22.062293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.062257 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6"] Oct 30 05:30:22.068461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.068434 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06eabedb-rn8z6"] Oct 30 05:30:22.313807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.313722 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" start-of-body= Oct 30 05:30:22.313807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.313778 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" podUID="5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" Oct 30 05:30:22.397231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.397186 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" event={"ID":"10cb04e6-7c18-43e9-ac64-fdcbbac05b78","Type":"ContainerStarted","Data":"41ea2fa53ad884247bc7ce3b2d6455cf0f288113cc749dead2c196e36d4705ed"} Oct 30 05:30:22.407339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.407268 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" event={"ID":"da8b3d3c-c341-4dab-9c82-f85d8e203f54","Type":"ContainerStarted","Data":"1aa5501be286111026ba70fd445789fc353d5152163fd44c3a1e51929702f3fb"} Oct 30 05:30:22.413381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.411165 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" event={"ID":"bb155f3e-3d01-431a-84a5-307d1109c9fa","Type":"ContainerStarted","Data":"f8cdd13323ab56048bbb5f252fc3f4d6bf1d63f55292bb4bd85e7245a4bc3a72"} Oct 30 05:30:22.425292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.414893 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" event={"ID":"d0de3126-e112-4a37-aa81-0f312100c7b6","Type":"ContainerStarted","Data":"febfea182a61173f709be4a27adc8fd723cc44b93106f670e668b821eb8e2fc8"} Oct 30 05:30:22.425292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.421951 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" event={"ID":"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1","Type":"ContainerStarted","Data":"3ce3264c072e25a95bdfcb45c66ed95e16100f317edc26d97599f51cde46770d"} Oct 30 05:30:22.482252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.476869 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" podStartSLOduration=22.476849847 podStartE2EDuration="22.476849847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:22.446199912 +0000 UTC m=+2746.973966698" watchObservedRunningTime="2026-10-30 05:30:22.476849847 +0000 UTC m=+2747.004616634" Oct 30 05:30:22.503751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.503690 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" podStartSLOduration=22.503671809 podStartE2EDuration="22.503671809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:22.499627999 +0000 UTC m=+2747.027394785" watchObservedRunningTime="2026-10-30 05:30:22.503671809 +0000 UTC m=+2747.031438599" Oct 30 05:30:22.634442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.541209 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:30:22.634442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.541280 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:30:22.655238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.648618 6587 scope.go:122] "RemoveContainer" containerID="81a184c323a7a8dfe0de6fd3625a538b0f7b553043bc4573c799b19a0cffc409" Oct 30 05:30:22.655238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.649681 6587 scope.go:122] "RemoveContainer" containerID="60b6b759b97d98e10e06c929a37eef3407a88ec45a0e312d34dbcf593e321448" Oct 30 05:30:22.732747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:22.722785 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" podStartSLOduration=22.722762992 podStartE2EDuration="22.722762992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:22.705971971 +0000 UTC m=+2747.233738758" watchObservedRunningTime="2026-10-30 05:30:22.722762992 +0000 UTC m=+2747.250529779" Oct 30 05:30:23.315045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.314978 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" start-of-body= Oct 30 05:30:23.319488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.319449 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" podUID="5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" Oct 30 05:30:23.472431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.472352 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h"] Oct 30 05:30:23.485373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.485317 6587 generic.go:356] "Generic (PLEG): container finished" podID="bb155f3e-3d01-431a-84a5-307d1109c9fa" containerID="f8cdd13323ab56048bbb5f252fc3f4d6bf1d63f55292bb4bd85e7245a4bc3a72" exitCode=60 Oct 30 05:30:23.485574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.485518 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" event={"ID":"bb155f3e-3d01-431a-84a5-307d1109c9fa","Type":"ContainerDied","Data":"f8cdd13323ab56048bbb5f252fc3f4d6bf1d63f55292bb4bd85e7245a4bc3a72"} Oct 30 05:30:23.485641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.485586 6587 scope.go:122] "RemoveContainer" containerID="13e47ec469fa4296347394e0a6cc573144685106da734f2e28bb86797ade77a9" Oct 30 05:30:23.487152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.486126 6587 scope.go:122] "RemoveContainer" containerID="f8cdd13323ab56048bbb5f252fc3f4d6bf1d63f55292bb4bd85e7245a4bc3a72" Oct 30 05:30:23.487152 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:23.486675 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq_odh-ai-gateway-infra(bb155f3e-3d01-431a-84a5-307d1109c9fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" Oct 30 05:30:23.493744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.489914 6587 generic.go:356] "Generic (PLEG): container finished" podID="d0de3126-e112-4a37-aa81-0f312100c7b6" containerID="febfea182a61173f709be4a27adc8fd723cc44b93106f670e668b821eb8e2fc8" exitCode=60 Oct 30 05:30:23.493744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.489986 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" event={"ID":"d0de3126-e112-4a37-aa81-0f312100c7b6","Type":"ContainerDied","Data":"febfea182a61173f709be4a27adc8fd723cc44b93106f670e668b821eb8e2fc8"} Oct 30 05:30:23.493744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.490498 6587 scope.go:122] "RemoveContainer" containerID="febfea182a61173f709be4a27adc8fd723cc44b93106f670e668b821eb8e2fc8" Oct 30 05:30:23.493744 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:23.490941 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g_odh-ai-gateway-infra(d0de3126-e112-4a37-aa81-0f312100c7b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" podUID="d0de3126-e112-4a37-aa81-0f312100c7b6" Oct 30 05:30:23.493744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.492155 6587 generic.go:356] "Generic (PLEG): container finished" podID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" containerID="3ce3264c072e25a95bdfcb45c66ed95e16100f317edc26d97599f51cde46770d" exitCode=60 Oct 30 05:30:23.493744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.492420 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" event={"ID":"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1","Type":"ContainerDied","Data":"3ce3264c072e25a95bdfcb45c66ed95e16100f317edc26d97599f51cde46770d"} Oct 30 05:30:23.493744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.492815 6587 scope.go:122] "RemoveContainer" containerID="3ce3264c072e25a95bdfcb45c66ed95e16100f317edc26d97599f51cde46770d" Oct 30 05:30:23.493744 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:23.493212 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888970-qrbbc_odh-ai-gateway-infra(cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" podUID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" Oct 30 05:30:23.496123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.496096 6587 generic.go:356] "Generic (PLEG): container finished" podID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" containerID="41ea2fa53ad884247bc7ce3b2d6455cf0f288113cc749dead2c196e36d4705ed" exitCode=60 Oct 30 05:30:23.496238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.496221 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" event={"ID":"10cb04e6-7c18-43e9-ac64-fdcbbac05b78","Type":"ContainerDied","Data":"41ea2fa53ad884247bc7ce3b2d6455cf0f288113cc749dead2c196e36d4705ed"} Oct 30 05:30:23.501180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.500415 6587 generic.go:356] "Generic (PLEG): container finished" podID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" containerID="1aa5501be286111026ba70fd445789fc353d5152163fd44c3a1e51929702f3fb" exitCode=60 Oct 30 05:30:23.501180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.500455 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" event={"ID":"da8b3d3c-c341-4dab-9c82-f85d8e203f54","Type":"ContainerDied","Data":"1aa5501be286111026ba70fd445789fc353d5152163fd44c3a1e51929702f3fb"} Oct 30 05:30:23.501180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.500787 6587 scope.go:122] "RemoveContainer" containerID="1aa5501be286111026ba70fd445789fc353d5152163fd44c3a1e51929702f3fb" Oct 30 05:30:23.501180 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:23.501126 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2_odh-ai-gateway-infra(da8b3d3c-c341-4dab-9c82-f85d8e203f54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" podUID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" Oct 30 05:30:23.514469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.514412 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2"] Oct 30 05:30:23.588141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.584285 6587 scope.go:122] "RemoveContainer" containerID="f13bb37578f0ebec286232d1b1821f6a5bf1960ba67955bb1eebdc6ceceebe4c" Oct 30 05:30:23.588141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.586071 6587 scope.go:122] "RemoveContainer" containerID="590df08b6555df99bb439076ebdd17547c9d29472d53d52115795fbcc8038ea4" Oct 30 05:30:23.588141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.586617 6587 scope.go:122] "RemoveContainer" containerID="d3159df56c691297a72da69a45070a624be9ccf1168d27089ba7ae862e0ff755" Oct 30 05:30:23.597441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.597418 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da8f630-2be6-4ec7-8c48-3c69b536dffd" path="/var/lib/kubelet/pods/8da8f630-2be6-4ec7-8c48-3c69b536dffd/volumes" Oct 30 05:30:23.598002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.597987 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq"] Oct 30 05:30:23.708019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.707759 6587 scope.go:122] "RemoveContainer" containerID="67af5a5a7c30555c708e73ccce5734eb8a38051e333838e43aad3991535fb98b" Oct 30 05:30:23.874663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.769271 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g"] Oct 30 05:30:23.874663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.802960 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc"] Oct 30 05:30:23.957933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.901681 6587 scope.go:122] "RemoveContainer" containerID="a8fb07f081d08fcdc173de8362fcf523a219fc28567a379034ba4db2f93481c7" Oct 30 05:30:23.999599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:23.999574 6587 scope.go:122] "RemoveContainer" containerID="c448be4d7f40fb822feda966b302b539ab90f7d40da539c0f1ff8d6001e4122e" Oct 30 05:30:24.271677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:24.201388 6587 scope.go:122] "RemoveContainer" containerID="ea4de1936b55ebca8aa73a4e228f9efd89998a9acff6a6e256d6e6b09fc24abb" Oct 30 05:30:24.326482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:24.313963 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" start-of-body= Oct 30 05:30:24.326482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:24.314168 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" podUID="5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" Oct 30 05:30:24.555733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:24.555683 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" event={"ID":"9f5876d8-6d01-4a99-950d-8af2c95c3bf8","Type":"ContainerStarted","Data":"6c0bf05e648c36feef5fcf5b94ae170499459f2cc49e82350fea323728cbaa1c"} Oct 30 05:30:24.565407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:24.565327 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" event={"ID":"5aabb8f8-1678-4f27-945e-40e30674980f","Type":"ContainerStarted","Data":"b714c622fb7804c492bc398cb93c11377eb988786fb2c7c8e61418cdbc1a62e9"} Oct 30 05:30:24.575697 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:24.573994 6587 scope.go:122] "RemoveContainer" containerID="c751fb4c2d505558460f1d46bad1ce33f94ae5fea988529db409925cdc32075b" Oct 30 05:30:24.579929 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:24.579623 6587 scope.go:122] "RemoveContainer" containerID="0b4b2b163281afbd9cdcffa9157b643bfba4d7dd4e3e2cada46bac9be5c013ac" Oct 30 05:30:24.580539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:24.580516 6587 scope.go:122] "RemoveContainer" containerID="d98dd3d6e2b05121525d45b8195c09849450fd7eab109f3419dd7b39593529b5" Oct 30 05:30:24.580936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:24.580903 6587 scope.go:122] "RemoveContainer" containerID="7c1a93f168be30e2b5923a428e9a6750b27e09eafaef2334004e508a83513dc5" Oct 30 05:30:24.581178 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:24.581159 6587 scope.go:122] "RemoveContainer" containerID="7c31976cc4288ae5f0cdf876edc5b88b414518a4b846789f4dc472a5684f5178" Oct 30 05:30:24.943641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:24.782964 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 05:30:24.943641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:24.783025 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" podUID="d11c4e59-293e-41b7-9f29-157a813083ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 05:30:24.943641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:24.897555 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" podStartSLOduration=24.897536743 podStartE2EDuration="24.897536743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:24.893822352 +0000 UTC m=+2749.421589139" watchObservedRunningTime="2026-10-30 05:30:24.897536743 +0000 UTC m=+2749.425303530" Oct 30 05:30:25.314695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.313756 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" start-of-body= Oct 30 05:30:25.314695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.313804 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" podUID="5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" Oct 30 05:30:25.584229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.584141 6587 scope.go:122] "RemoveContainer" containerID="9dfe1afa8e3b84c5752b126f61150f3d5948c6fe2f71ac360b37c64d892e385c" Oct 30 05:30:25.584876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.584856 6587 scope.go:122] "RemoveContainer" containerID="f9246019a66748a7aa7fa069caff20210e419a6de517568b087a2bb878d740ee" Oct 30 05:30:25.590288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.590254 6587 generic.go:356] "Generic (PLEG): container finished" podID="d11c4e59-293e-41b7-9f29-157a813083ad" containerID="80974c739aa434b1c7fcefa1218abc9c5b02e559ee6419f006cbb332ad5f9a06" exitCode=0 Oct 30 05:30:25.590450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.590389 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" event={"ID":"d11c4e59-293e-41b7-9f29-157a813083ad","Type":"ContainerDied","Data":"80974c739aa434b1c7fcefa1218abc9c5b02e559ee6419f006cbb332ad5f9a06"} Oct 30 05:30:25.592777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.592755 6587 generic.go:356] "Generic (PLEG): container finished" podID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" containerID="6c0bf05e648c36feef5fcf5b94ae170499459f2cc49e82350fea323728cbaa1c" exitCode=60 Oct 30 05:30:25.601512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.592824 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" event={"ID":"9f5876d8-6d01-4a99-950d-8af2c95c3bf8","Type":"ContainerDied","Data":"6c0bf05e648c36feef5fcf5b94ae170499459f2cc49e82350fea323728cbaa1c"} Oct 30 05:30:25.601512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.592870 6587 scope.go:122] "RemoveContainer" containerID="81a184c323a7a8dfe0de6fd3625a538b0f7b553043bc4573c799b19a0cffc409" Oct 30 05:30:25.601512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.593211 6587 scope.go:122] "RemoveContainer" containerID="6c0bf05e648c36feef5fcf5b94ae170499459f2cc49e82350fea323728cbaa1c" Oct 30 05:30:25.601512 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:25.593574 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx_odh-ai-gateway-infra(9f5876d8-6d01-4a99-950d-8af2c95c3bf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" podUID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" Oct 30 05:30:25.601512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.595451 6587 generic.go:356] "Generic (PLEG): container finished" podID="a8e85bdf-2d09-4979-8729-6e47998265c9" containerID="1f2df7f492356eba8832a50ad89c44c9cbc987db6a68c31f49962af819a80ae0" exitCode=60 Oct 30 05:30:25.601512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.595524 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" event={"ID":"a8e85bdf-2d09-4979-8729-6e47998265c9","Type":"ContainerDied","Data":"1f2df7f492356eba8832a50ad89c44c9cbc987db6a68c31f49962af819a80ae0"} Oct 30 05:30:25.601512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.595875 6587 scope.go:122] "RemoveContainer" containerID="1f2df7f492356eba8832a50ad89c44c9cbc987db6a68c31f49962af819a80ae0" Oct 30 05:30:25.601512 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:25.596172 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6_odh-ai-gateway-infra(a8e85bdf-2d09-4979-8729-6e47998265c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" podUID="a8e85bdf-2d09-4979-8729-6e47998265c9" Oct 30 05:30:25.601512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.600392 6587 generic.go:356] "Generic (PLEG): container finished" podID="247f6667-54e3-4e2c-98e7-9d5a85641098" containerID="81f6711d3f60e410db01a14a4f0e1adfe75df7b565263b31c6d1c0f4b8851197" exitCode=60 Oct 30 05:30:25.601512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.600460 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" event={"ID":"247f6667-54e3-4e2c-98e7-9d5a85641098","Type":"ContainerDied","Data":"81f6711d3f60e410db01a14a4f0e1adfe75df7b565263b31c6d1c0f4b8851197"} Oct 30 05:30:25.601512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.600843 6587 scope.go:122] "RemoveContainer" containerID="81f6711d3f60e410db01a14a4f0e1adfe75df7b565263b31c6d1c0f4b8851197" Oct 30 05:30:25.601512 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:25.601160 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5_odh-ai-gateway-infra(247f6667-54e3-4e2c-98e7-9d5a85641098)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" podUID="247f6667-54e3-4e2c-98e7-9d5a85641098" Oct 30 05:30:25.602690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.602670 6587 generic.go:356] "Generic (PLEG): container finished" podID="bdd4fcaf-282e-4541-8da1-872b78f3a564" containerID="bde25f8dc69a19b73438e9c2272659d950942fed3de7a2c2ed82ab7d39316c78" exitCode=60 Oct 30 05:30:25.602790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.602743 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" event={"ID":"bdd4fcaf-282e-4541-8da1-872b78f3a564","Type":"ContainerDied","Data":"bde25f8dc69a19b73438e9c2272659d950942fed3de7a2c2ed82ab7d39316c78"} Oct 30 05:30:25.603160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.603081 6587 scope.go:122] "RemoveContainer" containerID="bde25f8dc69a19b73438e9c2272659d950942fed3de7a2c2ed82ab7d39316c78" Oct 30 05:30:25.620925 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:25.603411 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq_odh-ai-gateway-infra(bdd4fcaf-282e-4541-8da1-872b78f3a564)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" podUID="bdd4fcaf-282e-4541-8da1-872b78f3a564" Oct 30 05:30:25.620925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.605894 6587 generic.go:356] "Generic (PLEG): container finished" podID="5aabb8f8-1678-4f27-945e-40e30674980f" containerID="b714c622fb7804c492bc398cb93c11377eb988786fb2c7c8e61418cdbc1a62e9" exitCode=60 Oct 30 05:30:25.620925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.605970 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" event={"ID":"5aabb8f8-1678-4f27-945e-40e30674980f","Type":"ContainerDied","Data":"b714c622fb7804c492bc398cb93c11377eb988786fb2c7c8e61418cdbc1a62e9"} Oct 30 05:30:25.620925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.606318 6587 scope.go:122] "RemoveContainer" containerID="b714c622fb7804c492bc398cb93c11377eb988786fb2c7c8e61418cdbc1a62e9" Oct 30 05:30:25.620925 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:25.606612 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb_odh-ai-gateway-infra(5aabb8f8-1678-4f27-945e-40e30674980f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" podUID="5aabb8f8-1678-4f27-945e-40e30674980f" Oct 30 05:30:25.642560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.642523 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx"] Oct 30 05:30:25.896696 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.896586 6587 scope.go:122] "RemoveContainer" containerID="f13bb37578f0ebec286232d1b1821f6a5bf1960ba67955bb1eebdc6ceceebe4c" Oct 30 05:30:25.919969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:25.919936 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb"] Oct 30 05:30:26.051991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.051956 6587 scope.go:122] "RemoveContainer" containerID="d3159df56c691297a72da69a45070a624be9ccf1168d27089ba7ae862e0ff755" Oct 30 05:30:26.146317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.146287 6587 scope.go:122] "RemoveContainer" containerID="590df08b6555df99bb439076ebdd17547c9d29472d53d52115795fbcc8038ea4" Oct 30 05:30:26.369943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.367681 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:26.370514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.370423 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:26.373814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.373574 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:30:26.400163 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.400135 6587 scope.go:122] "RemoveContainer" containerID="60b6b759b97d98e10e06c929a37eef3407a88ec45a0e312d34dbcf593e321448" Oct 30 05:30:26.603878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.603633 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" Oct 30 05:30:26.631999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.631968 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" Oct 30 05:30:26.645067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.635122 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" event={"ID":"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e","Type":"ContainerStarted","Data":"318b995efb87d9acddc576419f6e89707f92089540da02b76cbb68149222e5aa"} Oct 30 05:30:26.645067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.637058 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" event={"ID":"f8a5dcff-eb8e-418d-b0ea-26155015b2aa","Type":"ContainerStarted","Data":"2152c486258adf904bbcacb03eb90cbf134b5bae083306e9dcae67f2ba17332b"} Oct 30 05:30:26.645067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.638956 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" event={"ID":"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2","Type":"ContainerStarted","Data":"ee43849457ffd7f6061a3af665203a52f5a9d771729292a7da322476f9486e85"} Oct 30 05:30:26.645067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.640886 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" event={"ID":"da52a806-feb6-4661-b0bc-89296f15a88e","Type":"ContainerStarted","Data":"e2af6d2c7385536b4f4028822fc5cbe0fdaad322439852705de9f3bd794780ba"} Oct 30 05:30:26.645067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.642834 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" event={"ID":"d0de3126-e112-4a37-aa81-0f312100c7b6","Type":"ContainerDied","Data":"0df97b9994260f473c4d6e9787c8d2ca506f8b5ddf4a39e4e9d68b536a32b3fb"} Oct 30 05:30:26.645067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.642888 6587 scope.go:122] "RemoveContainer" containerID="febfea182a61173f709be4a27adc8fd723cc44b93106f670e668b821eb8e2fc8" Oct 30 05:30:26.645067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.642841 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g" Oct 30 05:30:26.645067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.644707 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" event={"ID":"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1","Type":"ContainerDied","Data":"4275ff1fbe2492fbca0ce8768ecec0eb076e4fe7461f4bd4594802fe6628eafa"} Oct 30 05:30:26.645067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.644744 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4275ff1fbe2492fbca0ce8768ecec0eb076e4fe7461f4bd4594802fe6628eafa" Oct 30 05:30:26.674088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.646518 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" event={"ID":"d11c4e59-293e-41b7-9f29-157a813083ad","Type":"ContainerDied","Data":"a9af2d4b043edce8d3350c38ecbdb546c20206db04ac08be78023af276567996"} Oct 30 05:30:26.674088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.646555 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9af2d4b043edce8d3350c38ecbdb546c20206db04ac08be78023af276567996" Oct 30 05:30:26.674088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.648727 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" event={"ID":"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff","Type":"ContainerStarted","Data":"6dfdec7ec20ff383361fcdb322dad39cd6d187e2e8542ff44fb9510c3b0afb7a"} Oct 30 05:30:26.674088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.652603 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" event={"ID":"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8","Type":"ContainerStarted","Data":"c9a21816239d490302de56d8437edb8e8b0d8df8c8dc24c2b58057a3ab54a3b3"} Oct 30 05:30:26.674088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.656451 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" event={"ID":"10cb04e6-7c18-43e9-ac64-fdcbbac05b78","Type":"ContainerDied","Data":"0420f646ce965de8748182e89fff0fc0e410d20b25553682b9ff8024e5e11e72"} Oct 30 05:30:26.674088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.656483 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0420f646ce965de8748182e89fff0fc0e410d20b25553682b9ff8024e5e11e72" Oct 30 05:30:26.674088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.658000 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" event={"ID":"da8b3d3c-c341-4dab-9c82-f85d8e203f54","Type":"ContainerDied","Data":"69be8b1403065bc9dbbb86a5d535eca111eaa15f01a1b6b08496da7ea6166b74"} Oct 30 05:30:26.674088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.658081 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69be8b1403065bc9dbbb86a5d535eca111eaa15f01a1b6b08496da7ea6166b74" Oct 30 05:30:26.674088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.659856 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" Oct 30 05:30:26.674088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.661734 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" Oct 30 05:30:26.674088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.661755 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq" event={"ID":"bb155f3e-3d01-431a-84a5-307d1109c9fa","Type":"ContainerDied","Data":"2fd520249f7a92f5e544f4fc3a3a38e3a28601db8f36d80b0dd5f682ff2f9a0c"} Oct 30 05:30:26.674088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.666344 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" event={"ID":"32594502-7640-4be4-86c2-8a21c066f46c","Type":"ContainerStarted","Data":"766dcc391c2ce2edc17c61a5afeedc516d2c8939a41fb7d29e0f4ad8b9d42160"} Oct 30 05:30:26.695190 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:26.675241 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:30:26.695190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.677149 6587 scope.go:122] "RemoveContainer" containerID="f8cdd13323ab56048bbb5f252fc3f4d6bf1d63f55292bb4bd85e7245a4bc3a72" Oct 30 05:30:26.695190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.689650 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0de3126-e112-4a37-aa81-0f312100c7b6-kube-api-access-ksntd\" (UniqueName: \"kubernetes.io/projected/d0de3126-e112-4a37-aa81-0f312100c7b6-kube-api-access-ksntd\") pod \"d0de3126-e112-4a37-aa81-0f312100c7b6\" (UID: \"d0de3126-e112-4a37-aa81-0f312100c7b6\") " Oct 30 05:30:26.695190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.689841 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0de3126-e112-4a37-aa81-0f312100c7b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0de3126-e112-4a37-aa81-0f312100c7b6-maas-api-service-ca\") pod \"d0de3126-e112-4a37-aa81-0f312100c7b6\" (UID: \"d0de3126-e112-4a37-aa81-0f312100c7b6\") " Oct 30 05:30:26.695190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.689895 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb155f3e-3d01-431a-84a5-307d1109c9fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb155f3e-3d01-431a-84a5-307d1109c9fa-maas-api-service-ca\") pod \"bb155f3e-3d01-431a-84a5-307d1109c9fa\" (UID: \"bb155f3e-3d01-431a-84a5-307d1109c9fa\") " Oct 30 05:30:26.695190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.690012 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb155f3e-3d01-431a-84a5-307d1109c9fa-kube-api-access-k7kf5\" (UniqueName: \"kubernetes.io/projected/bb155f3e-3d01-431a-84a5-307d1109c9fa-kube-api-access-k7kf5\") pod \"bb155f3e-3d01-431a-84a5-307d1109c9fa\" (UID: \"bb155f3e-3d01-431a-84a5-307d1109c9fa\") " Oct 30 05:30:26.695190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.690650 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb155f3e-3d01-431a-84a5-307d1109c9fa-maas-api-service-ca" pod "bb155f3e-3d01-431a-84a5-307d1109c9fa" (UID: "bb155f3e-3d01-431a-84a5-307d1109c9fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:26.695190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.690855 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0de3126-e112-4a37-aa81-0f312100c7b6-maas-api-service-ca" pod "d0de3126-e112-4a37-aa81-0f312100c7b6" (UID: "d0de3126-e112-4a37-aa81-0f312100c7b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:26.695190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.692306 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0de3126-e112-4a37-aa81-0f312100c7b6-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.695190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.692323 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb155f3e-3d01-431a-84a5-307d1109c9fa-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.695190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.692618 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" Oct 30 05:30:26.738721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.738627 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:30:26.756336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.747063 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6"] Oct 30 05:30:26.761416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.759208 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0de3126-e112-4a37-aa81-0f312100c7b6-kube-api-access-ksntd" pod "d0de3126-e112-4a37-aa81-0f312100c7b6" (UID: "d0de3126-e112-4a37-aa81-0f312100c7b6"). InnerVolumeSpecName "kube-api-access-ksntd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:26.761416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.759335 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb155f3e-3d01-431a-84a5-307d1109c9fa-kube-api-access-k7kf5" pod "bb155f3e-3d01-431a-84a5-307d1109c9fa" (UID: "bb155f3e-3d01-431a-84a5-307d1109c9fa"). InnerVolumeSpecName "kube-api-access-k7kf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:26.779129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.778569 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.793912 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-kube-api-access-jl6lm\" (UniqueName: \"kubernetes.io/projected/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-kube-api-access-jl6lm\") pod \"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1\" (UID: \"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1\") " Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.793961 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-credential-socket\") pod \"d11c4e59-293e-41b7-9f29-157a813083ad\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.794015 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-data\") pod \"d11c4e59-293e-41b7-9f29-157a813083ad\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.794079 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-socket\") pod \"d11c4e59-293e-41b7-9f29-157a813083ad\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.794131 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-envoy\") pod \"d11c4e59-293e-41b7-9f29-157a813083ad\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.794163 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-istio-token\" (UniqueName: \"kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-istio-token\") pod \"d11c4e59-293e-41b7-9f29-157a813083ad\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.794187 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8b3d3c-c341-4dab-9c82-f85d8e203f54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8b3d3c-c341-4dab-9c82-f85d8e203f54-maas-api-service-ca\") pod \"da8b3d3c-c341-4dab-9c82-f85d8e203f54\" (UID: \"da8b3d3c-c341-4dab-9c82-f85d8e203f54\") " Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.794206 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8b3d3c-c341-4dab-9c82-f85d8e203f54-kube-api-access-5hltv\" (UniqueName: \"kubernetes.io/projected/da8b3d3c-c341-4dab-9c82-f85d8e203f54-kube-api-access-5hltv\") pod \"da8b3d3c-c341-4dab-9c82-f85d8e203f54\" (UID: \"da8b3d3c-c341-4dab-9c82-f85d8e203f54\") " Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.794229 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d11c4e59-293e-41b7-9f29-157a813083ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d11c4e59-293e-41b7-9f29-157a813083ad-istio-podinfo\") pod \"d11c4e59-293e-41b7-9f29-157a813083ad\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.794302 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-maas-api-service-ca\") pod \"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1\" (UID: \"cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1\") " Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.794327 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d11c4e59-293e-41b7-9f29-157a813083ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d11c4e59-293e-41b7-9f29-157a813083ad-istiod-ca-cert\") pod \"d11c4e59-293e-41b7-9f29-157a813083ad\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.794428 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-certs\") pod \"d11c4e59-293e-41b7-9f29-157a813083ad\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.794487 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-kube-api-access-llchj\" (UniqueName: \"kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-kube-api-access-llchj\") pod \"d11c4e59-293e-41b7-9f29-157a813083ad\" (UID: \"d11c4e59-293e-41b7-9f29-157a813083ad\") " Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.795214 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksntd\" (UniqueName: \"kubernetes.io/projected/d0de3126-e112-4a37-aa81-0f312100c7b6-kube-api-access-ksntd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.795534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.795227 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7kf5\" (UniqueName: \"kubernetes.io/projected/bb155f3e-3d01-431a-84a5-307d1109c9fa-kube-api-access-k7kf5\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.795778 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-socket" pod "d11c4e59-293e-41b7-9f29-157a813083ad" (UID: "d11c4e59-293e-41b7-9f29-157a813083ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.796476 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8b3d3c-c341-4dab-9c82-f85d8e203f54-maas-api-service-ca" pod "da8b3d3c-c341-4dab-9c82-f85d8e203f54" (UID: "da8b3d3c-c341-4dab-9c82-f85d8e203f54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.796552 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-maas-api-service-ca" pod "cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" (UID: "cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.797232 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11c4e59-293e-41b7-9f29-157a813083ad-istiod-ca-cert" pod "d11c4e59-293e-41b7-9f29-157a813083ad" (UID: "d11c4e59-293e-41b7-9f29-157a813083ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.798665 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-credential-socket" pod "d11c4e59-293e-41b7-9f29-157a813083ad" (UID: "d11c4e59-293e-41b7-9f29-157a813083ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.798867 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-data" pod "d11c4e59-293e-41b7-9f29-157a813083ad" (UID: "d11c4e59-293e-41b7-9f29-157a813083ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.799088 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-certs" pod "d11c4e59-293e-41b7-9f29-157a813083ad" (UID: "d11c4e59-293e-41b7-9f29-157a813083ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.800037 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8b3d3c-c341-4dab-9c82-f85d8e203f54-kube-api-access-5hltv" pod "da8b3d3c-c341-4dab-9c82-f85d8e203f54" (UID: "da8b3d3c-c341-4dab-9c82-f85d8e203f54"). InnerVolumeSpecName "kube-api-access-5hltv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.800958 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-envoy" pod "d11c4e59-293e-41b7-9f29-157a813083ad" (UID: "d11c4e59-293e-41b7-9f29-157a813083ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.803297 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-kube-api-access-llchj" pod "d11c4e59-293e-41b7-9f29-157a813083ad" (UID: "d11c4e59-293e-41b7-9f29-157a813083ad"). InnerVolumeSpecName "kube-api-access-llchj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.809907 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d11c4e59-293e-41b7-9f29-157a813083ad-istio-podinfo" pod "d11c4e59-293e-41b7-9f29-157a813083ad" (UID: "d11c4e59-293e-41b7-9f29-157a813083ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.810511 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-istio-token" pod "d11c4e59-293e-41b7-9f29-157a813083ad" (UID: "d11c4e59-293e-41b7-9f29-157a813083ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.825393 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq"] Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.842840 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-kube-api-access-jl6lm" pod "cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" (UID: "cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1"). InnerVolumeSpecName "kube-api-access-jl6lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.898013 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-maas-api-service-ca\") pod \"10cb04e6-7c18-43e9-ac64-fdcbbac05b78\" (UID: \"10cb04e6-7c18-43e9-ac64-fdcbbac05b78\") " Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.898929 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-maas-api-service-ca" pod "10cb04e6-7c18-43e9-ac64-fdcbbac05b78" (UID: "10cb04e6-7c18-43e9-ac64-fdcbbac05b78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.898352 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-kube-api-access-8mmzn\" (UniqueName: \"kubernetes.io/projected/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-kube-api-access-8mmzn\") pod \"10cb04e6-7c18-43e9-ac64-fdcbbac05b78\" (UID: \"10cb04e6-7c18-43e9-ac64-fdcbbac05b78\") " Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901640 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901668 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901684 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8b3d3c-c341-4dab-9c82-f85d8e203f54-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901698 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hltv\" (UniqueName: \"kubernetes.io/projected/da8b3d3c-c341-4dab-9c82-f85d8e203f54-kube-api-access-5hltv\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901712 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d11c4e59-293e-41b7-9f29-157a813083ad-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901725 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901739 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d11c4e59-293e-41b7-9f29-157a813083ad-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901751 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901765 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llchj\" (UniqueName: \"kubernetes.io/projected/d11c4e59-293e-41b7-9f29-157a813083ad-kube-api-access-llchj\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901778 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901792 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl6lm\" (UniqueName: \"kubernetes.io/projected/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1-kube-api-access-jl6lm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901805 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901818 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.901831 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d11c4e59-293e-41b7-9f29-157a813083ad-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.907462 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-kube-api-access-8mmzn" pod "10cb04e6-7c18-43e9-ac64-fdcbbac05b78" (UID: "10cb04e6-7c18-43e9-ac64-fdcbbac05b78"). InnerVolumeSpecName "kube-api-access-8mmzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:26.951073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:26.909097 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5"] Oct 30 05:30:27.088706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.004507 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mmzn\" (UniqueName: \"kubernetes.io/projected/10cb04e6-7c18-43e9-ac64-fdcbbac05b78-kube-api-access-8mmzn\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:27.088706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.017088 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" Oct 30 05:30:27.143510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.107092 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-kube-api-access-lwgtn\" (UniqueName: \"kubernetes.io/projected/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-kube-api-access-lwgtn\") pod \"9f5876d8-6d01-4a99-950d-8af2c95c3bf8\" (UID: \"9f5876d8-6d01-4a99-950d-8af2c95c3bf8\") " Oct 30 05:30:27.143510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.107330 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-maas-api-service-ca\") pod \"9f5876d8-6d01-4a99-950d-8af2c95c3bf8\" (UID: \"9f5876d8-6d01-4a99-950d-8af2c95c3bf8\") " Oct 30 05:30:27.143510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.108379 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-maas-api-service-ca" pod "9f5876d8-6d01-4a99-950d-8af2c95c3bf8" (UID: "9f5876d8-6d01-4a99-950d-8af2c95c3bf8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:27.143510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.108627 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:27.143510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.121961 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-kube-api-access-lwgtn" pod "9f5876d8-6d01-4a99-950d-8af2c95c3bf8" (UID: "9f5876d8-6d01-4a99-950d-8af2c95c3bf8"). InnerVolumeSpecName "kube-api-access-lwgtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:27.155825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.155795 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" Oct 30 05:30:27.216662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.215854 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aabb8f8-1678-4f27-945e-40e30674980f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aabb8f8-1678-4f27-945e-40e30674980f-maas-api-service-ca\") pod \"5aabb8f8-1678-4f27-945e-40e30674980f\" (UID: \"5aabb8f8-1678-4f27-945e-40e30674980f\") " Oct 30 05:30:27.216662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.216151 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aabb8f8-1678-4f27-945e-40e30674980f-kube-api-access-hbgqp\" (UniqueName: \"kubernetes.io/projected/5aabb8f8-1678-4f27-945e-40e30674980f-kube-api-access-hbgqp\") pod \"5aabb8f8-1678-4f27-945e-40e30674980f\" (UID: \"5aabb8f8-1678-4f27-945e-40e30674980f\") " Oct 30 05:30:27.224144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.217136 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwgtn\" (UniqueName: \"kubernetes.io/projected/9f5876d8-6d01-4a99-950d-8af2c95c3bf8-kube-api-access-lwgtn\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:27.224144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.217999 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aabb8f8-1678-4f27-945e-40e30674980f-maas-api-service-ca" pod "5aabb8f8-1678-4f27-945e-40e30674980f" (UID: "5aabb8f8-1678-4f27-945e-40e30674980f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:27.240922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.227230 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aabb8f8-1678-4f27-945e-40e30674980f-kube-api-access-hbgqp" pod "5aabb8f8-1678-4f27-945e-40e30674980f" (UID: "5aabb8f8-1678-4f27-945e-40e30674980f"). InnerVolumeSpecName "kube-api-access-hbgqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:27.326451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.319642 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aabb8f8-1678-4f27-945e-40e30674980f-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:27.326451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.319675 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbgqp\" (UniqueName: \"kubernetes.io/projected/5aabb8f8-1678-4f27-945e-40e30674980f-kube-api-access-hbgqp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:27.461013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.360952 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" Oct 30 05:30:27.461013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.420423 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e85bdf-2d09-4979-8729-6e47998265c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e85bdf-2d09-4979-8729-6e47998265c9-maas-api-service-ca\") pod \"a8e85bdf-2d09-4979-8729-6e47998265c9\" (UID: \"a8e85bdf-2d09-4979-8729-6e47998265c9\") " Oct 30 05:30:27.461013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.420893 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e85bdf-2d09-4979-8729-6e47998265c9-kube-api-access-vbnxk\" (UniqueName: \"kubernetes.io/projected/a8e85bdf-2d09-4979-8729-6e47998265c9-kube-api-access-vbnxk\") pod \"a8e85bdf-2d09-4979-8729-6e47998265c9\" (UID: \"a8e85bdf-2d09-4979-8729-6e47998265c9\") " Oct 30 05:30:27.461013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.422456 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e85bdf-2d09-4979-8729-6e47998265c9-maas-api-service-ca" pod "a8e85bdf-2d09-4979-8729-6e47998265c9" (UID: "a8e85bdf-2d09-4979-8729-6e47998265c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:27.461013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.429585 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e85bdf-2d09-4979-8729-6e47998265c9-kube-api-access-vbnxk" pod "a8e85bdf-2d09-4979-8729-6e47998265c9" (UID: "a8e85bdf-2d09-4979-8729-6e47998265c9"). InnerVolumeSpecName "kube-api-access-vbnxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:27.602620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.523470 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbnxk\" (UniqueName: \"kubernetes.io/projected/a8e85bdf-2d09-4979-8729-6e47998265c9-kube-api-access-vbnxk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:27.602620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.523495 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e85bdf-2d09-4979-8729-6e47998265c9-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:27.687957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.680603 6587 generic.go:356] "Generic (PLEG): container finished" podID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" containerID="6dfdec7ec20ff383361fcdb322dad39cd6d187e2e8542ff44fb9510c3b0afb7a" exitCode=60 Oct 30 05:30:27.687957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.680671 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" event={"ID":"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff","Type":"ContainerDied","Data":"6dfdec7ec20ff383361fcdb322dad39cd6d187e2e8542ff44fb9510c3b0afb7a"} Oct 30 05:30:27.687957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.680730 6587 scope.go:122] "RemoveContainer" containerID="0b4b2b163281afbd9cdcffa9157b643bfba4d7dd4e3e2cada46bac9be5c013ac" Oct 30 05:30:27.687957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.681217 6587 scope.go:122] "RemoveContainer" containerID="6dfdec7ec20ff383361fcdb322dad39cd6d187e2e8542ff44fb9510c3b0afb7a" Oct 30 05:30:27.687957 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:27.682013 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws_odh-ai-gateway-infra(a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" podUID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.688450 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.688433 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx" event={"ID":"9f5876d8-6d01-4a99-950d-8af2c95c3bf8","Type":"ContainerDied","Data":"3e91fa3c52b55f2558e55d1aee77c4d3a9021015f917ea6331bc4e80af4088f5"} Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.690561 6587 generic.go:356] "Generic (PLEG): container finished" podID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" containerID="c9a21816239d490302de56d8437edb8e8b0d8df8c8dc24c2b58057a3ab54a3b3" exitCode=60 Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.690626 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" event={"ID":"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8","Type":"ContainerDied","Data":"c9a21816239d490302de56d8437edb8e8b0d8df8c8dc24c2b58057a3ab54a3b3"} Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.691046 6587 scope.go:122] "RemoveContainer" containerID="c9a21816239d490302de56d8437edb8e8b0d8df8c8dc24c2b58057a3ab54a3b3" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:27.691420 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw_odh-ai-gateway-infra(4427d4fe-f1ad-4635-94ad-b06b9d2f81c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" podUID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.692561 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.692600 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6" event={"ID":"a8e85bdf-2d09-4979-8729-6e47998265c9","Type":"ContainerDied","Data":"c5c2600a5eb34aeeb9b37f044180427be50ce1cb8eb88ee9ec965d8c5f2ac3ec"} Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.694156 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" event={"ID":"247f6667-54e3-4e2c-98e7-9d5a85641098","Type":"ContainerDied","Data":"05fd9fb83af9343af743fb4933176666137746dfbb122a2e1a70028442ba5175"} Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.694201 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05fd9fb83af9343af743fb4933176666137746dfbb122a2e1a70028442ba5175" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.697093 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" event={"ID":"bdd4fcaf-282e-4541-8da1-872b78f3a564","Type":"ContainerDied","Data":"f7c5a3c2882cb0879fdc0c8f8bc491282fffb2c45f53d7c00f724eedb15a91c5"} Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.697130 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7c5a3c2882cb0879fdc0c8f8bc491282fffb2c45f53d7c00f724eedb15a91c5" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.698660 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.698682 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb" event={"ID":"5aabb8f8-1678-4f27-945e-40e30674980f","Type":"ContainerDied","Data":"14483992a50737b59c4c380d2347d7bc5de071069d157c364d1555c9aa4b4250"} Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.703767 6587 generic.go:356] "Generic (PLEG): container finished" podID="32594502-7640-4be4-86c2-8a21c066f46c" containerID="766dcc391c2ce2edc17c61a5afeedc516d2c8939a41fb7d29e0f4ad8b9d42160" exitCode=60 Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.703880 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" event={"ID":"32594502-7640-4be4-86c2-8a21c066f46c","Type":"ContainerDied","Data":"766dcc391c2ce2edc17c61a5afeedc516d2c8939a41fb7d29e0f4ad8b9d42160"} Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.704338 6587 scope.go:122] "RemoveContainer" containerID="766dcc391c2ce2edc17c61a5afeedc516d2c8939a41fb7d29e0f4ad8b9d42160" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:27.704674 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr_odh-ai-gateway-infra(32594502-7640-4be4-86c2-8a21c066f46c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" podUID="32594502-7640-4be4-86c2-8a21c066f46c" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.707941 6587 generic.go:356] "Generic (PLEG): container finished" podID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" containerID="318b995efb87d9acddc576419f6e89707f92089540da02b76cbb68149222e5aa" exitCode=60 Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.708014 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" event={"ID":"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e","Type":"ContainerDied","Data":"318b995efb87d9acddc576419f6e89707f92089540da02b76cbb68149222e5aa"} Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.708445 6587 scope.go:122] "RemoveContainer" containerID="318b995efb87d9acddc576419f6e89707f92089540da02b76cbb68149222e5aa" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:27.708791 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq_odh-ai-gateway-infra(b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" podUID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.711043 6587 generic.go:356] "Generic (PLEG): container finished" podID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" containerID="2152c486258adf904bbcacb03eb90cbf134b5bae083306e9dcae67f2ba17332b" exitCode=60 Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.711124 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" event={"ID":"f8a5dcff-eb8e-418d-b0ea-26155015b2aa","Type":"ContainerDied","Data":"2152c486258adf904bbcacb03eb90cbf134b5bae083306e9dcae67f2ba17332b"} Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.712096 6587 scope.go:122] "RemoveContainer" containerID="2152c486258adf904bbcacb03eb90cbf134b5bae083306e9dcae67f2ba17332b" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:27.712472 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7_odh-ai-gateway-infra(f8a5dcff-eb8e-418d-b0ea-26155015b2aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" podUID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.713836 6587 generic.go:356] "Generic (PLEG): container finished" podID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" containerID="ee43849457ffd7f6061a3af665203a52f5a9d771729292a7da322476f9486e85" exitCode=60 Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.713918 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" event={"ID":"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2","Type":"ContainerDied","Data":"ee43849457ffd7f6061a3af665203a52f5a9d771729292a7da322476f9486e85"} Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.714217 6587 scope.go:122] "RemoveContainer" containerID="ee43849457ffd7f6061a3af665203a52f5a9d771729292a7da322476f9486e85" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:27.714538 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w_odh-ai-gateway-infra(f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.716375 6587 generic.go:356] "Generic (PLEG): container finished" podID="da52a806-feb6-4661-b0bc-89296f15a88e" containerID="e2af6d2c7385536b4f4028822fc5cbe0fdaad322439852705de9f3bd794780ba" exitCode=60 Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.716489 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" event={"ID":"da52a806-feb6-4661-b0bc-89296f15a88e","Type":"ContainerDied","Data":"e2af6d2c7385536b4f4028822fc5cbe0fdaad322439852705de9f3bd794780ba"} Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.716859 6587 scope.go:122] "RemoveContainer" containerID="e2af6d2c7385536b4f4028822fc5cbe0fdaad322439852705de9f3bd794780ba" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:27.717125 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r_odh-ai-gateway-infra(da52a806-feb6-4661-b0bc-89296f15a88e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" podUID="da52a806-feb6-4661-b0bc-89296f15a88e" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.717867 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.717884 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.717870 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.718058 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.721631 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx"] Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723127 6587 cpu_manager.go:414] "containerMap: removing container" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723146 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723171 6587 cpu_manager.go:414] "containerMap: removing container" podUID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723180 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723212 6587 cpu_manager.go:414] "containerMap: removing container" podUID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723219 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723233 6587 cpu_manager.go:414] "containerMap: removing container" podUID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723241 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723260 6587 cpu_manager.go:414] "containerMap: removing container" podUID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723268 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723278 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723286 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723294 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723299 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723311 6587 cpu_manager.go:414] "containerMap: removing container" podUID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723317 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723334 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5aabb8f8-1678-4f27-945e-40e30674980f" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723372 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aabb8f8-1678-4f27-945e-40e30674980f" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723387 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d0de3126-e112-4a37-aa81-0f312100c7b6" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723396 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0de3126-e112-4a37-aa81-0f312100c7b6" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723407 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5aabb8f8-1678-4f27-945e-40e30674980f" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723416 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aabb8f8-1678-4f27-945e-40e30674980f" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723430 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723435 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723447 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d0de3126-e112-4a37-aa81-0f312100c7b6" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723453 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0de3126-e112-4a37-aa81-0f312100c7b6" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723476 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a8e85bdf-2d09-4979-8729-6e47998265c9" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723483 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e85bdf-2d09-4979-8729-6e47998265c9" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723491 6587 cpu_manager.go:414] "containerMap: removing container" podUID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723496 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723506 6587 cpu_manager.go:414] "containerMap: removing container" podUID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723513 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723531 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d11c4e59-293e-41b7-9f29-157a813083ad" containerName="istio-proxy" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723535 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11c4e59-293e-41b7-9f29-157a813083ad" containerName="istio-proxy" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723558 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a8e85bdf-2d09-4979-8729-6e47998265c9" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723565 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e85bdf-2d09-4979-8729-6e47998265c9" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723575 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5aabb8f8-1678-4f27-945e-40e30674980f" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723580 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aabb8f8-1678-4f27-945e-40e30674980f" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723597 6587 cpu_manager.go:414] "containerMap: removing container" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723604 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723807 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723817 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e85bdf-2d09-4979-8729-6e47998265c9" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723826 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0de3126-e112-4a37-aa81-0f312100c7b6" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723834 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723840 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723847 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723854 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723861 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723868 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723874 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723881 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723888 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aabb8f8-1678-4f27-945e-40e30674980f" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723894 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0de3126-e112-4a37-aa81-0f312100c7b6" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723898 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aabb8f8-1678-4f27-945e-40e30674980f" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723903 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723908 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723915 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0de3126-e112-4a37-aa81-0f312100c7b6" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723922 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723928 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11c4e59-293e-41b7-9f29-157a813083ad" containerName="istio-proxy" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723936 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723944 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e85bdf-2d09-4979-8729-6e47998265c9" containerName="cleanup" Oct 30 05:30:27.742976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.723952 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aabb8f8-1678-4f27-945e-40e30674980f" containerName="cleanup" Oct 30 05:30:27.907185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.769335 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" podStartSLOduration=27.769316804 podStartE2EDuration="27.769316804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:27.756737671 +0000 UTC m=+2752.284504459" watchObservedRunningTime="2026-10-30 05:30:27.769316804 +0000 UTC m=+2752.297083591" Oct 30 05:30:27.907185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.879294 6587 scope.go:122] "RemoveContainer" containerID="6c0bf05e648c36feef5fcf5b94ae170499459f2cc49e82350fea323728cbaa1c" Oct 30 05:30:27.907185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.906907 6587 scope.go:122] "RemoveContainer" containerID="7c31976cc4288ae5f0cdf876edc5b88b414518a4b846789f4dc472a5684f5178" Oct 30 05:30:28.005558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.973841 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx"] Oct 30 05:30:28.005558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.973911 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r"] Oct 30 05:30:28.005558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:27.973960 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.007823 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.033449 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247f6667-54e3-4e2c-98e7-9d5a85641098-kube-api-access-nr997\" (UniqueName: \"kubernetes.io/projected/247f6667-54e3-4e2c-98e7-9d5a85641098-kube-api-access-nr997\") pod \"247f6667-54e3-4e2c-98e7-9d5a85641098\" (UID: \"247f6667-54e3-4e2c-98e7-9d5a85641098\") " Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.033574 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247f6667-54e3-4e2c-98e7-9d5a85641098-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247f6667-54e3-4e2c-98e7-9d5a85641098-maas-api-service-ca\") pod \"247f6667-54e3-4e2c-98e7-9d5a85641098\" (UID: \"247f6667-54e3-4e2c-98e7-9d5a85641098\") " Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.033967 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247f6667-54e3-4e2c-98e7-9d5a85641098-maas-api-service-ca" pod "247f6667-54e3-4e2c-98e7-9d5a85641098" (UID: "247f6667-54e3-4e2c-98e7-9d5a85641098"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.034128 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-certs\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.034225 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-data\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.034273 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmrnx\" (UniqueName: \"kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-kube-api-access-gmrnx\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.034298 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-token\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.034321 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-credential-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.034500 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-podinfo\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.034523 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istiod-ca-cert\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.034720 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.034810 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-envoy\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.034979 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247f6667-54e3-4e2c-98e7-9d5a85641098-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:28.040417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.035746 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247f6667-54e3-4e2c-98e7-9d5a85641098-kube-api-access-nr997" pod "247f6667-54e3-4e2c-98e7-9d5a85641098" (UID: "247f6667-54e3-4e2c-98e7-9d5a85641098"). InnerVolumeSpecName "kube-api-access-nr997". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:28.078599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.041027 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq"] Oct 30 05:30:28.078599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.050483 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0dba31-29888970-x2qkq"] Oct 30 05:30:28.126503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.126462 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7"] Oct 30 05:30:28.136164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.136132 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-certs\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.136340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.136227 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-data\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.136340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.136269 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrnx\" (UniqueName: \"kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-kube-api-access-gmrnx\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.136340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.136311 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-token\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.136550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.136343 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-credential-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.136550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.136475 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-podinfo\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.136550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.136510 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istiod-ca-cert\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.136788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.136766 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-certs\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.136855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.136783 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-data\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.136855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.136829 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.136935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.136849 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-credential-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.136935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.136921 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-envoy\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.137071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.137051 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.137167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.137150 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr997\" (UniqueName: \"kubernetes.io/projected/247f6667-54e3-4e2c-98e7-9d5a85641098-kube-api-access-nr997\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:28.137416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.137390 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istiod-ca-cert\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.139235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.139205 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-envoy\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.139560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.139541 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-podinfo\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.243650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.243600 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrnx\" (UniqueName: \"kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-kube-api-access-gmrnx\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.286104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.246071 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-token\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.286104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.253807 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:28.286104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.253948 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" Oct 30 05:30:28.324252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.314352 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" Oct 30 05:30:28.349923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.341671 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd4fcaf-282e-4541-8da1-872b78f3a564-kube-api-access-6mlmb\" (UniqueName: \"kubernetes.io/projected/bdd4fcaf-282e-4541-8da1-872b78f3a564-kube-api-access-6mlmb\") pod \"bdd4fcaf-282e-4541-8da1-872b78f3a564\" (UID: \"bdd4fcaf-282e-4541-8da1-872b78f3a564\") " Oct 30 05:30:28.349923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.341837 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd4fcaf-282e-4541-8da1-872b78f3a564-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd4fcaf-282e-4541-8da1-872b78f3a564-maas-api-service-ca\") pod \"bdd4fcaf-282e-4541-8da1-872b78f3a564\" (UID: \"bdd4fcaf-282e-4541-8da1-872b78f3a564\") " Oct 30 05:30:28.352398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.352192 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd4fcaf-282e-4541-8da1-872b78f3a564-maas-api-service-ca" pod "bdd4fcaf-282e-4541-8da1-872b78f3a564" (UID: "bdd4fcaf-282e-4541-8da1-872b78f3a564"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:28.357069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.356543 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd4fcaf-282e-4541-8da1-872b78f3a564-kube-api-access-6mlmb" pod "bdd4fcaf-282e-4541-8da1-872b78f3a564" (UID: "bdd4fcaf-282e-4541-8da1-872b78f3a564"). InnerVolumeSpecName "kube-api-access-6mlmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:28.390271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.390188 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g"] Oct 30 05:30:28.407486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.397421 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-a914e9-29888970-p9v4g"] Oct 30 05:30:28.445901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.445859 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd4fcaf-282e-4541-8da1-872b78f3a564-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:28.445901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.445903 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mlmb\" (UniqueName: \"kubernetes.io/projected/bdd4fcaf-282e-4541-8da1-872b78f3a564-kube-api-access-6mlmb\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:28.480644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.480188 6587 scope.go:122] "RemoveContainer" containerID="1f2df7f492356eba8832a50ad89c44c9cbc987db6a68c31f49962af819a80ae0" Oct 30 05:30:28.496201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.496175 6587 scope.go:122] "RemoveContainer" containerID="b714c622fb7804c492bc398cb93c11377eb988786fb2c7c8e61418cdbc1a62e9" Oct 30 05:30:28.508291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.508265 6587 scope.go:122] "RemoveContainer" containerID="f9246019a66748a7aa7fa069caff20210e419a6de517568b087a2bb878d740ee" Oct 30 05:30:28.531043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.530992 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws"] Oct 30 05:30:28.582297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.582259 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq"] Oct 30 05:30:28.626186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.626153 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw"] Oct 30 05:30:28.751713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.751680 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq" Oct 30 05:30:28.836479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.752575 6587 scope.go:122] "RemoveContainer" containerID="ee43849457ffd7f6061a3af665203a52f5a9d771729292a7da322476f9486e85" Oct 30 05:30:28.836479 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:28.753079 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w_odh-ai-gateway-infra(f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" Oct 30 05:30:28.836479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.753486 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5" Oct 30 05:30:28.836479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.767262 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr"] Oct 30 05:30:28.961172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:28.961144 6587 scope.go:122] "RemoveContainer" containerID="c751fb4c2d505558460f1d46bad1ce33f94ae5fea988529db409925cdc32075b" Oct 30 05:30:29.249345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.249310 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2"] Oct 30 05:30:29.256950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.256923 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888970-qckb2"] Oct 30 05:30:29.373877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.344483 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6"] Oct 30 05:30:29.373877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.354523 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d21ab-29888970-b7mz6"] Oct 30 05:30:29.434853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.434811 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq"] Oct 30 05:30:29.446666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.445449 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dfb231-29888970-928jq"] Oct 30 05:30:29.493697 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.488074 6587 scope.go:122] "RemoveContainer" containerID="d98dd3d6e2b05121525d45b8195c09849450fd7eab109f3419dd7b39593529b5" Oct 30 05:30:29.494237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.494198 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h"] Oct 30 05:30:29.547464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.547421 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34722e-29888970-5ww9h"] Oct 30 05:30:29.556489 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.556456 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc"] Oct 30 05:30:29.566562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.566524 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888970-qrbbc"] Oct 30 05:30:29.596057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.596024 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" path="/var/lib/kubelet/pods/10cb04e6-7c18-43e9-ac64-fdcbbac05b78/volumes" Oct 30 05:30:29.736833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.596814 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e85bdf-2d09-4979-8729-6e47998265c9" path="/var/lib/kubelet/pods/a8e85bdf-2d09-4979-8729-6e47998265c9/volumes" Oct 30 05:30:29.736833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.597410 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" path="/var/lib/kubelet/pods/bb155f3e-3d01-431a-84a5-307d1109c9fa/volumes" Oct 30 05:30:29.736833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.598467 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd4fcaf-282e-4541-8da1-872b78f3a564" path="/var/lib/kubelet/pods/bdd4fcaf-282e-4541-8da1-872b78f3a564/volumes" Oct 30 05:30:29.736833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.599064 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" path="/var/lib/kubelet/pods/cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1/volumes" Oct 30 05:30:29.736833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.599655 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0de3126-e112-4a37-aa81-0f312100c7b6" path="/var/lib/kubelet/pods/d0de3126-e112-4a37-aa81-0f312100c7b6/volumes" Oct 30 05:30:29.736833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.600601 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" path="/var/lib/kubelet/pods/da8b3d3c-c341-4dab-9c82-f85d8e203f54/volumes" Oct 30 05:30:29.736833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.717438 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx"] Oct 30 05:30:29.736833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.721863 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-674082-29888970-98pfx"] Oct 30 05:30:29.854738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.772226 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td"] Oct 30 05:30:29.854738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.786033 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td"] Oct 30 05:30:29.904427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.904387 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c"] Oct 30 05:30:29.905524 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905500 6587 cpu_manager.go:414] "containerMap: removing container" podUID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" containerName="cleanup" Oct 30 05:30:29.905524 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905523 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10cb04e6-7c18-43e9-ac64-fdcbbac05b78" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905538 6587 cpu_manager.go:414] "containerMap: removing container" podUID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905543 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8b3d3c-c341-4dab-9c82-f85d8e203f54" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905562 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d0de3126-e112-4a37-aa81-0f312100c7b6" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905568 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0de3126-e112-4a37-aa81-0f312100c7b6" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905582 6587 cpu_manager.go:414] "containerMap: removing container" podUID="bdd4fcaf-282e-4541-8da1-872b78f3a564" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905587 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd4fcaf-282e-4541-8da1-872b78f3a564" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905601 6587 cpu_manager.go:414] "containerMap: removing container" podUID="247f6667-54e3-4e2c-98e7-9d5a85641098" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905606 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247f6667-54e3-4e2c-98e7-9d5a85641098" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905615 6587 cpu_manager.go:414] "containerMap: removing container" podUID="bdd4fcaf-282e-4541-8da1-872b78f3a564" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905622 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd4fcaf-282e-4541-8da1-872b78f3a564" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905634 6587 cpu_manager.go:414] "containerMap: removing container" podUID="bdd4fcaf-282e-4541-8da1-872b78f3a564" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905641 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd4fcaf-282e-4541-8da1-872b78f3a564" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905659 6587 cpu_manager.go:414] "containerMap: removing container" podUID="247f6667-54e3-4e2c-98e7-9d5a85641098" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905667 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247f6667-54e3-4e2c-98e7-9d5a85641098" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905681 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a8e85bdf-2d09-4979-8729-6e47998265c9" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905688 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e85bdf-2d09-4979-8729-6e47998265c9" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905726 6587 cpu_manager.go:414] "containerMap: removing container" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" containerName="cleanup" Oct 30 05:30:29.905813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905732 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" containerName="cleanup" Oct 30 05:30:29.969348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905977 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="247f6667-54e3-4e2c-98e7-9d5a85641098" containerName="cleanup" Oct 30 05:30:29.969348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905993 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd4fcaf-282e-4541-8da1-872b78f3a564" containerName="cleanup" Oct 30 05:30:29.969348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.905999 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb18203a-9a1a-4c6d-9cc2-b5041b3fc6a1" containerName="cleanup" Oct 30 05:30:29.969348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.906006 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd4fcaf-282e-4541-8da1-872b78f3a564" containerName="cleanup" Oct 30 05:30:29.969348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.906014 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd4fcaf-282e-4541-8da1-872b78f3a564" containerName="cleanup" Oct 30 05:30:29.969348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.906021 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="247f6667-54e3-4e2c-98e7-9d5a85641098" containerName="cleanup" Oct 30 05:30:29.969348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.906029 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e85bdf-2d09-4979-8729-6e47998265c9" containerName="cleanup" Oct 30 05:30:29.969348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:29.906039 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb155f3e-3d01-431a-84a5-307d1109c9fa" containerName="cleanup" Oct 30 05:30:30.061532 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.061405 6587 scope.go:122] "RemoveContainer" containerID="9dfe1afa8e3b84c5752b126f61150f3d5948c6fe2f71ac360b37c64d892e385c" Oct 30 05:30:30.100883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.100803 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5"] Oct 30 05:30:30.100883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.100845 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9dc17a-29888970-fr6m5"] Oct 30 05:30:30.100883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.100882 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb"] Oct 30 05:30:30.101127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.100900 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6b3855-29888970-4rxcb"] Oct 30 05:30:30.101127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.100991 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.164025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.160685 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" Oct 30 05:30:30.172427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.171658 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w"] Oct 30 05:30:30.188603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.188574 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c"] Oct 30 05:30:30.209385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.208717 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.209385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.208849 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.209385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.208898 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.209385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.209039 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.209385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.209198 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.209385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.209346 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.209385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.209387 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87npp\" (UniqueName: \"kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-kube-api-access-87npp\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.213337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.209419 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.213337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.209470 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.235913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.229066 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx"] Oct 30 05:30:30.235913 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:30:30.229416 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8590e3a_bbaa_46f0_ae63_a47d18f6b55e.slice/crio-5de641351b1f0913b63b7218838b386bbcade255e4deb89dbc77f2e650b5677b WatchSource:0}: Error finding container 5de641351b1f0913b63b7218838b386bbcade255e4deb89dbc77f2e650b5677b: Status 404 returned error can't find the container with id 5de641351b1f0913b63b7218838b386bbcade255e4deb89dbc77f2e650b5677b Oct 30 05:30:30.235913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.234472 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:30.235913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.234559 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:30.235913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.234602 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.313131 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-kube-api-access-ttdcn\" (UniqueName: \"kubernetes.io/projected/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-kube-api-access-ttdcn\") pod \"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e\" (UID: \"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e\") " Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.313303 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-maas-api-service-ca\") pod \"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e\" (UID: \"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e\") " Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.314616 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.314659 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87npp\" (UniqueName: \"kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-kube-api-access-87npp\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.314696 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.314757 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.314899 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.315067 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.315114 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.315265 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.316003 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.316345 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.317808 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.317877 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.318434 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.318701 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.329580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.328737 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-maas-api-service-ca" pod "b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" (UID: "b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:30.352221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.348178 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" Oct 30 05:30:30.352221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.348760 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-kube-api-access-ttdcn" pod "b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" (UID: "b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e"). InnerVolumeSpecName "kube-api-access-ttdcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:30.352221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.348978 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.352221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.349483 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.364497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.363153 6587 scope.go:122] "RemoveContainer" containerID="7c1a93f168be30e2b5923a428e9a6750b27e09eafaef2334004e508a83513dc5" Oct 30 05:30:30.380289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.368583 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" Oct 30 05:30:30.401181 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.401053 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" Oct 30 05:30:30.404994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.404956 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.407066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.407032 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87npp\" (UniqueName: \"kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-kube-api-access-87npp\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.409012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.408986 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" Oct 30 05:30:30.420797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.418032 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-maas-api-service-ca\") pod \"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8\" (UID: \"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8\") " Oct 30 05:30:30.420797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.418341 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-kube-api-access-8gmhj\" (UniqueName: \"kubernetes.io/projected/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-kube-api-access-8gmhj\") pod \"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8\" (UID: \"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8\") " Oct 30 05:30:30.420797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.420118 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttdcn\" (UniqueName: \"kubernetes.io/projected/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-kube-api-access-ttdcn\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.420797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.420146 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.420797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.420595 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-maas-api-service-ca" pod "4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" (UID: "4427d4fe-f1ad-4635-94ad-b06b9d2f81c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:30.439061 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.439007 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-kube-api-access-8gmhj" pod "4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" (UID: "4427d4fe-f1ad-4635-94ad-b06b9d2f81c8"). InnerVolumeSpecName "kube-api-access-8gmhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:30.490832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.490072 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" Oct 30 05:30:30.521313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.520680 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32594502-7640-4be4-86c2-8a21c066f46c-kube-api-access-kg6wg\" (UniqueName: \"kubernetes.io/projected/32594502-7640-4be4-86c2-8a21c066f46c-kube-api-access-kg6wg\") pod \"32594502-7640-4be4-86c2-8a21c066f46c\" (UID: \"32594502-7640-4be4-86c2-8a21c066f46c\") " Oct 30 05:30:30.521313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.520720 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da52a806-feb6-4661-b0bc-89296f15a88e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da52a806-feb6-4661-b0bc-89296f15a88e-maas-api-service-ca\") pod \"da52a806-feb6-4661-b0bc-89296f15a88e\" (UID: \"da52a806-feb6-4661-b0bc-89296f15a88e\") " Oct 30 05:30:30.521313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.520787 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-kube-api-access-k6sqt\" (UniqueName: \"kubernetes.io/projected/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-kube-api-access-k6sqt\") pod \"f8a5dcff-eb8e-418d-b0ea-26155015b2aa\" (UID: \"f8a5dcff-eb8e-418d-b0ea-26155015b2aa\") " Oct 30 05:30:30.521313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.520834 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da52a806-feb6-4661-b0bc-89296f15a88e-kube-api-access-4n5qm\" (UniqueName: \"kubernetes.io/projected/da52a806-feb6-4661-b0bc-89296f15a88e-kube-api-access-4n5qm\") pod \"da52a806-feb6-4661-b0bc-89296f15a88e\" (UID: \"da52a806-feb6-4661-b0bc-89296f15a88e\") " Oct 30 05:30:30.521313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.520903 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-maas-api-service-ca\") pod \"f8a5dcff-eb8e-418d-b0ea-26155015b2aa\" (UID: \"f8a5dcff-eb8e-418d-b0ea-26155015b2aa\") " Oct 30 05:30:30.521313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.521041 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32594502-7640-4be4-86c2-8a21c066f46c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32594502-7640-4be4-86c2-8a21c066f46c-maas-api-service-ca\") pod \"32594502-7640-4be4-86c2-8a21c066f46c\" (UID: \"32594502-7640-4be4-86c2-8a21c066f46c\") " Oct 30 05:30:30.525121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.521740 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da52a806-feb6-4661-b0bc-89296f15a88e-maas-api-service-ca" pod "da52a806-feb6-4661-b0bc-89296f15a88e" (UID: "da52a806-feb6-4661-b0bc-89296f15a88e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:30.525121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.522059 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.525121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.522083 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gmhj\" (UniqueName: \"kubernetes.io/projected/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8-kube-api-access-8gmhj\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.525121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.522096 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da52a806-feb6-4661-b0bc-89296f15a88e-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.525121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.524815 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-maas-api-service-ca" pod "f8a5dcff-eb8e-418d-b0ea-26155015b2aa" (UID: "f8a5dcff-eb8e-418d-b0ea-26155015b2aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:30.525877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.525851 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32594502-7640-4be4-86c2-8a21c066f46c-maas-api-service-ca" pod "32594502-7640-4be4-86c2-8a21c066f46c" (UID: "32594502-7640-4be4-86c2-8a21c066f46c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:30.530069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.530032 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da52a806-feb6-4661-b0bc-89296f15a88e-kube-api-access-4n5qm" pod "da52a806-feb6-4661-b0bc-89296f15a88e" (UID: "da52a806-feb6-4661-b0bc-89296f15a88e"). InnerVolumeSpecName "kube-api-access-4n5qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:30.607949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.607915 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32594502-7640-4be4-86c2-8a21c066f46c-kube-api-access-kg6wg" pod "32594502-7640-4be4-86c2-8a21c066f46c" (UID: "32594502-7640-4be4-86c2-8a21c066f46c"). InnerVolumeSpecName "kube-api-access-kg6wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:30.625394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.625339 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-kube-api-access-b4z89\" (UniqueName: \"kubernetes.io/projected/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-kube-api-access-b4z89\") pod \"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff\" (UID: \"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff\") " Oct 30 05:30:30.629920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.625719 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-maas-api-service-ca\") pod \"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff\" (UID: \"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff\") " Oct 30 05:30:30.629920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.627295 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg6wg\" (UniqueName: \"kubernetes.io/projected/32594502-7640-4be4-86c2-8a21c066f46c-kube-api-access-kg6wg\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.629920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.627320 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n5qm\" (UniqueName: \"kubernetes.io/projected/da52a806-feb6-4661-b0bc-89296f15a88e-kube-api-access-4n5qm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.629920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.627337 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.629920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.627352 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32594502-7640-4be4-86c2-8a21c066f46c-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.629920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.628781 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-maas-api-service-ca" pod "a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" (UID: "a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:30.637200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.637171 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-kube-api-access-k6sqt" pod "f8a5dcff-eb8e-418d-b0ea-26155015b2aa" (UID: "f8a5dcff-eb8e-418d-b0ea-26155015b2aa"). InnerVolumeSpecName "kube-api-access-k6sqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:30.638239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.638208 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-kube-api-access-b4z89" pod "a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" (UID: "a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff"). InnerVolumeSpecName "kube-api-access-b4z89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:30.649851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.649825 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:30.728503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.728441 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" Oct 30 05:30:30.732219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.732174 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.732452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.732430 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4z89\" (UniqueName: \"kubernetes.io/projected/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff-kube-api-access-b4z89\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.732576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.732562 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6sqt\" (UniqueName: \"kubernetes.io/projected/f8a5dcff-eb8e-418d-b0ea-26155015b2aa-kube-api-access-k6sqt\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.801900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.801809 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" event={"ID":"32594502-7640-4be4-86c2-8a21c066f46c","Type":"ContainerDied","Data":"6c458bb57785c4cf95159a647ba159f31d316715f2ebef4247d5c0afce9867ee"} Oct 30 05:30:30.802668 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.802204 6587 scope.go:122] "RemoveContainer" containerID="766dcc391c2ce2edc17c61a5afeedc516d2c8939a41fb7d29e0f4ad8b9d42160" Oct 30 05:30:30.802668 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.802402 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr" Oct 30 05:30:30.823774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.823736 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" event={"ID":"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e","Type":"ContainerStarted","Data":"2216adce2e21d9d9960b2e0982a6a89b3e6d2f41c0bd1ed8426c5658cf2f078d"} Oct 30 05:30:30.824022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.824000 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" event={"ID":"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e","Type":"ContainerStarted","Data":"5de641351b1f0913b63b7218838b386bbcade255e4deb89dbc77f2e650b5677b"} Oct 30 05:30:30.829991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.828639 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" event={"ID":"b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e","Type":"ContainerDied","Data":"3c0ed43d076b7583c507c3c32160b82b384c95693f4398372f0435e458d56a7c"} Oct 30 05:30:30.829991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.828753 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq" Oct 30 05:30:30.835892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.834971 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-kube-api-access-n4hxd\" (UniqueName: \"kubernetes.io/projected/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-kube-api-access-n4hxd\") pod \"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2\" (UID: \"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2\") " Oct 30 05:30:30.835892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.835133 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-maas-api-service-ca\") pod \"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2\" (UID: \"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2\") " Oct 30 05:30:30.835892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.835856 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-maas-api-service-ca" pod "f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" (UID: "f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:30.836119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.836050 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.858795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.858765 6587 scope.go:122] "RemoveContainer" containerID="318b995efb87d9acddc576419f6e89707f92089540da02b76cbb68149222e5aa" Oct 30 05:30:30.859281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.859102 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" event={"ID":"f8a5dcff-eb8e-418d-b0ea-26155015b2aa","Type":"ContainerDied","Data":"31dd4609dad232629ac8f1be9cfb38bc3ec8a69b31ec73b5d9475818b4e8cbfd"} Oct 30 05:30:30.859281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.859215 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7" Oct 30 05:30:30.862483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.862455 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" event={"ID":"f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2","Type":"ContainerDied","Data":"674b4d3ef6582d9f805db06082d9ea5e3f021458c2b1534ae0839b0a6a98148f"} Oct 30 05:30:30.866292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.866267 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" event={"ID":"da52a806-feb6-4661-b0bc-89296f15a88e","Type":"ContainerDied","Data":"791de5eaddb3f2129edb8a00436c81dc92d72d7bd91a671a8351daeff6806d6f"} Oct 30 05:30:30.867269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.866510 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r" Oct 30 05:30:30.867977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.862641 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w" Oct 30 05:30:30.869324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.869180 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" event={"ID":"a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff","Type":"ContainerDied","Data":"f084a6f4216f8e990fae9bd6d525b18ef4e24a6aeb0d26ef4deb0d843b75c06f"} Oct 30 05:30:30.872492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.869488 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws" Oct 30 05:30:30.876283 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.875771 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" Oct 30 05:30:30.876608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.876550 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw" event={"ID":"4427d4fe-f1ad-4635-94ad-b06b9d2f81c8","Type":"ContainerDied","Data":"78d09e7c0d5db8a5dcb18466827a964048a51b281d07d511960b54cec31f7a9c"} Oct 30 05:30:30.884177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.882151 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-kube-api-access-n4hxd" pod "f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" (UID: "f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2"). InnerVolumeSpecName "kube-api-access-n4hxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:30.897152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.897048 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" podStartSLOduration=6.897032922 podStartE2EDuration="6.897032922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:30.891927994 +0000 UTC m=+2755.419694791" watchObservedRunningTime="2026-10-30 05:30:30.897032922 +0000 UTC m=+2755.424799711" Oct 30 05:30:30.903493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.903465 6587 scope.go:122] "RemoveContainer" containerID="2152c486258adf904bbcacb03eb90cbf134b5bae083306e9dcae67f2ba17332b" Oct 30 05:30:30.939741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.939701 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4hxd\" (UniqueName: \"kubernetes.io/projected/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2-kube-api-access-n4hxd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:30.943054 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.943023 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq"] Oct 30 05:30:30.952615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:30.952583 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-397179-29888970-s5mzq"] Oct 30 05:30:31.018870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.018840 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr"] Oct 30 05:30:31.020629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.019060 6587 scope.go:122] "RemoveContainer" containerID="ee43849457ffd7f6061a3af665203a52f5a9d771729292a7da322476f9486e85" Oct 30 05:30:31.027867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.027842 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-262c10-29888970-l2hnr"] Oct 30 05:30:31.041989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.041874 6587 scope.go:122] "RemoveContainer" containerID="e2af6d2c7385536b4f4028822fc5cbe0fdaad322439852705de9f3bd794780ba" Oct 30 05:30:31.077304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.076920 6587 scope.go:122] "RemoveContainer" containerID="6dfdec7ec20ff383361fcdb322dad39cd6d187e2e8542ff44fb9510c3b0afb7a" Oct 30 05:30:31.127041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.127009 6587 scope.go:122] "RemoveContainer" containerID="c9a21816239d490302de56d8437edb8e8b0d8df8c8dc24c2b58057a3ab54a3b3" Oct 30 05:30:31.176981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.176949 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r"] Oct 30 05:30:31.186025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.185991 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-097d90-29888970-dnc8r"] Oct 30 05:30:31.225580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.219458 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7"] Oct 30 05:30:31.231326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.227163 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e15c99-29888970-m76p7"] Oct 30 05:30:31.246672 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:30:31.240961 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d5a6a4_a651_41fa_8b85_02700b6e12e9.slice/crio-ea21a35e25a1c93ccce10dba1cfcf7cb47d7bcbec0ba79748c9a2a191baa40e0 WatchSource:0}: Error finding container ea21a35e25a1c93ccce10dba1cfcf7cb47d7bcbec0ba79748c9a2a191baa40e0: Status 404 returned error can't find the container with id ea21a35e25a1c93ccce10dba1cfcf7cb47d7bcbec0ba79748c9a2a191baa40e0 Oct 30 05:30:31.246672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.243791 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c"] Oct 30 05:30:31.246672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.245864 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:31.246672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.245944 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:31.246672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.245986 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:31.262291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.257959 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:31.263109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.262652 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" start-of-body= Oct 30 05:30:31.263109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.262700 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" podUID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" Oct 30 05:30:31.269840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.269818 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws"] Oct 30 05:30:31.278255 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.278224 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888970-tkpws"] Oct 30 05:30:31.307841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.307811 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw"] Oct 30 05:30:31.312949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.312924 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-35a94e-29888970-hmbxw"] Oct 30 05:30:31.356799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.356770 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w"] Oct 30 05:30:31.365765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.365738 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-60a531-29888970-gm64w"] Oct 30 05:30:31.599210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.599169 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247f6667-54e3-4e2c-98e7-9d5a85641098" path="/var/lib/kubelet/pods/247f6667-54e3-4e2c-98e7-9d5a85641098/volumes" Oct 30 05:30:31.604609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.604575 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32594502-7640-4be4-86c2-8a21c066f46c" path="/var/lib/kubelet/pods/32594502-7640-4be4-86c2-8a21c066f46c/volumes" Oct 30 05:30:31.605611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.605582 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" path="/var/lib/kubelet/pods/4427d4fe-f1ad-4635-94ad-b06b9d2f81c8/volumes" Oct 30 05:30:31.612797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.606218 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aabb8f8-1678-4f27-945e-40e30674980f" path="/var/lib/kubelet/pods/5aabb8f8-1678-4f27-945e-40e30674980f/volumes" Oct 30 05:30:31.612797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.607252 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f5876d8-6d01-4a99-950d-8af2c95c3bf8" path="/var/lib/kubelet/pods/9f5876d8-6d01-4a99-950d-8af2c95c3bf8/volumes" Oct 30 05:30:31.612797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.607868 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" path="/var/lib/kubelet/pods/a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff/volumes" Oct 30 05:30:31.612797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.608531 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" path="/var/lib/kubelet/pods/b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e/volumes" Oct 30 05:30:31.612797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.609617 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11c4e59-293e-41b7-9f29-157a813083ad" path="/var/lib/kubelet/pods/d11c4e59-293e-41b7-9f29-157a813083ad/volumes" Oct 30 05:30:31.612797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.610195 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da52a806-feb6-4661-b0bc-89296f15a88e" path="/var/lib/kubelet/pods/da52a806-feb6-4661-b0bc-89296f15a88e/volumes" Oct 30 05:30:31.612797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.610907 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" path="/var/lib/kubelet/pods/f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2/volumes" Oct 30 05:30:31.612797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.612026 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" path="/var/lib/kubelet/pods/f8a5dcff-eb8e-418d-b0ea-26155015b2aa/volumes" Oct 30 05:30:31.977693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.977650 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" event={"ID":"25d5a6a4-a651-41fa-8b85-02700b6e12e9","Type":"ContainerStarted","Data":"24d3ca7d646667ab144d96fcc3f191e8cd7b9137917ef92c69627cd30ecde2ac"} Oct 30 05:30:31.978054 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:31.978030 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" event={"ID":"25d5a6a4-a651-41fa-8b85-02700b6e12e9","Type":"ContainerStarted","Data":"ea21a35e25a1c93ccce10dba1cfcf7cb47d7bcbec0ba79748c9a2a191baa40e0"} Oct 30 05:30:32.087998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:32.087930 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" podStartSLOduration=8.087908301 podStartE2EDuration="8.087908301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:32.025962058 +0000 UTC m=+2756.553728859" watchObservedRunningTime="2026-10-30 05:30:32.087908301 +0000 UTC m=+2756.615675088" Oct 30 05:30:32.255432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:32.254202 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" start-of-body= Oct 30 05:30:32.255432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:32.254263 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" podUID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" Oct 30 05:30:32.668379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:32.663028 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:32.668379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:32.663191 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Oct 30 05:30:32.668379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:32.663236 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" podUID="25d5a6a4-a651-41fa-8b85-02700b6e12e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Oct 30 05:30:33.078277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:33.078234 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm"] Oct 30 05:30:33.140580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:33.093837 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf86982c-86zlm"] Oct 30 05:30:33.265381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:33.264765 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" start-of-body= Oct 30 05:30:33.265381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:33.264826 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" podUID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" Oct 30 05:30:33.590189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:33.590157 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18880bdf-d4f4-4510-9f9e-4ca7dbb47077" path="/var/lib/kubelet/pods/18880bdf-d4f4-4510-9f9e-4ca7dbb47077/volumes" Oct 30 05:30:33.683499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:33.679987 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Oct 30 05:30:33.683499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:33.680057 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" podUID="25d5a6a4-a651-41fa-8b85-02700b6e12e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Oct 30 05:30:34.267044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:34.256488 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" start-of-body= Oct 30 05:30:34.267044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:34.256565 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" podUID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" Oct 30 05:30:34.655381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:34.654704 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Oct 30 05:30:34.655381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:34.654778 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" podUID="25d5a6a4-a651-41fa-8b85-02700b6e12e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Oct 30 05:30:35.275384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:35.254770 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" start-of-body= Oct 30 05:30:35.275384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:35.254837 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" podUID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" Oct 30 05:30:35.682524 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:35.680645 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Oct 30 05:30:35.682524 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:35.680712 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" podUID="25d5a6a4-a651-41fa-8b85-02700b6e12e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Oct 30 05:30:36.111104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.111060 6587 generic.go:356] "Generic (PLEG): container finished" podID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerID="22dea87412cbf5bbc4a0e512fbb02cf95405e767b261bdb3c9c07677b580e6e2" exitCode=0 Oct 30 05:30:36.111274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.111131 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" event={"ID":"a9342248-e93d-4bb7-8f9e-1a20117d05be","Type":"ContainerDied","Data":"22dea87412cbf5bbc4a0e512fbb02cf95405e767b261bdb3c9c07677b580e6e2"} Oct 30 05:30:36.151526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.111802 6587 scope.go:122] "RemoveContainer" containerID="22dea87412cbf5bbc4a0e512fbb02cf95405e767b261bdb3c9c07677b580e6e2" Oct 30 05:30:36.151526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.114181 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:36.151526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.114246 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:36.151526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.114280 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:30:36.257105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.257065 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt"] Oct 30 05:30:36.281622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.281585 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" start-of-body= Oct 30 05:30:36.294430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.294386 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" podUID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" Oct 30 05:30:36.305477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.305445 6587 cpu_manager.go:414] "containerMap: removing container" podUID="da52a806-feb6-4661-b0bc-89296f15a88e" containerName="cleanup" Oct 30 05:30:36.464482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.443873 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da52a806-feb6-4661-b0bc-89296f15a88e" containerName="cleanup" Oct 30 05:30:36.464482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.443909 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" containerName="cleanup" Oct 30 05:30:36.464482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.444045 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" containerName="cleanup" Oct 30 05:30:36.464482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.444060 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" containerName="cleanup" Oct 30 05:30:36.464482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.444069 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" containerName="cleanup" Oct 30 05:30:36.464482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.444086 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" containerName="cleanup" Oct 30 05:30:36.464482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.444093 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" containerName="cleanup" Oct 30 05:30:36.464482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.444230 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" containerName="cleanup" Oct 30 05:30:36.464482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.444238 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" containerName="cleanup" Oct 30 05:30:36.464482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.444255 6587 cpu_manager.go:414] "containerMap: removing container" podUID="da52a806-feb6-4661-b0bc-89296f15a88e" containerName="cleanup" Oct 30 05:30:36.464482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.444261 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da52a806-feb6-4661-b0bc-89296f15a88e" containerName="cleanup" Oct 30 05:30:36.464482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.444281 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" containerName="cleanup" Oct 30 05:30:36.464482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.451751 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" containerName="cleanup" Oct 30 05:30:36.588394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.451800 6587 cpu_manager.go:414] "containerMap: removing container" podUID="247f6667-54e3-4e2c-98e7-9d5a85641098" containerName="cleanup" Oct 30 05:30:36.607790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.607747 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247f6667-54e3-4e2c-98e7-9d5a85641098" containerName="cleanup" Oct 30 05:30:36.608159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.608124 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" containerName="cleanup" Oct 30 05:30:36.609741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.609718 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" containerName="cleanup" Oct 30 05:30:36.609934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.609905 6587 cpu_manager.go:414] "containerMap: removing container" podUID="32594502-7640-4be4-86c2-8a21c066f46c" containerName="cleanup" Oct 30 05:30:36.611537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.611516 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32594502-7640-4be4-86c2-8a21c066f46c" containerName="cleanup" Oct 30 05:30:36.611699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.611683 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" containerName="cleanup" Oct 30 05:30:36.611823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.611788 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" containerName="cleanup" Oct 30 05:30:36.611949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.611933 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" containerName="cleanup" Oct 30 05:30:36.612044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.612032 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" containerName="cleanup" Oct 30 05:30:36.612181 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.612167 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" containerName="cleanup" Oct 30 05:30:36.612303 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.612288 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" containerName="cleanup" Oct 30 05:30:36.612446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.612433 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" containerName="cleanup" Oct 30 05:30:36.612606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.612591 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" containerName="cleanup" Oct 30 05:30:36.612733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.612722 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" containerName="cleanup" Oct 30 05:30:36.612825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.612805 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" containerName="cleanup" Oct 30 05:30:36.613006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.612995 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" containerName="cleanup" Oct 30 05:30:36.613098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.613086 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" containerName="cleanup" Oct 30 05:30:36.613204 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.613194 6587 cpu_manager.go:414] "containerMap: removing container" podUID="32594502-7640-4be4-86c2-8a21c066f46c" containerName="cleanup" Oct 30 05:30:36.613288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.613277 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32594502-7640-4be4-86c2-8a21c066f46c" containerName="cleanup" Oct 30 05:30:36.613447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.613435 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" containerName="cleanup" Oct 30 05:30:36.613548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.613538 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" containerName="cleanup" Oct 30 05:30:36.613925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.613893 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.613991 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="da52a806-feb6-4661-b0bc-89296f15a88e" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614008 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614036 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614048 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614059 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614069 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="32594502-7640-4be4-86c2-8a21c066f46c" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614077 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="32594502-7640-4be4-86c2-8a21c066f46c" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614087 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614104 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="da52a806-feb6-4661-b0bc-89296f15a88e" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614114 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614121 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614130 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614144 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614155 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d45e0d-a4c7-4c1a-aa38-72d8cc539fff" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614168 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="247f6667-54e3-4e2c-98e7-9d5a85641098" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614176 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="32594502-7640-4be4-86c2-8a21c066f46c" containerName="cleanup" Oct 30 05:30:36.621468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.614186 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cc16e9-a064-43fc-b11d-a00e5d0b9e6e" containerName="cleanup" Oct 30 05:30:36.701923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.698664 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Oct 30 05:30:36.701923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.698728 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" podUID="25d5a6a4-a651-41fa-8b85-02700b6e12e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Oct 30 05:30:36.706942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.706535 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt"] Oct 30 05:30:36.706942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.706621 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:30:36.707703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.707682 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" Oct 30 05:30:36.744153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.739097 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:30:36.744153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.739690 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:30:36.744153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.739804 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:30:36.744153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.740390 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:30:36.844387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.844169 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt\" (UID: \"c4551433-4795-4b7d-b1f7-84c78ec14d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" Oct 30 05:30:36.844895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.844452 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckdr8\" (UniqueName: \"kubernetes.io/projected/c4551433-4795-4b7d-b1f7-84c78ec14d66-kube-api-access-ckdr8\") pod \"maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt\" (UID: \"c4551433-4795-4b7d-b1f7-84c78ec14d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" Oct 30 05:30:36.844895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.844502 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt\" (UID: \"c4551433-4795-4b7d-b1f7-84c78ec14d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" Oct 30 05:30:36.947028 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.946987 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckdr8\" (UniqueName: \"kubernetes.io/projected/c4551433-4795-4b7d-b1f7-84c78ec14d66-kube-api-access-ckdr8\") pod \"maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt\" (UID: \"c4551433-4795-4b7d-b1f7-84c78ec14d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" Oct 30 05:30:36.947300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.947279 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt\" (UID: \"c4551433-4795-4b7d-b1f7-84c78ec14d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" Oct 30 05:30:36.947785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.947761 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt\" (UID: \"c4551433-4795-4b7d-b1f7-84c78ec14d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" Oct 30 05:30:36.969644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.969609 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt\" (UID: \"c4551433-4795-4b7d-b1f7-84c78ec14d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" Oct 30 05:30:36.990345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.990263 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt\" (UID: \"c4551433-4795-4b7d-b1f7-84c78ec14d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" Oct 30 05:30:36.998341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:36.998302 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckdr8\" (UniqueName: \"kubernetes.io/projected/c4551433-4795-4b7d-b1f7-84c78ec14d66-kube-api-access-ckdr8\") pod \"maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt\" (UID: \"c4551433-4795-4b7d-b1f7-84c78ec14d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" Oct 30 05:30:37.098395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.098311 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" Oct 30 05:30:37.203350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.203312 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" event={"ID":"a9342248-e93d-4bb7-8f9e-1a20117d05be","Type":"ContainerStarted","Data":"9eb9228aa36076f94b3a55f880c0f669fd0653015e8c74a55232fef3f8977eeb"} Oct 30 05:30:37.260441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.256685 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" start-of-body= Oct 30 05:30:37.260441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.256753 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" podUID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.89:15021/healthz/ready\": dial tcp 10.134.1.89:15021: connect: connection refused" Oct 30 05:30:37.608908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.608880 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s"] Oct 30 05:30:37.610851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.610817 6587 cpu_manager.go:414] "containerMap: removing container" podUID="da52a806-feb6-4661-b0bc-89296f15a88e" containerName="cleanup" Oct 30 05:30:37.610851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.610847 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da52a806-feb6-4661-b0bc-89296f15a88e" containerName="cleanup" Oct 30 05:30:37.611652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.611628 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" containerName="cleanup" Oct 30 05:30:37.612276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.611652 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" containerName="cleanup" Oct 30 05:30:37.612276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.611691 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" containerName="cleanup" Oct 30 05:30:37.612276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.611699 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" containerName="cleanup" Oct 30 05:30:37.612276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.611786 6587 cpu_manager.go:414] "containerMap: removing container" podUID="32594502-7640-4be4-86c2-8a21c066f46c" containerName="cleanup" Oct 30 05:30:37.612276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.611796 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32594502-7640-4be4-86c2-8a21c066f46c" containerName="cleanup" Oct 30 05:30:37.612276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.612049 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a5dcff-eb8e-418d-b0ea-26155015b2aa" containerName="cleanup" Oct 30 05:30:37.612276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.612078 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68f9e3a-73a0-43f6-bbac-51d6ab6d26c2" containerName="cleanup" Oct 30 05:30:37.612276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.612096 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="da52a806-feb6-4661-b0bc-89296f15a88e" containerName="cleanup" Oct 30 05:30:37.612276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.612106 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4427d4fe-f1ad-4635-94ad-b06b9d2f81c8" containerName="cleanup" Oct 30 05:30:37.660387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.656118 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Oct 30 05:30:37.660387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.656180 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" podUID="25d5a6a4-a651-41fa-8b85-02700b6e12e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Oct 30 05:30:37.675450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.675414 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" Oct 30 05:30:37.680630 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.680460 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s"] Oct 30 05:30:37.681818 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:30:37.681770 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4551433_4795_4b7d_b1f7_84c78ec14d66.slice/crio-fa1001c593c5ecb6237120550beb4cb1443eb4de746b6f5a7ae0c0008c68d2c2 WatchSource:0}: Error finding container fa1001c593c5ecb6237120550beb4cb1443eb4de746b6f5a7ae0c0008c68d2c2: Status 404 returned error can't find the container with id fa1001c593c5ecb6237120550beb4cb1443eb4de746b6f5a7ae0c0008c68d2c2 Oct 30 05:30:37.684662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.684615 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt"] Oct 30 05:30:37.792812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.792784 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhr8k\" (UniqueName: \"kubernetes.io/projected/e4c1ab68-48af-4566-a2b2-8556053d0a63-kube-api-access-xhr8k\") pod \"maas-api-revoke-sub-d8c3906b-kjb2s\" (UID: \"e4c1ab68-48af-4566-a2b2-8556053d0a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" Oct 30 05:30:37.792942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.792838 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8c3906b-kjb2s\" (UID: \"e4c1ab68-48af-4566-a2b2-8556053d0a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" Oct 30 05:30:37.793060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.793035 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8c3906b-kjb2s\" (UID: \"e4c1ab68-48af-4566-a2b2-8556053d0a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" Oct 30 05:30:37.903835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.894925 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhr8k\" (UniqueName: \"kubernetes.io/projected/e4c1ab68-48af-4566-a2b2-8556053d0a63-kube-api-access-xhr8k\") pod \"maas-api-revoke-sub-d8c3906b-kjb2s\" (UID: \"e4c1ab68-48af-4566-a2b2-8556053d0a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" Oct 30 05:30:37.903835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.895009 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8c3906b-kjb2s\" (UID: \"e4c1ab68-48af-4566-a2b2-8556053d0a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" Oct 30 05:30:37.903835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.895136 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8c3906b-kjb2s\" (UID: \"e4c1ab68-48af-4566-a2b2-8556053d0a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" Oct 30 05:30:37.903835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.896790 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8c3906b-kjb2s\" (UID: \"e4c1ab68-48af-4566-a2b2-8556053d0a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" Oct 30 05:30:37.903835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.903166 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8c3906b-kjb2s\" (UID: \"e4c1ab68-48af-4566-a2b2-8556053d0a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" Oct 30 05:30:37.922043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.922007 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhr8k\" (UniqueName: \"kubernetes.io/projected/e4c1ab68-48af-4566-a2b2-8556053d0a63-kube-api-access-xhr8k\") pod \"maas-api-revoke-sub-d8c3906b-kjb2s\" (UID: \"e4c1ab68-48af-4566-a2b2-8556053d0a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" Oct 30 05:30:37.944156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.944113 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:30:37.945549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.945482 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Oct 30 05:30:37.946053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:37.945987 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Oct 30 05:30:38.096319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:38.095067 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" Oct 30 05:30:38.223529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:38.223448 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" event={"ID":"c4551433-4795-4b7d-b1f7-84c78ec14d66","Type":"ContainerStarted","Data":"fa1001c593c5ecb6237120550beb4cb1443eb4de746b6f5a7ae0c0008c68d2c2"} Oct 30 05:30:38.339491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:38.339456 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:38.341645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:38.341615 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:38.346483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:38.345882 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:30:38.581147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:38.579850 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s"] Oct 30 05:30:38.676294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:38.676011 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:38.686083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:38.678298 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:38.686083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:38.679193 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:30:38.955382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:38.955146 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Oct 30 05:30:38.955382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:38.955207 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Oct 30 05:30:39.264294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:39.264254 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" event={"ID":"e4c1ab68-48af-4566-a2b2-8556053d0a63","Type":"ContainerStarted","Data":"f494c4b4e13712460f954ef5058049e06c489c55e8139c92825cfc35c364dd97"} Oct 30 05:30:39.266423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:39.266391 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" event={"ID":"c4551433-4795-4b7d-b1f7-84c78ec14d66","Type":"ContainerStarted","Data":"9876c91ab14fa065bfcc00a3789af4addf5408532105818b3426fbc05be91949"} Oct 30 05:30:39.344390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:39.335643 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" podStartSLOduration=3.335619789 podStartE2EDuration="3.335619789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:39.312649203 +0000 UTC m=+2763.840415989" watchObservedRunningTime="2026-10-30 05:30:39.335619789 +0000 UTC m=+2763.863386577" Oct 30 05:30:39.946646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:39.946614 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Oct 30 05:30:39.947291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:39.947261 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Oct 30 05:30:40.278605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:40.278516 6587 generic.go:356] "Generic (PLEG): container finished" podID="c4551433-4795-4b7d-b1f7-84c78ec14d66" containerID="9876c91ab14fa065bfcc00a3789af4addf5408532105818b3426fbc05be91949" exitCode=0 Oct 30 05:30:40.279433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:40.278602 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" event={"ID":"c4551433-4795-4b7d-b1f7-84c78ec14d66","Type":"ContainerDied","Data":"9876c91ab14fa065bfcc00a3789af4addf5408532105818b3426fbc05be91949"} Oct 30 05:30:40.280516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:40.280488 6587 generic.go:356] "Generic (PLEG): container finished" podID="e4c1ab68-48af-4566-a2b2-8556053d0a63" containerID="7ba9f382f55354686026f40d50a8c5b5f0b455c0b92fd640fe31634540d3fadd" exitCode=0 Oct 30 05:30:40.280635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:40.280562 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" event={"ID":"e4c1ab68-48af-4566-a2b2-8556053d0a63","Type":"ContainerDied","Data":"7ba9f382f55354686026f40d50a8c5b5f0b455c0b92fd640fe31634540d3fadd"} Oct 30 05:30:40.946380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:40.945654 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Oct 30 05:30:40.946380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:40.945714 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Oct 30 05:30:41.715283 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.715255 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" Oct 30 05:30:41.761382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.759530 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" Oct 30 05:30:41.880620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.879340 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-xks-ca\") pod \"e4c1ab68-48af-4566-a2b2-8556053d0a63\" (UID: \"e4c1ab68-48af-4566-a2b2-8556053d0a63\") " Oct 30 05:30:41.880620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.879403 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-openshift-ca\") pod \"e4c1ab68-48af-4566-a2b2-8556053d0a63\" (UID: \"e4c1ab68-48af-4566-a2b2-8556053d0a63\") " Oct 30 05:30:41.880620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.879453 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-openshift-ca\") pod \"c4551433-4795-4b7d-b1f7-84c78ec14d66\" (UID: \"c4551433-4795-4b7d-b1f7-84c78ec14d66\") " Oct 30 05:30:41.880620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.879573 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c1ab68-48af-4566-a2b2-8556053d0a63-kube-api-access-xhr8k\" (UniqueName: \"kubernetes.io/projected/e4c1ab68-48af-4566-a2b2-8556053d0a63-kube-api-access-xhr8k\") pod \"e4c1ab68-48af-4566-a2b2-8556053d0a63\" (UID: \"e4c1ab68-48af-4566-a2b2-8556053d0a63\") " Oct 30 05:30:41.880620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.879643 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4551433-4795-4b7d-b1f7-84c78ec14d66-kube-api-access-ckdr8\" (UniqueName: \"kubernetes.io/projected/c4551433-4795-4b7d-b1f7-84c78ec14d66-kube-api-access-ckdr8\") pod \"c4551433-4795-4b7d-b1f7-84c78ec14d66\" (UID: \"c4551433-4795-4b7d-b1f7-84c78ec14d66\") " Oct 30 05:30:41.880620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.879700 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-xks-ca\") pod \"c4551433-4795-4b7d-b1f7-84c78ec14d66\" (UID: \"c4551433-4795-4b7d-b1f7-84c78ec14d66\") " Oct 30 05:30:41.882403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.881795 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-openshift-ca" pod "c4551433-4795-4b7d-b1f7-84c78ec14d66" (UID: "c4551433-4795-4b7d-b1f7-84c78ec14d66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:41.882701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.882651 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-openshift-ca" pod "e4c1ab68-48af-4566-a2b2-8556053d0a63" (UID: "e4c1ab68-48af-4566-a2b2-8556053d0a63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:41.886645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.886603 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-xks-ca" pod "c4551433-4795-4b7d-b1f7-84c78ec14d66" (UID: "c4551433-4795-4b7d-b1f7-84c78ec14d66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:30:41.886761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.886734 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-xks-ca" pod "e4c1ab68-48af-4566-a2b2-8556053d0a63" (UID: "e4c1ab68-48af-4566-a2b2-8556053d0a63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:30:41.887133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.887105 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4551433-4795-4b7d-b1f7-84c78ec14d66-kube-api-access-ckdr8" pod "c4551433-4795-4b7d-b1f7-84c78ec14d66" (UID: "c4551433-4795-4b7d-b1f7-84c78ec14d66"). InnerVolumeSpecName "kube-api-access-ckdr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:41.887539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.887514 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c1ab68-48af-4566-a2b2-8556053d0a63-kube-api-access-xhr8k" pod "e4c1ab68-48af-4566-a2b2-8556053d0a63" (UID: "e4c1ab68-48af-4566-a2b2-8556053d0a63"). InnerVolumeSpecName "kube-api-access-xhr8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:41.952460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.946889 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Oct 30 05:30:41.952460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.946950 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Oct 30 05:30:41.983390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.983318 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:41.983390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.983366 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:41.983390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.983381 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c1ab68-48af-4566-a2b2-8556053d0a63-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:41.983390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.983396 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4551433-4795-4b7d-b1f7-84c78ec14d66-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:41.983648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.983408 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhr8k\" (UniqueName: \"kubernetes.io/projected/e4c1ab68-48af-4566-a2b2-8556053d0a63-kube-api-access-xhr8k\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:41.983648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:41.983422 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckdr8\" (UniqueName: \"kubernetes.io/projected/c4551433-4795-4b7d-b1f7-84c78ec14d66-kube-api-access-ckdr8\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:30:42.082025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:42.081994 6587 scope.go:122] "RemoveContainer" containerID="e600cabd6e21e178102acac701ac2f617442d9c0f2fb005cf85455e7f9a5047b" Oct 30 05:30:42.096424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:42.096403 6587 scope.go:122] "RemoveContainer" containerID="450460be9adfb5d1cd8ac66ddefc0e2c49e50e3827cb6375a649bdfd1576a506" Oct 30 05:30:42.109009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:42.108983 6587 scope.go:122] "RemoveContainer" containerID="80974c739aa434b1c7fcefa1218abc9c5b02e559ee6419f006cbb332ad5f9a06" Oct 30 05:30:42.373317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:42.373277 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" event={"ID":"e4c1ab68-48af-4566-a2b2-8556053d0a63","Type":"ContainerDied","Data":"f494c4b4e13712460f954ef5058049e06c489c55e8139c92825cfc35c364dd97"} Oct 30 05:30:42.373974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:42.373339 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f494c4b4e13712460f954ef5058049e06c489c55e8139c92825cfc35c364dd97" Oct 30 05:30:42.373974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:42.373400 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s" Oct 30 05:30:42.387007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:42.386983 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" Oct 30 05:30:42.387192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:42.387065 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt" event={"ID":"c4551433-4795-4b7d-b1f7-84c78ec14d66","Type":"ContainerDied","Data":"fa1001c593c5ecb6237120550beb4cb1443eb4de746b6f5a7ae0c0008c68d2c2"} Oct 30 05:30:42.387192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:42.387116 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa1001c593c5ecb6237120550beb4cb1443eb4de746b6f5a7ae0c0008c68d2c2" Oct 30 05:30:42.949089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:42.949054 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Oct 30 05:30:42.949928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:42.949894 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Oct 30 05:30:44.004674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:44.004635 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:30:44.005291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:44.005131 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:30:44.005956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:44.005934 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:30:46.479826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.479783 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz"] Oct 30 05:30:46.482795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.481829 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c4551433-4795-4b7d-b1f7-84c78ec14d66" containerName="revoke-keys" Oct 30 05:30:46.482795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.481859 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4551433-4795-4b7d-b1f7-84c78ec14d66" containerName="revoke-keys" Oct 30 05:30:46.482795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.481973 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e4c1ab68-48af-4566-a2b2-8556053d0a63" containerName="revoke-keys" Oct 30 05:30:46.482795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.481981 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c1ab68-48af-4566-a2b2-8556053d0a63" containerName="revoke-keys" Oct 30 05:30:46.482795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.482239 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c1ab68-48af-4566-a2b2-8556053d0a63" containerName="revoke-keys" Oct 30 05:30:46.482795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.482261 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4551433-4795-4b7d-b1f7-84c78ec14d66" containerName="revoke-keys" Oct 30 05:30:46.514424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.512168 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz"] Oct 30 05:30:46.514424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.512301 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" Oct 30 05:30:46.545378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.544678 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2d21ab\"" Oct 30 05:30:46.545378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.544760 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2d21ab-dockercfg-wmdt2\"" Oct 30 05:30:46.611832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.611778 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7nbr\" (UniqueName: \"kubernetes.io/projected/370172e2-ad90-418b-8068-83529060cff3-kube-api-access-q7nbr\") pod \"payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz\" (UID: \"370172e2-ad90-418b-8068-83529060cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" Oct 30 05:30:46.612037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.611871 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370172e2-ad90-418b-8068-83529060cff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz\" (UID: \"370172e2-ad90-418b-8068-83529060cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" Oct 30 05:30:46.714505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.714132 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nbr\" (UniqueName: \"kubernetes.io/projected/370172e2-ad90-418b-8068-83529060cff3-kube-api-access-q7nbr\") pod \"payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz\" (UID: \"370172e2-ad90-418b-8068-83529060cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" Oct 30 05:30:46.714505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.714219 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370172e2-ad90-418b-8068-83529060cff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz\" (UID: \"370172e2-ad90-418b-8068-83529060cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" Oct 30 05:30:46.715939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.715828 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370172e2-ad90-418b-8068-83529060cff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz\" (UID: \"370172e2-ad90-418b-8068-83529060cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" Oct 30 05:30:46.741413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.741311 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nbr\" (UniqueName: \"kubernetes.io/projected/370172e2-ad90-418b-8068-83529060cff3-kube-api-access-q7nbr\") pod \"payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz\" (UID: \"370172e2-ad90-418b-8068-83529060cff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" Oct 30 05:30:46.873211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:46.873170 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" Oct 30 05:30:47.258019 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:30:47.257987 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370172e2_ad90_418b_8068_83529060cff3.slice/crio-f0dc335d1947f236fefc8badd488bcdf878e418b24ae960784cc75807af7f3de WatchSource:0}: Error finding container f0dc335d1947f236fefc8badd488bcdf878e418b24ae960784cc75807af7f3de: Status 404 returned error can't find the container with id f0dc335d1947f236fefc8badd488bcdf878e418b24ae960784cc75807af7f3de Oct 30 05:30:47.259523 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:47.259496 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz"] Oct 30 05:30:47.483468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:47.483429 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" event={"ID":"370172e2-ad90-418b-8068-83529060cff3","Type":"ContainerStarted","Data":"3b20d53248fb248a52bfac9de09cb6af938cc4c1a5a0f4753a63c11ac8f36272"} Oct 30 05:30:47.483964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:47.483500 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" event={"ID":"370172e2-ad90-418b-8068-83529060cff3","Type":"ContainerStarted","Data":"f0dc335d1947f236fefc8badd488bcdf878e418b24ae960784cc75807af7f3de"} Oct 30 05:30:47.483964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:47.483543 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" Oct 30 05:30:47.563384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:47.559578 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" podStartSLOduration=1.559557146 podStartE2EDuration="1.559557146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:47.527837101 +0000 UTC m=+2772.055603929" watchObservedRunningTime="2026-10-30 05:30:47.559557146 +0000 UTC m=+2772.087323933" Oct 30 05:30:56.312382 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:30:56.304564 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:30:58.501156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:30:58.501121 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" Oct 30 05:31:03.534165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.534124 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf"] Oct 30 05:31:03.539860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.539832 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" Oct 30 05:31:03.560672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.560641 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf"] Oct 30 05:31:03.566517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.566492 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:31:03.566823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.566805 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:31:03.583085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.583060 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:31:03.585938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.585914 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:31:03.614266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.614227 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9wqt\" (UniqueName: \"kubernetes.io/projected/ca5b72fa-94e2-47ea-805b-13192f14c1a3-kube-api-access-v9wqt\") pod \"maas-api-revoke-sub-68396a5e-759vf\" (UID: \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" Oct 30 05:31:03.614521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.614505 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68396a5e-759vf\" (UID: \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" Oct 30 05:31:03.614580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.614544 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68396a5e-759vf\" (UID: \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" Oct 30 05:31:03.715848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.715810 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68396a5e-759vf\" (UID: \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" Oct 30 05:31:03.716071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.715865 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68396a5e-759vf\" (UID: \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" Oct 30 05:31:03.716278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.716249 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9wqt\" (UniqueName: \"kubernetes.io/projected/ca5b72fa-94e2-47ea-805b-13192f14c1a3-kube-api-access-v9wqt\") pod \"maas-api-revoke-sub-68396a5e-759vf\" (UID: \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" Oct 30 05:31:03.716584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.716559 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68396a5e-759vf\" (UID: \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" Oct 30 05:31:03.720477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.720446 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68396a5e-759vf\" (UID: \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" Oct 30 05:31:03.738856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.738826 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9wqt\" (UniqueName: \"kubernetes.io/projected/ca5b72fa-94e2-47ea-805b-13192f14c1a3-kube-api-access-v9wqt\") pod \"maas-api-revoke-sub-68396a5e-759vf\" (UID: \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" Oct 30 05:31:03.856623 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:03.856592 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" Oct 30 05:31:04.246946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:04.246913 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf"] Oct 30 05:31:04.247477 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:31:04.247446 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5b72fa_94e2_47ea_805b_13192f14c1a3.slice/crio-7a0d034ec2f90b46aa07ee267a6d23e177f841abadf1624fb277630c8a8b825f WatchSource:0}: Error finding container 7a0d034ec2f90b46aa07ee267a6d23e177f841abadf1624fb277630c8a8b825f: Status 404 returned error can't find the container with id 7a0d034ec2f90b46aa07ee267a6d23e177f841abadf1624fb277630c8a8b825f Oct 30 05:31:04.872521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:04.872482 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" event={"ID":"ca5b72fa-94e2-47ea-805b-13192f14c1a3","Type":"ContainerStarted","Data":"768c21873b45826251da87c41e04650ce921d84ec756b14a89ca92a7144bade2"} Oct 30 05:31:04.872998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:04.872551 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" event={"ID":"ca5b72fa-94e2-47ea-805b-13192f14c1a3","Type":"ContainerStarted","Data":"7a0d034ec2f90b46aa07ee267a6d23e177f841abadf1624fb277630c8a8b825f"} Oct 30 05:31:04.919980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:04.919880 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" podStartSLOduration=1.919863879 podStartE2EDuration="1.919863879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:04.915852993 +0000 UTC m=+2789.443619779" watchObservedRunningTime="2026-10-30 05:31:04.919863879 +0000 UTC m=+2789.447630665" Oct 30 05:31:05.944109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:05.944062 6587 generic.go:356] "Generic (PLEG): container finished" podID="ca5b72fa-94e2-47ea-805b-13192f14c1a3" containerID="768c21873b45826251da87c41e04650ce921d84ec756b14a89ca92a7144bade2" exitCode=0 Oct 30 05:31:05.944673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:05.944141 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" event={"ID":"ca5b72fa-94e2-47ea-805b-13192f14c1a3","Type":"ContainerDied","Data":"768c21873b45826251da87c41e04650ce921d84ec756b14a89ca92a7144bade2"} Oct 30 05:31:07.225679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:07.225618 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" Oct 30 05:31:07.319970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:07.319930 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-xks-ca\") pod \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\" (UID: \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\") " Oct 30 05:31:07.320175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:07.320113 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5b72fa-94e2-47ea-805b-13192f14c1a3-kube-api-access-v9wqt\" (UniqueName: \"kubernetes.io/projected/ca5b72fa-94e2-47ea-805b-13192f14c1a3-kube-api-access-v9wqt\") pod \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\" (UID: \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\") " Oct 30 05:31:07.320401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:07.320371 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-openshift-ca\") pod \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\" (UID: \"ca5b72fa-94e2-47ea-805b-13192f14c1a3\") " Oct 30 05:31:07.321651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:07.321623 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-openshift-ca" pod "ca5b72fa-94e2-47ea-805b-13192f14c1a3" (UID: "ca5b72fa-94e2-47ea-805b-13192f14c1a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:07.328981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:07.328954 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-xks-ca" pod "ca5b72fa-94e2-47ea-805b-13192f14c1a3" (UID: "ca5b72fa-94e2-47ea-805b-13192f14c1a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:07.329128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:07.329009 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5b72fa-94e2-47ea-805b-13192f14c1a3-kube-api-access-v9wqt" pod "ca5b72fa-94e2-47ea-805b-13192f14c1a3" (UID: "ca5b72fa-94e2-47ea-805b-13192f14c1a3"). InnerVolumeSpecName "kube-api-access-v9wqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:07.423424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:07.423347 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:07.423424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:07.423428 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5b72fa-94e2-47ea-805b-13192f14c1a3-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:07.423424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:07.423445 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9wqt\" (UniqueName: \"kubernetes.io/projected/ca5b72fa-94e2-47ea-805b-13192f14c1a3-kube-api-access-v9wqt\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:07.970066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:07.969988 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" Oct 30 05:31:07.970066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:07.970016 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf" event={"ID":"ca5b72fa-94e2-47ea-805b-13192f14c1a3","Type":"ContainerDied","Data":"7a0d034ec2f90b46aa07ee267a6d23e177f841abadf1624fb277630c8a8b825f"} Oct 30 05:31:07.971559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:07.970069 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a0d034ec2f90b46aa07ee267a6d23e177f841abadf1624fb277630c8a8b825f" Oct 30 05:31:13.853452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:13.853410 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp"] Oct 30 05:31:13.854505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:13.854468 6587 cpu_manager.go:414] "containerMap: removing container" podUID="ca5b72fa-94e2-47ea-805b-13192f14c1a3" containerName="revoke-keys" Oct 30 05:31:13.854505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:13.854501 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5b72fa-94e2-47ea-805b-13192f14c1a3" containerName="revoke-keys" Oct 30 05:31:13.855854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:13.854720 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5b72fa-94e2-47ea-805b-13192f14c1a3" containerName="revoke-keys" Oct 30 05:31:13.858923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:13.858905 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" Oct 30 05:31:13.880348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:13.880315 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp"] Oct 30 05:31:13.887319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:13.887280 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:31:13.887319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:13.887282 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:31:13.888219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:13.887324 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:31:13.889939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:13.889916 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:31:13.972227 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:13.972195 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26545910-vvpxp\" (UID: \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" Oct 30 05:31:13.972444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:13.972262 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26545910-vvpxp\" (UID: \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" Oct 30 05:31:13.972444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:13.972401 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8w9x\" (UniqueName: \"kubernetes.io/projected/d829ce66-6d19-46de-8029-2ebf1f3fbe09-kube-api-access-m8w9x\") pod \"maas-api-revoke-sub-26545910-vvpxp\" (UID: \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" Oct 30 05:31:14.074999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:14.074965 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26545910-vvpxp\" (UID: \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" Oct 30 05:31:14.076986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:14.075055 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26545910-vvpxp\" (UID: \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" Oct 30 05:31:14.076986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:14.075096 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8w9x\" (UniqueName: \"kubernetes.io/projected/d829ce66-6d19-46de-8029-2ebf1f3fbe09-kube-api-access-m8w9x\") pod \"maas-api-revoke-sub-26545910-vvpxp\" (UID: \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" Oct 30 05:31:14.076986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:14.076046 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26545910-vvpxp\" (UID: \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" Oct 30 05:31:14.079645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:14.079623 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26545910-vvpxp\" (UID: \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" Oct 30 05:31:14.096640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:14.096618 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8w9x\" (UniqueName: \"kubernetes.io/projected/d829ce66-6d19-46de-8029-2ebf1f3fbe09-kube-api-access-m8w9x\") pod \"maas-api-revoke-sub-26545910-vvpxp\" (UID: \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" Oct 30 05:31:14.179249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:14.178810 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" Oct 30 05:31:14.556316 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:31:14.556276 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd829ce66_6d19_46de_8029_2ebf1f3fbe09.slice/crio-9b8d04da92c84aab850c48718747b3ce2ed0b03d29e187d14bc1e724ee14c601 WatchSource:0}: Error finding container 9b8d04da92c84aab850c48718747b3ce2ed0b03d29e187d14bc1e724ee14c601: Status 404 returned error can't find the container with id 9b8d04da92c84aab850c48718747b3ce2ed0b03d29e187d14bc1e724ee14c601 Oct 30 05:31:14.557536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:14.557288 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp"] Oct 30 05:31:15.050809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:15.050776 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" event={"ID":"d829ce66-6d19-46de-8029-2ebf1f3fbe09","Type":"ContainerStarted","Data":"5e9d2f5630bf2f7d42cc61ff547996080f54c5878720b1792188193b044b39d8"} Oct 30 05:31:15.051159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:15.050833 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" event={"ID":"d829ce66-6d19-46de-8029-2ebf1f3fbe09","Type":"ContainerStarted","Data":"9b8d04da92c84aab850c48718747b3ce2ed0b03d29e187d14bc1e724ee14c601"} Oct 30 05:31:15.101225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:15.101135 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" podStartSLOduration=2.101120741 podStartE2EDuration="2.101120741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:15.096268871 +0000 UTC m=+2799.624035657" watchObservedRunningTime="2026-10-30 05:31:15.101120741 +0000 UTC m=+2799.628887552" Oct 30 05:31:16.081069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:16.081030 6587 generic.go:356] "Generic (PLEG): container finished" podID="d829ce66-6d19-46de-8029-2ebf1f3fbe09" containerID="5e9d2f5630bf2f7d42cc61ff547996080f54c5878720b1792188193b044b39d8" exitCode=0 Oct 30 05:31:16.082619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:16.081126 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" event={"ID":"d829ce66-6d19-46de-8029-2ebf1f3fbe09","Type":"ContainerDied","Data":"5e9d2f5630bf2f7d42cc61ff547996080f54c5878720b1792188193b044b39d8"} Oct 30 05:31:17.318627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:17.318602 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" Oct 30 05:31:17.344415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:17.344386 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d829ce66-6d19-46de-8029-2ebf1f3fbe09-kube-api-access-m8w9x\" (UniqueName: \"kubernetes.io/projected/d829ce66-6d19-46de-8029-2ebf1f3fbe09-kube-api-access-m8w9x\") pod \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\" (UID: \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\") " Oct 30 05:31:17.344583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:17.344531 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-xks-ca\") pod \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\" (UID: \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\") " Oct 30 05:31:17.344681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:17.344663 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-openshift-ca\") pod \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\" (UID: \"d829ce66-6d19-46de-8029-2ebf1f3fbe09\") " Oct 30 05:31:17.345082 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:17.345051 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-openshift-ca" pod "d829ce66-6d19-46de-8029-2ebf1f3fbe09" (UID: "d829ce66-6d19-46de-8029-2ebf1f3fbe09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:17.345760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:17.345737 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:17.347290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:17.347223 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-xks-ca" pod "d829ce66-6d19-46de-8029-2ebf1f3fbe09" (UID: "d829ce66-6d19-46de-8029-2ebf1f3fbe09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:17.347582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:17.347556 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d829ce66-6d19-46de-8029-2ebf1f3fbe09-kube-api-access-m8w9x" pod "d829ce66-6d19-46de-8029-2ebf1f3fbe09" (UID: "d829ce66-6d19-46de-8029-2ebf1f3fbe09"). InnerVolumeSpecName "kube-api-access-m8w9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:17.454410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:17.452150 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d829ce66-6d19-46de-8029-2ebf1f3fbe09-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:17.454410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:17.452189 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8w9x\" (UniqueName: \"kubernetes.io/projected/d829ce66-6d19-46de-8029-2ebf1f3fbe09-kube-api-access-m8w9x\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:18.103767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:18.103737 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" Oct 30 05:31:18.104016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:18.103768 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp" event={"ID":"d829ce66-6d19-46de-8029-2ebf1f3fbe09","Type":"ContainerDied","Data":"9b8d04da92c84aab850c48718747b3ce2ed0b03d29e187d14bc1e724ee14c601"} Oct 30 05:31:18.104016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:18.103815 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b8d04da92c84aab850c48718747b3ce2ed0b03d29e187d14bc1e724ee14c601" Oct 30 05:31:22.038098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:22.038056 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt"] Oct 30 05:31:22.050887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:22.050849 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b270a6-2n9bt"] Oct 30 05:31:23.588975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:23.588942 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025c2255-76a8-49c2-8956-3d81e7a4e725" path="/var/lib/kubelet/pods/025c2255-76a8-49c2-8956-3d81e7a4e725/volumes" Oct 30 05:31:24.974214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:24.974176 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59"] Oct 30 05:31:24.977834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:24.977802 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d829ce66-6d19-46de-8029-2ebf1f3fbe09" containerName="revoke-keys" Oct 30 05:31:24.983470 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:24.983444 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d829ce66-6d19-46de-8029-2ebf1f3fbe09" containerName="revoke-keys" Oct 30 05:31:24.984191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:24.984165 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d829ce66-6d19-46de-8029-2ebf1f3fbe09" containerName="revoke-keys" Oct 30 05:31:25.006778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.006742 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59"] Oct 30 05:31:25.217558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.217520 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" Oct 30 05:31:25.243511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.238573 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:31:25.329578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.329536 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:31:25.329986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.329965 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:31:25.346522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.330242 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:31:25.393925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.366664 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqv54\" (UniqueName: \"kubernetes.io/projected/4786587b-cb5c-472b-81af-8a0a9eeb1d70-kube-api-access-dqv54\") pod \"maas-api-revoke-sub-79d87fd3-xpw59\" (UID: \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" Oct 30 05:31:25.393925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.366851 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d87fd3-xpw59\" (UID: \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" Oct 30 05:31:25.393925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.366902 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d87fd3-xpw59\" (UID: \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" Oct 30 05:31:25.479640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.479594 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d87fd3-xpw59\" (UID: \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" Oct 30 05:31:25.481625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.481595 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d87fd3-xpw59\" (UID: \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" Oct 30 05:31:25.482934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.482907 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqv54\" (UniqueName: \"kubernetes.io/projected/4786587b-cb5c-472b-81af-8a0a9eeb1d70-kube-api-access-dqv54\") pod \"maas-api-revoke-sub-79d87fd3-xpw59\" (UID: \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" Oct 30 05:31:25.495995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.482579 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d87fd3-xpw59\" (UID: \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" Oct 30 05:31:25.544070 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.544037 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d87fd3-xpw59\" (UID: \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" Oct 30 05:31:25.698392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.683159 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqv54\" (UniqueName: \"kubernetes.io/projected/4786587b-cb5c-472b-81af-8a0a9eeb1d70-kube-api-access-dqv54\") pod \"maas-api-revoke-sub-79d87fd3-xpw59\" (UID: \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" Oct 30 05:31:25.809447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:25.787080 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" Oct 30 05:31:26.361700 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:31:26.361661 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:31:26.990172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:26.990127 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:31:27.139546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:27.137584 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59"] Oct 30 05:31:27.215811 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:31:27.169772 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4786587b_cb5c_472b_81af_8a0a9eeb1d70.slice/crio-99f2e70bad5da861f61636cf1a9e161e371c7c63fe45ed6a9d38d6d5fd8248eb WatchSource:0}: Error finding container 99f2e70bad5da861f61636cf1a9e161e371c7c63fe45ed6a9d38d6d5fd8248eb: Status 404 returned error can't find the container with id 99f2e70bad5da861f61636cf1a9e161e371c7c63fe45ed6a9d38d6d5fd8248eb Oct 30 05:31:27.811751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:27.811706 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" event={"ID":"4786587b-cb5c-472b-81af-8a0a9eeb1d70","Type":"ContainerStarted","Data":"99f2e70bad5da861f61636cf1a9e161e371c7c63fe45ed6a9d38d6d5fd8248eb"} Oct 30 05:31:28.825689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:28.825220 6587 generic.go:356] "Generic (PLEG): container finished" podID="4786587b-cb5c-472b-81af-8a0a9eeb1d70" containerID="dbbaa1c95ec23aad52ed5bd74180cb91c111774b978f1fa7fb83fea1ee69691e" exitCode=0 Oct 30 05:31:28.825689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:28.825297 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" event={"ID":"4786587b-cb5c-472b-81af-8a0a9eeb1d70","Type":"ContainerDied","Data":"dbbaa1c95ec23aad52ed5bd74180cb91c111774b978f1fa7fb83fea1ee69691e"} Oct 30 05:31:30.083175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.083141 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" Oct 30 05:31:30.206975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.206879 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4786587b-cb5c-472b-81af-8a0a9eeb1d70-kube-api-access-dqv54\" (UniqueName: \"kubernetes.io/projected/4786587b-cb5c-472b-81af-8a0a9eeb1d70-kube-api-access-dqv54\") pod \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\" (UID: \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\") " Oct 30 05:31:30.207166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.206994 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-openshift-ca\") pod \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\" (UID: \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\") " Oct 30 05:31:30.207166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.207068 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-xks-ca\") pod \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\" (UID: \"4786587b-cb5c-472b-81af-8a0a9eeb1d70\") " Oct 30 05:31:30.207519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.207481 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-openshift-ca" pod "4786587b-cb5c-472b-81af-8a0a9eeb1d70" (UID: "4786587b-cb5c-472b-81af-8a0a9eeb1d70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:30.208350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.208050 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:30.213155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.213126 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-xks-ca" pod "4786587b-cb5c-472b-81af-8a0a9eeb1d70" (UID: "4786587b-cb5c-472b-81af-8a0a9eeb1d70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:30.213260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.213186 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4786587b-cb5c-472b-81af-8a0a9eeb1d70-kube-api-access-dqv54" pod "4786587b-cb5c-472b-81af-8a0a9eeb1d70" (UID: "4786587b-cb5c-472b-81af-8a0a9eeb1d70"). InnerVolumeSpecName "kube-api-access-dqv54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:30.310498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.310459 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqv54\" (UniqueName: \"kubernetes.io/projected/4786587b-cb5c-472b-81af-8a0a9eeb1d70-kube-api-access-dqv54\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:30.310498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.310497 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4786587b-cb5c-472b-81af-8a0a9eeb1d70-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:30.626665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.626624 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds"] Oct 30 05:31:30.632379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.632330 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" podUID="66c8e0c3-624f-44f9-b797-b7837a2deb53" containerName="payload-processing" containerID="cri-o://3f515f4285dbcf1db1d53b8430d28d0549c0d4ab0169a5dd9d5bc1811b31f957" gracePeriod=30 Oct 30 05:31:30.876834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.876738 6587 generic.go:356] "Generic (PLEG): container finished" podID="66c8e0c3-624f-44f9-b797-b7837a2deb53" containerID="3f515f4285dbcf1db1d53b8430d28d0549c0d4ab0169a5dd9d5bc1811b31f957" exitCode=0 Oct 30 05:31:30.878129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.878014 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" event={"ID":"66c8e0c3-624f-44f9-b797-b7837a2deb53","Type":"ContainerDied","Data":"3f515f4285dbcf1db1d53b8430d28d0549c0d4ab0169a5dd9d5bc1811b31f957"} Oct 30 05:31:30.883740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.882165 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" event={"ID":"4786587b-cb5c-472b-81af-8a0a9eeb1d70","Type":"ContainerDied","Data":"99f2e70bad5da861f61636cf1a9e161e371c7c63fe45ed6a9d38d6d5fd8248eb"} Oct 30 05:31:30.884334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.883895 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99f2e70bad5da861f61636cf1a9e161e371c7c63fe45ed6a9d38d6d5fd8248eb" Oct 30 05:31:30.884334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.883988 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59" Oct 30 05:31:30.964459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:30.964432 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" Oct 30 05:31:31.034853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.034820 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq"] Oct 30 05:31:31.035812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.035793 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4786587b-cb5c-472b-81af-8a0a9eeb1d70" containerName="revoke-keys" Oct 30 05:31:31.035873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.035814 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4786587b-cb5c-472b-81af-8a0a9eeb1d70" containerName="revoke-keys" Oct 30 05:31:31.036664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.035877 6587 cpu_manager.go:414] "containerMap: removing container" podUID="66c8e0c3-624f-44f9-b797-b7837a2deb53" containerName="payload-processing" Oct 30 05:31:31.036664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.035883 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c8e0c3-624f-44f9-b797-b7837a2deb53" containerName="payload-processing" Oct 30 05:31:31.036664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.036015 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c8e0c3-624f-44f9-b797-b7837a2deb53" containerName="payload-processing" Oct 30 05:31:31.036664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.036030 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4786587b-cb5c-472b-81af-8a0a9eeb1d70" containerName="revoke-keys" Oct 30 05:31:31.052225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.042758 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" Oct 30 05:31:31.052225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.043645 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66c8e0c3-624f-44f9-b797-b7837a2deb53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c8e0c3-624f-44f9-b797-b7837a2deb53-plugins-config-volume\") pod \"66c8e0c3-624f-44f9-b797-b7837a2deb53\" (UID: \"66c8e0c3-624f-44f9-b797-b7837a2deb53\") " Oct 30 05:31:31.052225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.043743 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c8e0c3-624f-44f9-b797-b7837a2deb53-kube-api-access-628vf\" (UniqueName: \"kubernetes.io/projected/66c8e0c3-624f-44f9-b797-b7837a2deb53-kube-api-access-628vf\") pod \"66c8e0c3-624f-44f9-b797-b7837a2deb53\" (UID: \"66c8e0c3-624f-44f9-b797-b7837a2deb53\") " Oct 30 05:31:31.052225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.044918 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c8e0c3-624f-44f9-b797-b7837a2deb53-plugins-config-volume" pod "66c8e0c3-624f-44f9-b797-b7837a2deb53" (UID: "66c8e0c3-624f-44f9-b797-b7837a2deb53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:31.052225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.046974 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c8e0c3-624f-44f9-b797-b7837a2deb53-kube-api-access-628vf" pod "66c8e0c3-624f-44f9-b797-b7837a2deb53" (UID: "66c8e0c3-624f-44f9-b797-b7837a2deb53"). InnerVolumeSpecName "kube-api-access-628vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:31.076727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.076690 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq"] Oct 30 05:31:31.143493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.125534 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv"] Oct 30 05:31:31.143493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.125741 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" podUID="9396c9bf-5ba4-4baf-accb-f63fd7f0d69a" containerName="payload-processing" containerID="cri-o://0f7a40811e85d5a066e68586bd083fa443ea710d08a1529cde50ff955365ac66" gracePeriod=30 Oct 30 05:31:31.149273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.147580 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4wkt\" (UniqueName: \"kubernetes.io/projected/34622d6f-d009-43d4-affa-f16d4b51e5e6-kube-api-access-f4wkt\") pod \"payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq\" (UID: \"34622d6f-d009-43d4-affa-f16d4b51e5e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" Oct 30 05:31:31.149273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.147646 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34622d6f-d009-43d4-affa-f16d4b51e5e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq\" (UID: \"34622d6f-d009-43d4-affa-f16d4b51e5e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" Oct 30 05:31:31.149273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.148139 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c8e0c3-624f-44f9-b797-b7837a2deb53-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:31.149273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.148156 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-628vf\" (UniqueName: \"kubernetes.io/projected/66c8e0c3-624f-44f9-b797-b7837a2deb53-kube-api-access-628vf\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:31.252398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.252346 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4wkt\" (UniqueName: \"kubernetes.io/projected/34622d6f-d009-43d4-affa-f16d4b51e5e6-kube-api-access-f4wkt\") pod \"payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq\" (UID: \"34622d6f-d009-43d4-affa-f16d4b51e5e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" Oct 30 05:31:31.252517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.252444 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34622d6f-d009-43d4-affa-f16d4b51e5e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq\" (UID: \"34622d6f-d009-43d4-affa-f16d4b51e5e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" Oct 30 05:31:31.254324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.253197 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34622d6f-d009-43d4-affa-f16d4b51e5e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq\" (UID: \"34622d6f-d009-43d4-affa-f16d4b51e5e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" Oct 30 05:31:31.289595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.289528 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4wkt\" (UniqueName: \"kubernetes.io/projected/34622d6f-d009-43d4-affa-f16d4b51e5e6-kube-api-access-f4wkt\") pod \"payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq\" (UID: \"34622d6f-d009-43d4-affa-f16d4b51e5e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" Oct 30 05:31:31.342522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.342162 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v"] Oct 30 05:31:31.349845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.349811 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3855-cbf8v"] Oct 30 05:31:31.425822 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.425725 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" Oct 30 05:31:31.485628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.485600 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" Oct 30 05:31:31.565382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.559106 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-plugins-config-volume\") pod \"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a\" (UID: \"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a\") " Oct 30 05:31:31.565382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.559201 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-kube-api-access-6xqqc\" (UniqueName: \"kubernetes.io/projected/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-kube-api-access-6xqqc\") pod \"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a\" (UID: \"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a\") " Oct 30 05:31:31.565382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.564136 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-plugins-config-volume" pod "9396c9bf-5ba4-4baf-accb-f63fd7f0d69a" (UID: "9396c9bf-5ba4-4baf-accb-f63fd7f0d69a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:31.573379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.571548 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-kube-api-access-6xqqc" pod "9396c9bf-5ba4-4baf-accb-f63fd7f0d69a" (UID: "9396c9bf-5ba4-4baf-accb-f63fd7f0d69a"). InnerVolumeSpecName "kube-api-access-6xqqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:31.886158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.886104 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:31.909307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.896959 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xqqc\" (UniqueName: \"kubernetes.io/projected/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a-kube-api-access-6xqqc\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:31.910279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.910229 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5c804d-fffb-4fcc-8686-080b8f61703c" path="/var/lib/kubelet/pods/fc5c804d-fffb-4fcc-8686-080b8f61703c/volumes" Oct 30 05:31:31.953725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.953684 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz"] Oct 30 05:31:31.956013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.955982 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9396c9bf-5ba4-4baf-accb-f63fd7f0d69a" containerName="payload-processing" Oct 30 05:31:31.956138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.956015 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9396c9bf-5ba4-4baf-accb-f63fd7f0d69a" containerName="payload-processing" Oct 30 05:31:31.956649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.956625 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9396c9bf-5ba4-4baf-accb-f63fd7f0d69a" containerName="payload-processing" Oct 30 05:31:31.970638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.970611 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz"] Oct 30 05:31:31.970846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.970826 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:31.998936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.998908 6587 generic.go:356] "Generic (PLEG): container finished" podID="9396c9bf-5ba4-4baf-accb-f63fd7f0d69a" containerID="0f7a40811e85d5a066e68586bd083fa443ea710d08a1529cde50ff955365ac66" exitCode=0 Oct 30 05:31:32.000670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.999019 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" event={"ID":"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a","Type":"ContainerDied","Data":"0f7a40811e85d5a066e68586bd083fa443ea710d08a1529cde50ff955365ac66"} Oct 30 05:31:32.000670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.999067 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" event={"ID":"9396c9bf-5ba4-4baf-accb-f63fd7f0d69a","Type":"ContainerDied","Data":"78d17c83664e4621dc09bbba23023349cec25a44904f5a5b650e8e287a130916"} Oct 30 05:31:32.000670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.999109 6587 scope.go:122] "RemoveContainer" containerID="0f7a40811e85d5a066e68586bd083fa443ea710d08a1529cde50ff955365ac66" Oct 30 05:31:32.000670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:31.999290 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv" Oct 30 05:31:32.005845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.004589 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" event={"ID":"66c8e0c3-624f-44f9-b797-b7837a2deb53","Type":"ContainerDied","Data":"b834dac7c259d81ea8b1f5097c5baeef3e1930b26c727b01fcdc5eea5c4a398b"} Oct 30 05:31:32.005845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.004598 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds" Oct 30 05:31:32.012712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.012138 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.012712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.012212 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.012712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.012301 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.012712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.012341 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdwxz\" (UniqueName: \"kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-kube-api-access-vdwxz\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.012712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.012439 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.012712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.012509 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.015332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.012944 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.015332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.013053 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.015332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.013139 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.030772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.030738 6587 scope.go:122] "RemoveContainer" containerID="0f7a40811e85d5a066e68586bd083fa443ea710d08a1529cde50ff955365ac66" Oct 30 05:31:32.032164 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:31:32.032022 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f7a40811e85d5a066e68586bd083fa443ea710d08a1529cde50ff955365ac66\": container with ID starting with 0f7a40811e85d5a066e68586bd083fa443ea710d08a1529cde50ff955365ac66 not found: ID does not exist" containerID="0f7a40811e85d5a066e68586bd083fa443ea710d08a1529cde50ff955365ac66" Oct 30 05:31:32.032164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.032066 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7a40811e85d5a066e68586bd083fa443ea710d08a1529cde50ff955365ac66"} err="failed to get container status \"0f7a40811e85d5a066e68586bd083fa443ea710d08a1529cde50ff955365ac66\": rpc error: code = NotFound desc = could not find container \"0f7a40811e85d5a066e68586bd083fa443ea710d08a1529cde50ff955365ac66\": container with ID starting with 0f7a40811e85d5a066e68586bd083fa443ea710d08a1529cde50ff955365ac66 not found: ID does not exist" Oct 30 05:31:32.032164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.032092 6587 scope.go:122] "RemoveContainer" containerID="3f515f4285dbcf1db1d53b8430d28d0549c0d4ab0169a5dd9d5bc1811b31f957" Oct 30 05:31:32.055079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.054095 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq"] Oct 30 05:31:32.062625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.062596 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7526ac7b-76wgq"] Oct 30 05:31:32.085973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.085693 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv"] Oct 30 05:31:32.097257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.097176 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv"] Oct 30 05:31:32.116424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.114679 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.116424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.114821 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.116424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.114914 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.116424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.115045 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.116424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.115052 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.116424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.115104 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.116424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.115160 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.116424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.115185 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdwxz\" (UniqueName: \"kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-kube-api-access-vdwxz\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.116424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.115227 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.116424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.115289 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.116424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.115982 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.116424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.116149 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.120594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.116963 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.120594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.117214 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.120594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.118429 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.120594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.120378 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.148556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.145076 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds"] Oct 30 05:31:32.157276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.157236 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.157599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.157574 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds"] Oct 30 05:31:32.168941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.168916 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdwxz\" (UniqueName: \"kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-kube-api-access-vdwxz\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.227064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.227034 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq"] Oct 30 05:31:32.227943 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:31:32.227910 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34622d6f_d009_43d4_affa_f16d4b51e5e6.slice/crio-ae9a16f48cdbcc515e5207224e7d9a3354904efb8cc243d885bd37f46a271403 WatchSource:0}: Error finding container ae9a16f48cdbcc515e5207224e7d9a3354904efb8cc243d885bd37f46a271403: Status 404 returned error can't find the container with id ae9a16f48cdbcc515e5207224e7d9a3354904efb8cc243d885bd37f46a271403 Oct 30 05:31:32.232773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.232745 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt"] Oct 30 05:31:32.250479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.250453 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-60a531-5bbkt"] Oct 30 05:31:32.297449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.297424 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:32.697158 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:31:32.697129 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffeb6a7_0efc_4336_89f8_2704a03d1398.slice/crio-0314b35dae08d65edd72b8e527d10f2137929231fb0281bb08cad7317b4bad05 WatchSource:0}: Error finding container 0314b35dae08d65edd72b8e527d10f2137929231fb0281bb08cad7317b4bad05: Status 404 returned error can't find the container with id 0314b35dae08d65edd72b8e527d10f2137929231fb0281bb08cad7317b4bad05 Oct 30 05:31:32.699712 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.699556 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz"] Oct 30 05:31:32.701067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.701034 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:31:32.701171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.701107 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:31:32.701171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:32.701140 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:31:33.020184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.020076 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" event={"ID":"34622d6f-d009-43d4-affa-f16d4b51e5e6","Type":"ContainerStarted","Data":"238cdf3903459c83cc6f781768b73e73b3bb5b2592f685f3199eabb2bd6e6a4b"} Oct 30 05:31:33.020184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.020143 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" event={"ID":"34622d6f-d009-43d4-affa-f16d4b51e5e6","Type":"ContainerStarted","Data":"ae9a16f48cdbcc515e5207224e7d9a3354904efb8cc243d885bd37f46a271403"} Oct 30 05:31:33.022625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.022595 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" Oct 30 05:31:33.023556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.022650 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" event={"ID":"3ffeb6a7-0efc-4336-89f8-2704a03d1398","Type":"ContainerStarted","Data":"c2de7d60740f9b53c4e3454bb42b8eee64c9c96fc5b36525432f42c7cd29cdac"} Oct 30 05:31:33.023556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.022682 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" event={"ID":"3ffeb6a7-0efc-4336-89f8-2704a03d1398","Type":"ContainerStarted","Data":"0314b35dae08d65edd72b8e527d10f2137929231fb0281bb08cad7317b4bad05"} Oct 30 05:31:33.071987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.071935 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" podStartSLOduration=47.071921304 podStartE2EDuration="47.071921304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:33.070603738 +0000 UTC m=+2817.598370525" watchObservedRunningTime="2026-10-30 05:31:33.071921304 +0000 UTC m=+2817.599688090" Oct 30 05:31:33.128276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.128212 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" podStartSLOduration=61.128193974 podStartE2EDuration="1m1.128193974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:33.12417963 +0000 UTC m=+2817.651946406" watchObservedRunningTime="2026-10-30 05:31:33.128193974 +0000 UTC m=+2817.655960762" Oct 30 05:31:33.297946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.297861 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:33.302169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.302137 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Oct 30 05:31:33.302343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.302194 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" podUID="3ffeb6a7-0efc-4336-89f8-2704a03d1398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Oct 30 05:31:33.588746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.588668 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c8e0c3-624f-44f9-b797-b7837a2deb53" path="/var/lib/kubelet/pods/66c8e0c3-624f-44f9-b797-b7837a2deb53/volumes" Oct 30 05:31:33.589263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.589234 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dfc61e5-0c2d-44a8-9398-b35decd958dd" path="/var/lib/kubelet/pods/8dfc61e5-0c2d-44a8-9398-b35decd958dd/volumes" Oct 30 05:31:33.589732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.589717 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9396c9bf-5ba4-4baf-accb-f63fd7f0d69a" path="/var/lib/kubelet/pods/9396c9bf-5ba4-4baf-accb-f63fd7f0d69a/volumes" Oct 30 05:31:33.591140 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:33.591123 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4551433-4795-4b7d-b1f7-84c78ec14d66" path="/var/lib/kubelet/pods/c4551433-4795-4b7d-b1f7-84c78ec14d66/volumes" Oct 30 05:31:34.298351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:34.298318 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Oct 30 05:31:34.298351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:34.298383 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" podUID="3ffeb6a7-0efc-4336-89f8-2704a03d1398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Oct 30 05:31:35.298004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.297969 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Oct 30 05:31:35.298221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.298018 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" podUID="3ffeb6a7-0efc-4336-89f8-2704a03d1398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Oct 30 05:31:35.390174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.390137 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5"] Oct 30 05:31:35.395089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.395067 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" Oct 30 05:31:35.412912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.412888 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5"] Oct 30 05:31:35.424235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.422745 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:31:35.424235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.423378 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:31:35.424235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.423543 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:31:35.428624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.428587 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:31:35.543307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.543130 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09fccf51-9zfq5\" (UID: \"5dc2230a-a2ff-4826-811f-4920130cef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" Oct 30 05:31:35.543307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.543240 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09fccf51-9zfq5\" (UID: \"5dc2230a-a2ff-4826-811f-4920130cef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" Oct 30 05:31:35.543307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.543306 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8qp\" (UniqueName: \"kubernetes.io/projected/5dc2230a-a2ff-4826-811f-4920130cef0c-kube-api-access-zn8qp\") pod \"maas-api-revoke-sub-09fccf51-9zfq5\" (UID: \"5dc2230a-a2ff-4826-811f-4920130cef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" Oct 30 05:31:35.644917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.644799 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8qp\" (UniqueName: \"kubernetes.io/projected/5dc2230a-a2ff-4826-811f-4920130cef0c-kube-api-access-zn8qp\") pod \"maas-api-revoke-sub-09fccf51-9zfq5\" (UID: \"5dc2230a-a2ff-4826-811f-4920130cef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" Oct 30 05:31:35.645260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.645234 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09fccf51-9zfq5\" (UID: \"5dc2230a-a2ff-4826-811f-4920130cef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" Oct 30 05:31:35.645425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.645339 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09fccf51-9zfq5\" (UID: \"5dc2230a-a2ff-4826-811f-4920130cef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" Oct 30 05:31:35.646275 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.646234 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09fccf51-9zfq5\" (UID: \"5dc2230a-a2ff-4826-811f-4920130cef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" Oct 30 05:31:35.648197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.648177 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09fccf51-9zfq5\" (UID: \"5dc2230a-a2ff-4826-811f-4920130cef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" Oct 30 05:31:35.668056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.668030 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8qp\" (UniqueName: \"kubernetes.io/projected/5dc2230a-a2ff-4826-811f-4920130cef0c-kube-api-access-zn8qp\") pod \"maas-api-revoke-sub-09fccf51-9zfq5\" (UID: \"5dc2230a-a2ff-4826-811f-4920130cef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" Oct 30 05:31:35.731508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.731469 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:31:35.735379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:35.733231 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" Oct 30 05:31:36.071806 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:31:36.071443 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc2230a_a2ff_4826_811f_4920130cef0c.slice/crio-e0aebb99223284300523dfa177494c876d1363def7e2a7be6adf18065ff2d70c WatchSource:0}: Error finding container e0aebb99223284300523dfa177494c876d1363def7e2a7be6adf18065ff2d70c: Status 404 returned error can't find the container with id e0aebb99223284300523dfa177494c876d1363def7e2a7be6adf18065ff2d70c Oct 30 05:31:36.071806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:36.071657 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5"] Oct 30 05:31:36.298368 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:36.298312 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Oct 30 05:31:36.298604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:36.298414 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" podUID="3ffeb6a7-0efc-4336-89f8-2704a03d1398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Oct 30 05:31:37.097001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:37.096937 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" event={"ID":"5dc2230a-a2ff-4826-811f-4920130cef0c","Type":"ContainerStarted","Data":"34752b18ecbcabbd1d8f01ba9d4bb59e6dc3182bb3b69b6e359f9e6a50591670"} Oct 30 05:31:37.097526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:37.097390 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" event={"ID":"5dc2230a-a2ff-4826-811f-4920130cef0c","Type":"ContainerStarted","Data":"e0aebb99223284300523dfa177494c876d1363def7e2a7be6adf18065ff2d70c"} Oct 30 05:31:37.175071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:37.175025 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" podStartSLOduration=2.17501024 podStartE2EDuration="2.17501024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:37.170218036 +0000 UTC m=+2821.697984825" watchObservedRunningTime="2026-10-30 05:31:37.17501024 +0000 UTC m=+2821.702777025" Oct 30 05:31:37.298002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:37.297971 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Oct 30 05:31:37.298157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:37.298019 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" podUID="3ffeb6a7-0efc-4336-89f8-2704a03d1398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Oct 30 05:31:38.111056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:38.111022 6587 generic.go:356] "Generic (PLEG): container finished" podID="5dc2230a-a2ff-4826-811f-4920130cef0c" containerID="34752b18ecbcabbd1d8f01ba9d4bb59e6dc3182bb3b69b6e359f9e6a50591670" exitCode=0 Oct 30 05:31:38.111538 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:38.111109 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" event={"ID":"5dc2230a-a2ff-4826-811f-4920130cef0c","Type":"ContainerDied","Data":"34752b18ecbcabbd1d8f01ba9d4bb59e6dc3182bb3b69b6e359f9e6a50591670"} Oct 30 05:31:38.298646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:38.298612 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Oct 30 05:31:38.298823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:38.298669 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" podUID="3ffeb6a7-0efc-4336-89f8-2704a03d1398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Oct 30 05:31:39.304404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.304377 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:39.305132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.305108 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:39.306055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.306035 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:31:39.339837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.339811 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" Oct 30 05:31:39.463895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.463812 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc2230a-a2ff-4826-811f-4920130cef0c-kube-api-access-zn8qp\" (UniqueName: \"kubernetes.io/projected/5dc2230a-a2ff-4826-811f-4920130cef0c-kube-api-access-zn8qp\") pod \"5dc2230a-a2ff-4826-811f-4920130cef0c\" (UID: \"5dc2230a-a2ff-4826-811f-4920130cef0c\") " Oct 30 05:31:39.464065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.463899 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-xks-ca\") pod \"5dc2230a-a2ff-4826-811f-4920130cef0c\" (UID: \"5dc2230a-a2ff-4826-811f-4920130cef0c\") " Oct 30 05:31:39.464156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.464140 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-openshift-ca\") pod \"5dc2230a-a2ff-4826-811f-4920130cef0c\" (UID: \"5dc2230a-a2ff-4826-811f-4920130cef0c\") " Oct 30 05:31:39.465353 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.465328 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-openshift-ca" pod "5dc2230a-a2ff-4826-811f-4920130cef0c" (UID: "5dc2230a-a2ff-4826-811f-4920130cef0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:39.500870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.500830 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc2230a-a2ff-4826-811f-4920130cef0c-kube-api-access-zn8qp" pod "5dc2230a-a2ff-4826-811f-4920130cef0c" (UID: "5dc2230a-a2ff-4826-811f-4920130cef0c"). InnerVolumeSpecName "kube-api-access-zn8qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:39.502925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.502895 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-xks-ca" pod "5dc2230a-a2ff-4826-811f-4920130cef0c" (UID: "5dc2230a-a2ff-4826-811f-4920130cef0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:39.567331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.567290 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:39.567331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.567330 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn8qp\" (UniqueName: \"kubernetes.io/projected/5dc2230a-a2ff-4826-811f-4920130cef0c-kube-api-access-zn8qp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:39.567565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.567348 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc2230a-a2ff-4826-811f-4920130cef0c-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:39.649220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.649180 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7"] Oct 30 05:31:39.650242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:39.649462 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" podUID="111bc68f-98c2-4041-a39f-daa438577eaf" containerName="istio-proxy" containerID="cri-o://c623275b4b69848e013f2bf0bf22db1d354e77da1fe75b9b10e686051786e1ae" gracePeriod=30 Oct 30 05:31:40.145127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.145085 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" event={"ID":"5dc2230a-a2ff-4826-811f-4920130cef0c","Type":"ContainerDied","Data":"e0aebb99223284300523dfa177494c876d1363def7e2a7be6adf18065ff2d70c"} Oct 30 05:31:40.146175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.145391 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0aebb99223284300523dfa177494c876d1363def7e2a7be6adf18065ff2d70c" Oct 30 05:31:40.146175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.145117 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5" Oct 30 05:31:40.199183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.199152 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8"] Oct 30 05:31:40.200183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.200164 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5dc2230a-a2ff-4826-811f-4920130cef0c" containerName="revoke-keys" Oct 30 05:31:40.200259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.200184 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc2230a-a2ff-4826-811f-4920130cef0c" containerName="revoke-keys" Oct 30 05:31:40.200392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.200380 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc2230a-a2ff-4826-811f-4920130cef0c" containerName="revoke-keys" Oct 30 05:31:40.204340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.204322 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" Oct 30 05:31:40.222645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.222618 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:31:40.222833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.222639 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:31:40.225883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.225855 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8"] Oct 30 05:31:40.314656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.314622 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8\" (UID: \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" Oct 30 05:31:40.315626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.314743 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lktvk\" (UniqueName: \"kubernetes.io/projected/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-kube-api-access-lktvk\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8\" (UID: \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" Oct 30 05:31:40.315626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.314852 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8\" (UID: \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" Oct 30 05:31:40.316964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.316940 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:31:40.317671 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.316975 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:31:40.416390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.416296 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8\" (UID: \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" Oct 30 05:31:40.417233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.416429 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktvk\" (UniqueName: \"kubernetes.io/projected/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-kube-api-access-lktvk\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8\" (UID: \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" Oct 30 05:31:40.417233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.416521 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8\" (UID: \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" Oct 30 05:31:40.417233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.417060 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8\" (UID: \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" Oct 30 05:31:40.419516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.419183 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8\" (UID: \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" Oct 30 05:31:40.442968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.442942 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktvk\" (UniqueName: \"kubernetes.io/projected/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-kube-api-access-lktvk\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8\" (UID: \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" Oct 30 05:31:40.521344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.521309 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" Oct 30 05:31:40.854265 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:31:40.854234 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1a717f_0bbc_4d19_92aa_1f1fad07b872.slice/crio-3b9a92267a3c11e1715bf1a0aac956de6a0f383615f95da26181eb9dbb6d0ef7 WatchSource:0}: Error finding container 3b9a92267a3c11e1715bf1a0aac956de6a0f383615f95da26181eb9dbb6d0ef7: Status 404 returned error can't find the container with id 3b9a92267a3c11e1715bf1a0aac956de6a0f383615f95da26181eb9dbb6d0ef7 Oct 30 05:31:40.855966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:40.854924 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8"] Oct 30 05:31:41.155515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:41.155477 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" event={"ID":"ad1a717f-0bbc-4d19-92aa-1f1fad07b872","Type":"ContainerStarted","Data":"3b9a92267a3c11e1715bf1a0aac956de6a0f383615f95da26181eb9dbb6d0ef7"} Oct 30 05:31:42.186626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:42.186527 6587 generic.go:356] "Generic (PLEG): container finished" podID="ad1a717f-0bbc-4d19-92aa-1f1fad07b872" containerID="eae619ff7a871863a55967672b5285626416cfe4618ef24e47ae222636b4ed06" exitCode=0 Oct 30 05:31:42.187091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:42.186829 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" event={"ID":"ad1a717f-0bbc-4d19-92aa-1f1fad07b872","Type":"ContainerDied","Data":"eae619ff7a871863a55967672b5285626416cfe4618ef24e47ae222636b4ed06"} Oct 30 05:31:42.509717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:42.509628 6587 scope.go:122] "RemoveContainer" containerID="a5be515343348f7ffbacc91604f5b6c9b6281562135e7615d2755663eec275c7" Oct 30 05:31:42.525284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:42.525256 6587 scope.go:122] "RemoveContainer" containerID="5e75bab47e1e84df7e285036b1f5be590a81bffdcc178130434980be468b7d62" Oct 30 05:31:42.537917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:42.537896 6587 scope.go:122] "RemoveContainer" containerID="cdd98a7ccde20d19aec50711daee7bcb6864d351ebc62060b367d08ae445c250" Oct 30 05:31:42.550865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:42.550838 6587 scope.go:122] "RemoveContainer" containerID="cd7199aae94b4cbebdc7ad4d0086cd1414e689d6efb8e9440032ad6a31fbf9f3" Oct 30 05:31:42.564063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:42.564037 6587 scope.go:122] "RemoveContainer" containerID="9f5dbb228bd11ae6d8c71b7869b073e3b95b0dfc15bffbda75757740812eb6c8" Oct 30 05:31:42.579241 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:42.579217 6587 scope.go:122] "RemoveContainer" containerID="8c8c2e17b64efedf51abc3d5d80b555e75f276a7c319742b791dfa8e0ed3a203" Oct 30 05:31:43.511550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:43.511525 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" Oct 30 05:31:43.642711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:43.642675 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-xks-ca\") pod \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\" (UID: \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\") " Oct 30 05:31:43.642911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:43.642883 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-kube-api-access-lktvk\" (UniqueName: \"kubernetes.io/projected/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-kube-api-access-lktvk\") pod \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\" (UID: \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\") " Oct 30 05:31:43.643097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:43.643076 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-openshift-ca\") pod \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\" (UID: \"ad1a717f-0bbc-4d19-92aa-1f1fad07b872\") " Oct 30 05:31:43.643940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:43.643915 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-openshift-ca" pod "ad1a717f-0bbc-4d19-92aa-1f1fad07b872" (UID: "ad1a717f-0bbc-4d19-92aa-1f1fad07b872"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:43.644192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:43.644171 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:43.645970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:43.645949 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-xks-ca" pod "ad1a717f-0bbc-4d19-92aa-1f1fad07b872" (UID: "ad1a717f-0bbc-4d19-92aa-1f1fad07b872"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:43.650858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:43.650834 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-kube-api-access-lktvk" pod "ad1a717f-0bbc-4d19-92aa-1f1fad07b872" (UID: "ad1a717f-0bbc-4d19-92aa-1f1fad07b872"). InnerVolumeSpecName "kube-api-access-lktvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:43.746392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:43.746253 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lktvk\" (UniqueName: \"kubernetes.io/projected/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-kube-api-access-lktvk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:43.746392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:43.746288 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1a717f-0bbc-4d19-92aa-1f1fad07b872-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:44.206870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:44.206841 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" Oct 30 05:31:44.207086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:44.206835 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8" event={"ID":"ad1a717f-0bbc-4d19-92aa-1f1fad07b872","Type":"ContainerDied","Data":"3b9a92267a3c11e1715bf1a0aac956de6a0f383615f95da26181eb9dbb6d0ef7"} Oct 30 05:31:44.207086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:44.206983 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b9a92267a3c11e1715bf1a0aac956de6a0f383615f95da26181eb9dbb6d0ef7" Oct 30 05:31:45.011350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.011324 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:31:45.065798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.065769 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" Oct 30 05:31:45.106646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.106608 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w"] Oct 30 05:31:45.108257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.107656 6587 cpu_manager.go:414] "containerMap: removing container" podUID="111bc68f-98c2-4041-a39f-daa438577eaf" containerName="istio-proxy" Oct 30 05:31:45.108257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.107676 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111bc68f-98c2-4041-a39f-daa438577eaf" containerName="istio-proxy" Oct 30 05:31:45.108257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.107694 6587 cpu_manager.go:414] "containerMap: removing container" podUID="ad1a717f-0bbc-4d19-92aa-1f1fad07b872" containerName="revoke-keys" Oct 30 05:31:45.108257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.107699 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1a717f-0bbc-4d19-92aa-1f1fad07b872" containerName="revoke-keys" Oct 30 05:31:45.108257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.107729 6587 cpu_manager.go:414] "containerMap: removing container" podUID="111bc68f-98c2-4041-a39f-daa438577eaf" containerName="istio-proxy" Oct 30 05:31:45.108257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.107734 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111bc68f-98c2-4041-a39f-daa438577eaf" containerName="istio-proxy" Oct 30 05:31:45.108257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.107868 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="111bc68f-98c2-4041-a39f-daa438577eaf" containerName="istio-proxy" Oct 30 05:31:45.108257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.107881 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1a717f-0bbc-4d19-92aa-1f1fad07b872" containerName="revoke-keys" Oct 30 05:31:45.108257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.107897 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="111bc68f-98c2-4041-a39f-daa438577eaf" containerName="istio-proxy" Oct 30 05:31:45.111865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.111845 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.155392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.152719 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w"] Oct 30 05:31:45.175107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.175065 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-certs\") pod \"111bc68f-98c2-4041-a39f-daa438577eaf\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " Oct 30 05:31:45.175329 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.175206 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/111bc68f-98c2-4041-a39f-daa438577eaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111bc68f-98c2-4041-a39f-daa438577eaf-istio-podinfo\") pod \"111bc68f-98c2-4041-a39f-daa438577eaf\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " Oct 30 05:31:45.175329 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.175297 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-data\") pod \"111bc68f-98c2-4041-a39f-daa438577eaf\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " Oct 30 05:31:45.175502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.175443 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-credential-socket\") pod \"111bc68f-98c2-4041-a39f-daa438577eaf\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " Oct 30 05:31:45.175502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.175492 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-istio-token\" (UniqueName: \"kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-istio-token\") pod \"111bc68f-98c2-4041-a39f-daa438577eaf\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " Oct 30 05:31:45.175623 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.175563 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111bc68f-98c2-4041-a39f-daa438577eaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111bc68f-98c2-4041-a39f-daa438577eaf-istiod-ca-cert\") pod \"111bc68f-98c2-4041-a39f-daa438577eaf\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " Oct 30 05:31:45.175702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.175631 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-kube-api-access-2ccq7\" (UniqueName: \"kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-kube-api-access-2ccq7\") pod \"111bc68f-98c2-4041-a39f-daa438577eaf\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " Oct 30 05:31:45.176632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.175723 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-envoy\") pod \"111bc68f-98c2-4041-a39f-daa438577eaf\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " Oct 30 05:31:45.176632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.175868 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-socket\") pod \"111bc68f-98c2-4041-a39f-daa438577eaf\" (UID: \"111bc68f-98c2-4041-a39f-daa438577eaf\") " Oct 30 05:31:45.176940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.176908 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-credential-socket" pod "111bc68f-98c2-4041-a39f-daa438577eaf" (UID: "111bc68f-98c2-4041-a39f-daa438577eaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:45.177538 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.177507 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-data" pod "111bc68f-98c2-4041-a39f-daa438577eaf" (UID: "111bc68f-98c2-4041-a39f-daa438577eaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:45.177837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.177258 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-certs" pod "111bc68f-98c2-4041-a39f-daa438577eaf" (UID: "111bc68f-98c2-4041-a39f-daa438577eaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:45.178007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.177784 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-socket" pod "111bc68f-98c2-4041-a39f-daa438577eaf" (UID: "111bc68f-98c2-4041-a39f-daa438577eaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:45.178798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.178155 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111bc68f-98c2-4041-a39f-daa438577eaf-istiod-ca-cert" pod "111bc68f-98c2-4041-a39f-daa438577eaf" (UID: "111bc68f-98c2-4041-a39f-daa438577eaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:45.179503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.179426 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:45.179503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.179448 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111bc68f-98c2-4041-a39f-daa438577eaf-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:45.179503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.179457 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:45.179503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.179466 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:45.179503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.179474 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:45.179932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.179843 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-kube-api-access-2ccq7" pod "111bc68f-98c2-4041-a39f-daa438577eaf" (UID: "111bc68f-98c2-4041-a39f-daa438577eaf"). InnerVolumeSpecName "kube-api-access-2ccq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:45.195232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.195196 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/111bc68f-98c2-4041-a39f-daa438577eaf-istio-podinfo" pod "111bc68f-98c2-4041-a39f-daa438577eaf" (UID: "111bc68f-98c2-4041-a39f-daa438577eaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:31:45.195428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.195296 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-envoy" pod "111bc68f-98c2-4041-a39f-daa438577eaf" (UID: "111bc68f-98c2-4041-a39f-daa438577eaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:45.196219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.195635 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-istio-token" pod "111bc68f-98c2-4041-a39f-daa438577eaf" (UID: "111bc68f-98c2-4041-a39f-daa438577eaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:45.230715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.230093 6587 generic.go:356] "Generic (PLEG): container finished" podID="111bc68f-98c2-4041-a39f-daa438577eaf" containerID="c623275b4b69848e013f2bf0bf22db1d354e77da1fe75b9b10e686051786e1ae" exitCode=0 Oct 30 05:31:45.230715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.230432 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" event={"ID":"111bc68f-98c2-4041-a39f-daa438577eaf","Type":"ContainerDied","Data":"c623275b4b69848e013f2bf0bf22db1d354e77da1fe75b9b10e686051786e1ae"} Oct 30 05:31:45.230715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.230487 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" event={"ID":"111bc68f-98c2-4041-a39f-daa438577eaf","Type":"ContainerDied","Data":"4416b2e73ad46c157f6ef305359980ade87342bb9b5ece900114e6134999d44d"} Oct 30 05:31:45.230715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.230514 6587 scope.go:122] "RemoveContainer" containerID="c623275b4b69848e013f2bf0bf22db1d354e77da1fe75b9b10e686051786e1ae" Oct 30 05:31:45.230715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.230673 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7" Oct 30 05:31:45.256206 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.256180 6587 scope.go:122] "RemoveContainer" containerID="6f0835455192fd83e9ef561dd2773d9e3402b63514a24b6f502e56e637b020fa" Oct 30 05:31:45.269713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.269691 6587 scope.go:122] "RemoveContainer" containerID="c623275b4b69848e013f2bf0bf22db1d354e77da1fe75b9b10e686051786e1ae" Oct 30 05:31:45.270012 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:31:45.269989 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c623275b4b69848e013f2bf0bf22db1d354e77da1fe75b9b10e686051786e1ae\": container with ID starting with c623275b4b69848e013f2bf0bf22db1d354e77da1fe75b9b10e686051786e1ae not found: ID does not exist" containerID="c623275b4b69848e013f2bf0bf22db1d354e77da1fe75b9b10e686051786e1ae" Oct 30 05:31:45.270057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.270022 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c623275b4b69848e013f2bf0bf22db1d354e77da1fe75b9b10e686051786e1ae"} err="failed to get container status \"c623275b4b69848e013f2bf0bf22db1d354e77da1fe75b9b10e686051786e1ae\": rpc error: code = NotFound desc = could not find container \"c623275b4b69848e013f2bf0bf22db1d354e77da1fe75b9b10e686051786e1ae\": container with ID starting with c623275b4b69848e013f2bf0bf22db1d354e77da1fe75b9b10e686051786e1ae not found: ID does not exist" Oct 30 05:31:45.270057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.270040 6587 scope.go:122] "RemoveContainer" containerID="6f0835455192fd83e9ef561dd2773d9e3402b63514a24b6f502e56e637b020fa" Oct 30 05:31:45.270308 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:31:45.270287 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0835455192fd83e9ef561dd2773d9e3402b63514a24b6f502e56e637b020fa\": container with ID starting with 6f0835455192fd83e9ef561dd2773d9e3402b63514a24b6f502e56e637b020fa not found: ID does not exist" containerID="6f0835455192fd83e9ef561dd2773d9e3402b63514a24b6f502e56e637b020fa" Oct 30 05:31:45.270368 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.270314 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0835455192fd83e9ef561dd2773d9e3402b63514a24b6f502e56e637b020fa"} err="failed to get container status \"6f0835455192fd83e9ef561dd2773d9e3402b63514a24b6f502e56e637b020fa\": rpc error: code = NotFound desc = could not find container \"6f0835455192fd83e9ef561dd2773d9e3402b63514a24b6f502e56e637b020fa\": container with ID starting with 6f0835455192fd83e9ef561dd2773d9e3402b63514a24b6f502e56e637b020fa not found: ID does not exist" Oct 30 05:31:45.281830 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.281802 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2942b-f563-4e66-b8b5-f46845890275-istiod-ca-cert\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.281975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.281958 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-istio-token\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.282099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.282081 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.282156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.282143 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-data\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.282213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.282167 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2942b-f563-4e66-b8b5-f46845890275-istio-podinfo\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.282319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.282301 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-credential-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.282421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.282403 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-certs\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.282483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.282450 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsdhx\" (UniqueName: \"kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-kube-api-access-xsdhx\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.282550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.282537 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-envoy\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.282709 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.282682 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111bc68f-98c2-4041-a39f-daa438577eaf-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:45.282851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.282711 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:45.282851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.282725 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ccq7\" (UniqueName: \"kubernetes.io/projected/111bc68f-98c2-4041-a39f-daa438577eaf-kube-api-access-2ccq7\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:45.282851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.282736 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111bc68f-98c2-4041-a39f-daa438577eaf-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:31:45.298205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.298176 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7"] Oct 30 05:31:45.303549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.303520 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7"] Oct 30 05:31:45.386194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.386147 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.386501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.386241 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-data\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.386501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.386271 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2942b-f563-4e66-b8b5-f46845890275-istio-podinfo\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.386501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.386333 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-credential-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.386501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.386410 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-certs\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.386501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.386461 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsdhx\" (UniqueName: \"kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-kube-api-access-xsdhx\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.386918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.386512 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-envoy\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.386918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.386791 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2942b-f563-4e66-b8b5-f46845890275-istiod-ca-cert\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.386918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.386910 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-istio-token\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.389557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.389527 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.390337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.389554 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-certs\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.390337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.389896 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-credential-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.390337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.390113 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-data\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.390580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.390343 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2942b-f563-4e66-b8b5-f46845890275-istiod-ca-cert\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.392235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.392210 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-envoy\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.392502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.392481 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2942b-f563-4e66-b8b5-f46845890275-istio-podinfo\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.413036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.413002 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-istio-token\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.413187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.413093 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsdhx\" (UniqueName: \"kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-kube-api-access-xsdhx\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.454758 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.454720 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:45.592561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.592527 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111bc68f-98c2-4041-a39f-daa438577eaf" path="/var/lib/kubelet/pods/111bc68f-98c2-4041-a39f-daa438577eaf/volumes" Oct 30 05:31:45.790771 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:31:45.790735 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d2942b_f563_4e66_b8b5_f46845890275.slice/crio-467601ced7aa8fa9424732069291c05e4ad90f44a13278cd224fc0175715afca WatchSource:0}: Error finding container 467601ced7aa8fa9424732069291c05e4ad90f44a13278cd224fc0175715afca: Status 404 returned error can't find the container with id 467601ced7aa8fa9424732069291c05e4ad90f44a13278cd224fc0175715afca Oct 30 05:31:45.792335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.792303 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w"] Oct 30 05:31:45.794010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.793978 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:31:45.794724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.794057 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:31:45.794724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:45.794101 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:31:46.246642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:46.246604 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" event={"ID":"58d2942b-f563-4e66-b8b5-f46845890275","Type":"ContainerStarted","Data":"b36ebdcb6a80f376458cf97bb50df624534720fd1c900b56f45bd352d6153d38"} Oct 30 05:31:46.247124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:46.246677 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" event={"ID":"58d2942b-f563-4e66-b8b5-f46845890275","Type":"ContainerStarted","Data":"467601ced7aa8fa9424732069291c05e4ad90f44a13278cd224fc0175715afca"} Oct 30 05:31:46.306200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:46.306149 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" podStartSLOduration=12.306133812 podStartE2EDuration="12.306133812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:46.303616466 +0000 UTC m=+2830.831383252" watchObservedRunningTime="2026-10-30 05:31:46.306133812 +0000 UTC m=+2830.833900598" Oct 30 05:31:46.460029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:46.459985 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s"] Oct 30 05:31:46.462444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:46.462415 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:46.463372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:46.463328 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Oct 30 05:31:46.463502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:46.463391 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" podUID="58d2942b-f563-4e66-b8b5-f46845890275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Oct 30 05:31:47.473383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:47.471421 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Oct 30 05:31:47.473383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:47.471485 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" podUID="58d2942b-f563-4e66-b8b5-f46845890275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Oct 30 05:31:48.477424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:48.463002 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Oct 30 05:31:48.477424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:48.463069 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" podUID="58d2942b-f563-4e66-b8b5-f46845890275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Oct 30 05:31:49.460889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:49.460849 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Oct 30 05:31:49.461220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:49.461189 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" podUID="58d2942b-f563-4e66-b8b5-f46845890275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Oct 30 05:31:50.544383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:50.533485 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Oct 30 05:31:50.544383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:50.533546 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" podUID="58d2942b-f563-4e66-b8b5-f46845890275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Oct 30 05:31:51.458828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:51.458787 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Oct 30 05:31:51.459162 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:51.459125 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" podUID="58d2942b-f563-4e66-b8b5-f46845890275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Oct 30 05:31:52.482428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:52.458407 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Oct 30 05:31:52.482428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:52.458492 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" podUID="58d2942b-f563-4e66-b8b5-f46845890275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Oct 30 05:31:52.791595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:52.791020 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr"] Oct 30 05:31:52.880383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:52.879412 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv"] Oct 30 05:31:53.456997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:53.456956 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Oct 30 05:31:53.457329 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:53.457296 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" podUID="58d2942b-f563-4e66-b8b5-f46845890275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Oct 30 05:31:54.462432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:54.462399 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:54.463032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:54.463012 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:54.463990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:54.463961 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:31:54.757145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:54.757061 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl"] Oct 30 05:31:56.044504 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:31:56.009093 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:31:57.792436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:57.792372 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" podUID="d13e466d-8257-409c-8d60-e8d7ec1a70f7" containerName="payload-processing" containerID="cri-o://5efacb5c4507c357f0805ffc5636bc77382a545fbd1da59aba4b35df9bd4ab57" gracePeriod=25 Oct 30 05:31:57.880156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:57.880109 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" podUID="b966db7f-5185-48c0-80f4-4584735d9c89" containerName="payload-pre-processing" containerID="cri-o://c3db0ab38785dbb9eecc80b6bfb58df7babb27951dd3bbe95e30977d82f6f286" gracePeriod=25 Oct 30 05:31:58.993922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:58.993888 6587 generic.go:356] "Generic (PLEG): container finished" podID="d13e466d-8257-409c-8d60-e8d7ec1a70f7" containerID="5efacb5c4507c357f0805ffc5636bc77382a545fbd1da59aba4b35df9bd4ab57" exitCode=0 Oct 30 05:31:58.994405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:58.993966 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" event={"ID":"d13e466d-8257-409c-8d60-e8d7ec1a70f7","Type":"ContainerDied","Data":"5efacb5c4507c357f0805ffc5636bc77382a545fbd1da59aba4b35df9bd4ab57"} Oct 30 05:31:59.093078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:59.092769 6587 generic.go:356] "Generic (PLEG): container finished" podID="b966db7f-5185-48c0-80f4-4584735d9c89" containerID="c3db0ab38785dbb9eecc80b6bfb58df7babb27951dd3bbe95e30977d82f6f286" exitCode=0 Oct 30 05:31:59.093078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:59.092833 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" event={"ID":"b966db7f-5185-48c0-80f4-4584735d9c89","Type":"ContainerDied","Data":"c3db0ab38785dbb9eecc80b6bfb58df7babb27951dd3bbe95e30977d82f6f286"} Oct 30 05:31:59.314932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:59.314885 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5"] Oct 30 05:31:59.315184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:59.315161 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" podUID="f7dfe16d-d341-4c77-9279-0e95437cdf49" containerName="payload-processing" containerID="cri-o://5c9eb5fa118e0bb8322ccfe0bedbf52ee51c5dfac8f1ada24d9237aeaf2fdeb0" gracePeriod=30 Oct 30 05:31:59.341547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:59.341496 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:31:59.341739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:31:59.341576 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:32:00.071422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:00.071376 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8"] Oct 30 05:32:00.144842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:00.085007 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec92c7-2vtk8"] Oct 30 05:32:00.343761 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:32:00.343683 6587 logging.go:55] [core] [Channel #13638 SubChannel #13639]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.82:50052", ServerName: "10.134.1.82:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.82:50052: connect: connection refused" Oct 30 05:32:00.448202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:00.448152 6587 patch_prober.go:29] interesting pod/metrics-server-7d6768cb7d-ssqrt container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:10250/livez\": context deadline exceeded" start-of-body= Oct 30 05:32:00.449988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:00.448249 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" podUID="d9fd93f4-68f8-4711-ad54-410d10af6c35" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.12:10250/livez\": context deadline exceeded" Oct 30 05:32:00.621557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:00.621468 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-2v6v6" podUID="ac919fe3-ac98-44cd-8d95-6df70a406fa4" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.18:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:32:00.907894 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:00.907796 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" podUID="b966db7f-5185-48c0-80f4-4584735d9c89" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.82:50052\" within 1s: context deadline exceeded" Oct 30 05:32:01.368575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:01.368523 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" podUID="d296c368-73b0-4326-87bc-cd3508c843b9" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:32:01.403671 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:32:01.403638 6587 logging.go:55] [core] [Channel #13646 SubChannel #13647]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.83:50052", ServerName: "10.134.1.83:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.83:50052: connect: connection refused" Oct 30 05:32:01.573735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:01.507829 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" podUID="c382def9-ebcf-44e3-a70f-cca002550278" containerName="main" containerID="cri-o://ff3c5b9c4b60163c97a925d956d395d44e76ec3b91645cb148567b979d3f60a5" gracePeriod=45 Oct 30 05:32:01.639672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:01.585563 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad1a717f-0bbc-4d19-92aa-1f1fad07b872" path="/var/lib/kubelet/pods/ad1a717f-0bbc-4d19-92aa-1f1fad07b872/volumes" Oct 30 05:32:01.948951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:01.927149 6587 patch_prober.go:29] interesting pod/metrics-server-7d6768cb7d-ssqrt container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.12:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:32:01.948951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:01.927230 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-7d6768cb7d-ssqrt" podUID="d9fd93f4-68f8-4711-ad54-410d10af6c35" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.12:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:32:02.386667 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:02.177936 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-tw5cx" podUID="d02fa1f2-298d-4a23-a302-6a59e1cbd188" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.30:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:32:02.403322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:02.403280 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" podUID="d13e466d-8257-409c-8d60-e8d7ec1a70f7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.83:50052\" within 1s: context deadline exceeded" Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.077227 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" podUID="c382def9-ebcf-44e3-a70f-cca002550278" containerName="main" probeResult="failure" output="Get \"https://10.134.1.50:8000/ready\": dial tcp 10.134.1.50:8000: connect: connection refused" Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.248806 6587 generic.go:356] "Generic (PLEG): container finished" podID="c382def9-ebcf-44e3-a70f-cca002550278" containerID="ff3c5b9c4b60163c97a925d956d395d44e76ec3b91645cb148567b979d3f60a5" exitCode=0 Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.248884 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" event={"ID":"c382def9-ebcf-44e3-a70f-cca002550278","Type":"ContainerDied","Data":"ff3c5b9c4b60163c97a925d956d395d44e76ec3b91645cb148567b979d3f60a5"} Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.251145 6587 generic.go:356] "Generic (PLEG): container finished" podID="f7dfe16d-d341-4c77-9279-0e95437cdf49" containerID="5c9eb5fa118e0bb8322ccfe0bedbf52ee51c5dfac8f1ada24d9237aeaf2fdeb0" exitCode=0 Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.251210 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" event={"ID":"f7dfe16d-d341-4c77-9279-0e95437cdf49","Type":"ContainerDied","Data":"5c9eb5fa118e0bb8322ccfe0bedbf52ee51c5dfac8f1ada24d9237aeaf2fdeb0"} Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.714733 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.779790 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13e466d-8257-409c-8d60-e8d7ec1a70f7-kube-api-access-ff6jm\" (UniqueName: \"kubernetes.io/projected/d13e466d-8257-409c-8d60-e8d7ec1a70f7-kube-api-access-ff6jm\") pod \"d13e466d-8257-409c-8d60-e8d7ec1a70f7\" (UID: \"d13e466d-8257-409c-8d60-e8d7ec1a70f7\") " Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.780230 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13e466d-8257-409c-8d60-e8d7ec1a70f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13e466d-8257-409c-8d60-e8d7ec1a70f7-plugins-config-volume\") pod \"d13e466d-8257-409c-8d60-e8d7ec1a70f7\" (UID: \"d13e466d-8257-409c-8d60-e8d7ec1a70f7\") " Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.780713 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13e466d-8257-409c-8d60-e8d7ec1a70f7-plugins-config-volume" pod "d13e466d-8257-409c-8d60-e8d7ec1a70f7" (UID: "d13e466d-8257-409c-8d60-e8d7ec1a70f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.781255 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13e466d-8257-409c-8d60-e8d7ec1a70f7-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.927148 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.989640 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b966db7f-5185-48c0-80f4-4584735d9c89-kube-api-access-22cqm\" (UniqueName: \"kubernetes.io/projected/b966db7f-5185-48c0-80f4-4584735d9c89-kube-api-access-22cqm\") pod \"b966db7f-5185-48c0-80f4-4584735d9c89\" (UID: \"b966db7f-5185-48c0-80f4-4584735d9c89\") " Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.990037 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b966db7f-5185-48c0-80f4-4584735d9c89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b966db7f-5185-48c0-80f4-4584735d9c89-plugins-config-volume\") pod \"b966db7f-5185-48c0-80f4-4584735d9c89\" (UID: \"b966db7f-5185-48c0-80f4-4584735d9c89\") " Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.990427 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b966db7f-5185-48c0-80f4-4584735d9c89-plugins-config-volume" pod "b966db7f-5185-48c0-80f4-4584735d9c89" (UID: "b966db7f-5185-48c0-80f4-4584735d9c89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:04.991784 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b966db7f-5185-48c0-80f4-4584735d9c89-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:05.203290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.125528 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq"] Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.312947 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b966db7f-5185-48c0-80f4-4584735d9c89" containerName="payload-pre-processing" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.312976 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b966db7f-5185-48c0-80f4-4584735d9c89" containerName="payload-pre-processing" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.313088 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d13e466d-8257-409c-8d60-e8d7ec1a70f7" containerName="payload-processing" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.313105 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13e466d-8257-409c-8d60-e8d7ec1a70f7" containerName="payload-processing" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.313737 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13e466d-8257-409c-8d60-e8d7ec1a70f7" containerName="payload-processing" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.313788 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b966db7f-5185-48c0-80f4-4584735d9c89" containerName="payload-pre-processing" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.325719 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.396184 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.414211 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.414436 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.416561 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7dfe16d-d341-4c77-9279-0e95437cdf49-kube-api-access-ckr8r\" (UniqueName: \"kubernetes.io/projected/f7dfe16d-d341-4c77-9279-0e95437cdf49-kube-api-access-ckr8r\") pod \"f7dfe16d-d341-4c77-9279-0e95437cdf49\" (UID: \"f7dfe16d-d341-4c77-9279-0e95437cdf49\") " Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.416674 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7dfe16d-d341-4c77-9279-0e95437cdf49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7dfe16d-d341-4c77-9279-0e95437cdf49-plugins-config-volume\") pod \"f7dfe16d-d341-4c77-9279-0e95437cdf49\" (UID: \"f7dfe16d-d341-4c77-9279-0e95437cdf49\") " Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.417554 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7dfe16d-d341-4c77-9279-0e95437cdf49-plugins-config-volume" pod "f7dfe16d-d341-4c77-9279-0e95437cdf49" (UID: "f7dfe16d-d341-4c77-9279-0e95437cdf49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.508983 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.518613 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c382def9-ebcf-44e3-a70f-cca002550278-kube-api-access-w8qz8\" (UniqueName: \"kubernetes.io/projected/c382def9-ebcf-44e3-a70f-cca002550278-kube-api-access-w8qz8\") pod \"c382def9-ebcf-44e3-a70f-cca002550278\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.518957 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-model-cache\") pod \"c382def9-ebcf-44e3-a70f-cca002550278\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.519023 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-tmp-dir\") pod \"c382def9-ebcf-44e3-a70f-cca002550278\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.519080 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c382def9-ebcf-44e3-a70f-cca002550278-tls-certs\" (UniqueName: \"kubernetes.io/secret/c382def9-ebcf-44e3-a70f-cca002550278-tls-certs\") pod \"c382def9-ebcf-44e3-a70f-cca002550278\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.519169 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-home\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-home\") pod \"c382def9-ebcf-44e3-a70f-cca002550278\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.519319 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-dshm\") pod \"c382def9-ebcf-44e3-a70f-cca002550278\" (UID: \"c382def9-ebcf-44e3-a70f-cca002550278\") " Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.521227 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7dfe16d-d341-4c77-9279-0e95437cdf49-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.522271 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-tmp-dir" pod "c382def9-ebcf-44e3-a70f-cca002550278" (UID: "c382def9-ebcf-44e3-a70f-cca002550278"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.522507 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-model-cache" pod "c382def9-ebcf-44e3-a70f-cca002550278" (UID: "c382def9-ebcf-44e3-a70f-cca002550278"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.522900 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-home" pod "c382def9-ebcf-44e3-a70f-cca002550278" (UID: "c382def9-ebcf-44e3-a70f-cca002550278"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.528995 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c382def9-ebcf-44e3-a70f-cca002550278-kube-api-access-w8qz8" pod "c382def9-ebcf-44e3-a70f-cca002550278" (UID: "c382def9-ebcf-44e3-a70f-cca002550278"). InnerVolumeSpecName "kube-api-access-w8qz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.577614 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c382def9-ebcf-44e3-a70f-cca002550278-tls-certs" pod "c382def9-ebcf-44e3-a70f-cca002550278" (UID: "c382def9-ebcf-44e3-a70f-cca002550278"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.577704 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b966db7f-5185-48c0-80f4-4584735d9c89-kube-api-access-22cqm" pod "b966db7f-5185-48c0-80f4-4584735d9c89" (UID: "b966db7f-5185-48c0-80f4-4584735d9c89"). InnerVolumeSpecName "kube-api-access-22cqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.580735 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13e466d-8257-409c-8d60-e8d7ec1a70f7-kube-api-access-ff6jm" pod "d13e466d-8257-409c-8d60-e8d7ec1a70f7" (UID: "d13e466d-8257-409c-8d60-e8d7ec1a70f7"). InnerVolumeSpecName "kube-api-access-ff6jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.580739 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-dshm" pod "c382def9-ebcf-44e3-a70f-cca002550278" (UID: "c382def9-ebcf-44e3-a70f-cca002550278"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.622671 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7dfe16d-d341-4c77-9279-0e95437cdf49-kube-api-access-ckr8r" pod "f7dfe16d-d341-4c77-9279-0e95437cdf49" (UID: "f7dfe16d-d341-4c77-9279-0e95437cdf49"). InnerVolumeSpecName "kube-api-access-ckr8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.623519 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff6jm\" (UniqueName: \"kubernetes.io/projected/d13e466d-8257-409c-8d60-e8d7ec1a70f7-kube-api-access-ff6jm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.623553 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.623571 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8qz8\" (UniqueName: \"kubernetes.io/projected/c382def9-ebcf-44e3-a70f-cca002550278-kube-api-access-w8qz8\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.623585 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22cqm\" (UniqueName: \"kubernetes.io/projected/b966db7f-5185-48c0-80f4-4584735d9c89-kube-api-access-22cqm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.623599 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckr8r\" (UniqueName: \"kubernetes.io/projected/f7dfe16d-d341-4c77-9279-0e95437cdf49-kube-api-access-ckr8r\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.623613 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.623669 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.623682 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c382def9-ebcf-44e3-a70f-cca002550278-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:05.645530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.623694 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c382def9-ebcf-44e3-a70f-cca002550278-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.798902 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.812139 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv"] Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.813605 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c382def9-ebcf-44e3-a70f-cca002550278" containerName="main" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.813626 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c382def9-ebcf-44e3-a70f-cca002550278" containerName="main" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.813675 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f7dfe16d-d341-4c77-9279-0e95437cdf49" containerName="payload-processing" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.813684 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7dfe16d-d341-4c77-9279-0e95437cdf49" containerName="payload-processing" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.813916 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c382def9-ebcf-44e3-a70f-cca002550278" containerName="main" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.813938 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7dfe16d-d341-4c77-9279-0e95437cdf49" containerName="payload-processing" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.826812 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-credential-socket\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.826865 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-token\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:05.998174 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ec1091-openshift-default-dockercfg-v4nv2\"" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.097529 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5" event={"ID":"f7dfe16d-d341-4c77-9279-0e95437cdf49","Type":"ContainerDied","Data":"f767a90230efe556a95d01804bef0a909982622134866c6b28c81f10c37d4d96"} Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.097587 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" event={"ID":"c382def9-ebcf-44e3-a70f-cca002550278","Type":"ContainerDied","Data":"eb428c40a144a116932480c0369728439d183e3776dee4c17739e3e9096bed19"} Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.097608 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr" event={"ID":"d13e466d-8257-409c-8d60-e8d7ec1a70f7","Type":"ContainerDied","Data":"3d0783d263400c19f23a6b543b1dd9d961b74da0fd845735c899755a134f6cce"} Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.097634 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" event={"ID":"b966db7f-5185-48c0-80f4-4584735d9c89","Type":"ContainerDied","Data":"e014ea6db21119204e9272d71f369b3d7705e1f14bd465a2de7e47b09b81d80c"} Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.098062 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.098538 6587 scope.go:122] "RemoveContainer" containerID="5c9eb5fa118e0bb8322ccfe0bedbf52ee51c5dfac8f1ada24d9237aeaf2fdeb0" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.098590 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" podUID="c382def9-ebcf-44e3-a70f-cca002550278" containerName="main" probeResult="failure" output="Get \"https://10.134.1.50:8000/ready\": context deadline exceeded" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.098700 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.125127 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5530d72-64d2-48b1-9f51-362f4a41b916-istiod-ca-cert\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.125294 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-podinfo\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.125325 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwzpx\" (UniqueName: \"kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-kube-api-access-pwzpx\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.125351 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-data\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.125560 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-envoy\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.125615 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-socket\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.125807 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-certs\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.227517 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.227618 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160a7128-f92e-420b-bd00-c44142af02bc-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.227668 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-credential-socket\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.227701 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160a7128-f92e-420b-bd00-c44142af02bc-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.227739 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-token\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.227767 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqd6\" (UniqueName: \"kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-kube-api-access-pbqd6\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.227820 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.227893 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5530d72-64d2-48b1-9f51-362f4a41b916-istiod-ca-cert\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.227995 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-podinfo\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.228022 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.228051 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwzpx\" (UniqueName: \"kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-kube-api-access-pwzpx\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.228079 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-data\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.228176 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.228206 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-envoy\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.228251 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-socket\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.228279 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.228454 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.228537 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-certs\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.229377 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-certs\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.229673 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-credential-socket\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.230566 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5530d72-64d2-48b1-9f51-362f4a41b916-istiod-ca-cert\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.231416 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-socket\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.231685 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-data\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.297014 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr"] Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.303801 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr"] Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.331218 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.331292 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160a7128-f92e-420b-bd00-c44142af02bc-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.331325 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160a7128-f92e-420b-bd00-c44142af02bc-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.331369 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqd6\" (UniqueName: \"kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-kube-api-access-pbqd6\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.331426 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.331612 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.331733 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.331784 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.331899 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.332303 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160a7128-f92e-420b-bd00-c44142af02bc-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.332376 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.332599 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.332886 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.333270 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.337710 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5"] Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.345113 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5"] Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.355953 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.381597 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-envoy\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.381848 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.381987 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-podinfo\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.382897 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwzpx\" (UniqueName: \"kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-kube-api-access-pwzpx\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.383208 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-token\") pod \"e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.383619 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.383802 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160a7128-f92e-420b-bd00-c44142af02bc-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.384181 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqd6\" (UniqueName: \"kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-kube-api-access-pbqd6\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.451123 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s"] Oct 30 05:32:06.585453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.466451 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34722e/sub-test-model-34722e-kserve-5cc88f6d79-5s64s"] Oct 30 05:32:07.469591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.777765 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:07.469591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:06.981039 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:07.511582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:07.470942 6587 scope.go:122] "RemoveContainer" containerID="ff3c5b9c4b60163c97a925d956d395d44e76ec3b91645cb148567b979d3f60a5" Oct 30 05:32:07.728150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:07.587928 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c382def9-ebcf-44e3-a70f-cca002550278" path="/var/lib/kubelet/pods/c382def9-ebcf-44e3-a70f-cca002550278/volumes" Oct 30 05:32:07.728150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:07.588516 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13e466d-8257-409c-8d60-e8d7ec1a70f7" path="/var/lib/kubelet/pods/d13e466d-8257-409c-8d60-e8d7ec1a70f7/volumes" Oct 30 05:32:07.728150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:07.665081 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7dfe16d-d341-4c77-9279-0e95437cdf49" path="/var/lib/kubelet/pods/f7dfe16d-d341-4c77-9279-0e95437cdf49/volumes" Oct 30 05:32:08.356865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:07.935138 6587 scope.go:122] "RemoveContainer" containerID="5efacb5c4507c357f0805ffc5636bc77382a545fbd1da59aba4b35df9bd4ab57" Oct 30 05:32:08.356865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:08.048004 6587 scope.go:122] "RemoveContainer" containerID="c3db0ab38785dbb9eecc80b6bfb58df7babb27951dd3bbe95e30977d82f6f286" Oct 30 05:32:08.651578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:08.431930 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv"] Oct 30 05:32:08.743261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:08.724542 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq"] Oct 30 05:32:09.575511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:09.575472 6587 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-67crm container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:32:09.575964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:09.575543 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67crm" podUID="a0780f4f-55b4-49c6-80a1-4026cc98f858" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:32:10.316895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:10.316816 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" podUID="79db524a-27ca-40a0-9599-a3a172b15ecb" containerName="main" containerID="cri-o://729d1aac14e1a36f4994282f755f9a0461c4987cf09813c38b4b63355d05d243" gracePeriod=45 Oct 30 05:32:11.004244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:10.978474 6587 generic.go:356] "Generic (PLEG): container finished" podID="79db524a-27ca-40a0-9599-a3a172b15ecb" containerID="729d1aac14e1a36f4994282f755f9a0461c4987cf09813c38b4b63355d05d243" exitCode=0 Oct 30 05:32:11.004244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:10.978542 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" event={"ID":"79db524a-27ca-40a0-9599-a3a172b15ecb","Type":"ContainerDied","Data":"729d1aac14e1a36f4994282f755f9a0461c4987cf09813c38b4b63355d05d243"} Oct 30 05:32:11.020400 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:32:11.020367 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79db524a_27ca_40a0_9599_a3a172b15ecb.slice/crio-conmon-729d1aac14e1a36f4994282f755f9a0461c4987cf09813c38b4b63355d05d243.scope\": RecentStats: unable to find data in memory cache]" Oct 30 05:32:11.424931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:11.424885 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" podUID="79db524a-27ca-40a0-9599-a3a172b15ecb" containerName="main" probeResult="failure" output="Get \"https://10.134.1.58:8000/ready\": dial tcp 10.134.1.58:8000: connect: connection refused" Oct 30 05:32:12.449583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:12.449522 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" podUID="79db524a-27ca-40a0-9599-a3a172b15ecb" containerName="main" probeResult="failure" output="Get \"https://10.134.1.58:8000/ready\": dial tcp 10.134.1.58:8000: connect: connection refused" Oct 30 05:32:12.463565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:12.449698 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:32:13.426064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:13.425804 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" podUID="79db524a-27ca-40a0-9599-a3a172b15ecb" containerName="main" probeResult="failure" output="Get \"https://10.134.1.58:8000/ready\": dial tcp 10.134.1.58:8000: connect: connection refused" Oct 30 05:32:14.376724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.215982 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" event={"ID":"79db524a-27ca-40a0-9599-a3a172b15ecb","Type":"ContainerDied","Data":"3f8bf197aff9fd29708ecb41dc8097e91853e3a6a49fb372a9f60fdba50cb48f"} Oct 30 05:32:14.376724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.216028 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f8bf197aff9fd29708ecb41dc8097e91853e3a6a49fb372a9f60fdba50cb48f" Oct 30 05:32:14.376724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.268076 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:32:14.376724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.341505 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:32:14.376724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.341564 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.426838 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-dshm\") pod \"79db524a-27ca-40a0-9599-a3a172b15ecb\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.426955 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-model-cache\") pod \"79db524a-27ca-40a0-9599-a3a172b15ecb\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.427159 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-tmp-dir\") pod \"79db524a-27ca-40a0-9599-a3a172b15ecb\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.427216 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79db524a-27ca-40a0-9599-a3a172b15ecb-kube-api-access-wz857\" (UniqueName: \"kubernetes.io/projected/79db524a-27ca-40a0-9599-a3a172b15ecb-kube-api-access-wz857\") pod \"79db524a-27ca-40a0-9599-a3a172b15ecb\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.427291 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79db524a-27ca-40a0-9599-a3a172b15ecb-tls-certs\" (UniqueName: \"kubernetes.io/secret/79db524a-27ca-40a0-9599-a3a172b15ecb-tls-certs\") pod \"79db524a-27ca-40a0-9599-a3a172b15ecb\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.427509 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-home\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-home\") pod \"79db524a-27ca-40a0-9599-a3a172b15ecb\" (UID: \"79db524a-27ca-40a0-9599-a3a172b15ecb\") " Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.428612 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-tmp-dir" pod "79db524a-27ca-40a0-9599-a3a172b15ecb" (UID: "79db524a-27ca-40a0-9599-a3a172b15ecb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.428718 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-model-cache" pod "79db524a-27ca-40a0-9599-a3a172b15ecb" (UID: "79db524a-27ca-40a0-9599-a3a172b15ecb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.429050 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-home" pod "79db524a-27ca-40a0-9599-a3a172b15ecb" (UID: "79db524a-27ca-40a0-9599-a3a172b15ecb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.439141 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.439185 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.439200 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.466318 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq"] Oct 30 05:32:14.597955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.579793 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv"] Oct 30 05:32:14.665638 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:32:14.665526 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5530d72_64d2_48b1_9f51_362f4a41b916.slice/crio-906f10d5f38d6b7a0c1479377ffa208a5728b386335260ea9bdf0b01600ecfda WatchSource:0}: Error finding container 906f10d5f38d6b7a0c1479377ffa208a5728b386335260ea9bdf0b01600ecfda: Status 404 returned error can't find the container with id 906f10d5f38d6b7a0c1479377ffa208a5728b386335260ea9bdf0b01600ecfda Oct 30 05:32:14.690349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.690122 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79db524a-27ca-40a0-9599-a3a172b15ecb-tls-certs" pod "79db524a-27ca-40a0-9599-a3a172b15ecb" (UID: "79db524a-27ca-40a0-9599-a3a172b15ecb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:14.690349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.690218 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-dshm" pod "79db524a-27ca-40a0-9599-a3a172b15ecb" (UID: "79db524a-27ca-40a0-9599-a3a172b15ecb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:14.690669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.690637 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79db524a-27ca-40a0-9599-a3a172b15ecb-kube-api-access-wz857" pod "79db524a-27ca-40a0-9599-a3a172b15ecb" (UID: "79db524a-27ca-40a0-9599-a3a172b15ecb"). InnerVolumeSpecName "kube-api-access-wz857". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:14.715032 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:32:14.714997 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160a7128_f92e_420b_bd00_c44142af02bc.slice/crio-45aed2d972b4dcd12451d7644bdcf532dcfcfd2730277575f657294e37ff7b8e WatchSource:0}: Error finding container 45aed2d972b4dcd12451d7644bdcf532dcfcfd2730277575f657294e37ff7b8e: Status 404 returned error can't find the container with id 45aed2d972b4dcd12451d7644bdcf532dcfcfd2730277575f657294e37ff7b8e Oct 30 05:32:14.742099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.742050 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:14.742337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.742153 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:14.742337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.742202 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:14.742337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.742050 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:14.742337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.742275 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:14.742337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.742318 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:14.773909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.761339 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79db524a-27ca-40a0-9599-a3a172b15ecb-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:14.773909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.761423 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz857\" (UniqueName: \"kubernetes.io/projected/79db524a-27ca-40a0-9599-a3a172b15ecb-kube-api-access-wz857\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:14.773909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:14.761435 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79db524a-27ca-40a0-9599-a3a172b15ecb-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:15.228774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:15.228651 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" event={"ID":"c5530d72-64d2-48b1-9f51-362f4a41b916","Type":"ContainerStarted","Data":"2e57348e3f31ad7d3b65738d60f17caddbb1ed44c802e1e3679ceac58dfca93f"} Oct 30 05:32:15.228774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:15.228722 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" event={"ID":"c5530d72-64d2-48b1-9f51-362f4a41b916","Type":"ContainerStarted","Data":"906f10d5f38d6b7a0c1479377ffa208a5728b386335260ea9bdf0b01600ecfda"} Oct 30 05:32:15.231025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:15.230992 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl" Oct 30 05:32:15.231025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:15.231011 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" event={"ID":"160a7128-f92e-420b-bd00-c44142af02bc","Type":"ContainerStarted","Data":"976450c7c8d070691b9e44338a8fd6c203deb82ef7d38565ae0356a1874679b7"} Oct 30 05:32:15.231254 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:15.231077 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" event={"ID":"160a7128-f92e-420b-bd00-c44142af02bc","Type":"ContainerStarted","Data":"45aed2d972b4dcd12451d7644bdcf532dcfcfd2730277575f657294e37ff7b8e"} Oct 30 05:32:15.339129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:15.339066 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" podStartSLOduration=20.339043932 podStartE2EDuration="20.339043932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:15.291492323 +0000 UTC m=+2859.819259104" watchObservedRunningTime="2026-10-30 05:32:15.339043932 +0000 UTC m=+2859.866810717" Oct 30 05:32:15.340234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:15.340209 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl"] Oct 30 05:32:15.352315 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:15.352283 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0dba31/sub-test-model-0dba31-kserve-6f76b77f95-mxzwl"] Oct 30 05:32:15.474383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:15.450585 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" podStartSLOduration=103.450557762 podStartE2EDuration="1m43.450557762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:15.394346006 +0000 UTC m=+2859.922112793" watchObservedRunningTime="2026-10-30 05:32:15.450557762 +0000 UTC m=+2859.978324606" Oct 30 05:32:15.586290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:15.586254 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79db524a-27ca-40a0-9599-a3a172b15ecb" path="/var/lib/kubelet/pods/79db524a-27ca-40a0-9599-a3a172b15ecb/volumes" Oct 30 05:32:15.796331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:15.796286 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:15.983938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:15.983854 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:16.045108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:16.045027 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq"] Oct 30 05:32:16.055145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:16.055114 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af9c958d-qgdcq"] Oct 30 05:32:16.804829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:16.804799 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:16.986132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:16.986090 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" start-of-body= Oct 30 05:32:16.986320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:16.986148 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" podUID="160a7128-f92e-420b-bd00-c44142af02bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" Oct 30 05:32:17.264143 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:17.263681 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:17.264707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:17.264658 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:32:17.589212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:17.589152 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b888b790-c2be-440a-bc24-b9a7dc7d3509" path="/var/lib/kubelet/pods/b888b790-c2be-440a-bc24-b9a7dc7d3509/volumes" Oct 30 05:32:17.984816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:17.984725 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" start-of-body= Oct 30 05:32:17.984816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:17.984778 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" podUID="160a7128-f92e-420b-bd00-c44142af02bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" Oct 30 05:32:18.993955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:18.990642 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" start-of-body= Oct 30 05:32:18.993955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:18.993446 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" podUID="160a7128-f92e-420b-bd00-c44142af02bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" Oct 30 05:32:20.009014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:19.996558 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" start-of-body= Oct 30 05:32:20.009014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:19.996621 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" podUID="160a7128-f92e-420b-bd00-c44142af02bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" Oct 30 05:32:20.983977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:20.983935 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" start-of-body= Oct 30 05:32:20.984184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:20.983999 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" podUID="160a7128-f92e-420b-bd00-c44142af02bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" Oct 30 05:32:21.984588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:21.984547 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" start-of-body= Oct 30 05:32:21.985063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:21.984606 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" podUID="160a7128-f92e-420b-bd00-c44142af02bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" Oct 30 05:32:21.987101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:21.987078 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:22.469028 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:22.468990 6587 generic.go:356] "Generic (PLEG): container finished" podID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerID="0ac27b7a3b27da22512bbab85a70a89e84cfc05fd0dbce34b7256a6f4fde8346" exitCode=0 Oct 30 05:32:22.469231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:22.469065 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" event={"ID":"c9207797-cae2-4c27-9e1b-bdce5405161c","Type":"ContainerDied","Data":"0ac27b7a3b27da22512bbab85a70a89e84cfc05fd0dbce34b7256a6f4fde8346"} Oct 30 05:32:22.469705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:22.469685 6587 scope.go:122] "RemoveContainer" containerID="0ac27b7a3b27da22512bbab85a70a89e84cfc05fd0dbce34b7256a6f4fde8346" Oct 30 05:32:22.472221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:22.472191 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:22.472946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:22.472277 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:22.472946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:22.472330 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:22.988932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:22.988901 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:22.990074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:22.990052 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:32:23.504337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:23.502916 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" event={"ID":"c9207797-cae2-4c27-9e1b-bdce5405161c","Type":"ContainerStarted","Data":"a31d983a8f270d33976ca394ffa55180771242cbb78e8e543a9390e1a6cf34b8"} Oct 30 05:32:23.938381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:23.932787 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:32:23.951695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:23.950604 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.64:15021/healthz/ready\": dial tcp 10.134.1.64:15021: connect: connection refused" start-of-body= Oct 30 05:32:23.951695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:23.950689 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.64:15021/healthz/ready\": dial tcp 10.134.1.64:15021: connect: connection refused" Oct 30 05:32:24.938039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:24.937489 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.64:15021/healthz/ready\": dial tcp 10.134.1.64:15021: connect: connection refused" start-of-body= Oct 30 05:32:24.938039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:24.937565 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.64:15021/healthz/ready\": dial tcp 10.134.1.64:15021: connect: connection refused" Oct 30 05:32:25.056156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:25.055950 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv"] Oct 30 05:32:25.062139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:25.062108 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785e362b-6gbsv"] Oct 30 05:32:25.585742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:25.585707 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a5d829-ba9b-4533-9ce0-003d04c3e1d5" path="/var/lib/kubelet/pods/b7a5d829-ba9b-4533-9ce0-003d04c3e1d5/volumes" Oct 30 05:32:25.953459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:25.952506 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.64:15021/healthz/ready\": dial tcp 10.134.1.64:15021: connect: connection refused" start-of-body= Oct 30 05:32:25.953459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:25.952578 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.64:15021/healthz/ready\": dial tcp 10.134.1.64:15021: connect: connection refused" Oct 30 05:32:26.103998 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:32:26.103952 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:32:26.940383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:26.935953 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.64:15021/healthz/ready\": dial tcp 10.134.1.64:15021: connect: connection refused" start-of-body= Oct 30 05:32:26.940383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:26.936034 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.64:15021/healthz/ready\": dial tcp 10.134.1.64:15021: connect: connection refused" Oct 30 05:32:27.936250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:27.936211 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.64:15021/healthz/ready\": dial tcp 10.134.1.64:15021: connect: connection refused" start-of-body= Oct 30 05:32:27.936932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:27.936899 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.64:15021/healthz/ready\": dial tcp 10.134.1.64:15021: connect: connection refused" Oct 30 05:32:28.299047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:28.299028 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:32:28.932900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:28.932861 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.64:15021/healthz/ready\": dial tcp 10.134.1.64:15021: connect: connection refused" start-of-body= Oct 30 05:32:28.933284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:28.933249 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.64:15021/healthz/ready\": dial tcp 10.134.1.64:15021: connect: connection refused" Oct 30 05:32:29.296457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:29.295347 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l"] Oct 30 05:32:29.296457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:29.296074 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" podUID="18c027be-26de-4cd2-bfcf-b353ccc68924" containerName="istio-proxy" containerID="cri-o://ea59fe5a92dd7b1a945e041e81920c594f8f0cc3a480310069e921ea44967a76" gracePeriod=30 Oct 30 05:32:30.151869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:30.151837 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:32:30.201380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:30.182660 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:32:34.654788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.654753 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:32:34.759257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.759218 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-certs\") pod \"18c027be-26de-4cd2-bfcf-b353ccc68924\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " Oct 30 05:32:34.759551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.759520 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-podinfo\") pod \"18c027be-26de-4cd2-bfcf-b353ccc68924\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " Oct 30 05:32:34.759692 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.759591 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c027be-26de-4cd2-bfcf-b353ccc68924-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c027be-26de-4cd2-bfcf-b353ccc68924-istiod-ca-cert\") pod \"18c027be-26de-4cd2-bfcf-b353ccc68924\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " Oct 30 05:32:34.759692 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.759648 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-envoy\") pod \"18c027be-26de-4cd2-bfcf-b353ccc68924\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " Oct 30 05:32:34.759821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.759695 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-token\" (UniqueName: \"kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-token\") pod \"18c027be-26de-4cd2-bfcf-b353ccc68924\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " Oct 30 05:32:34.759821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.759766 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-credential-socket\") pod \"18c027be-26de-4cd2-bfcf-b353ccc68924\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " Oct 30 05:32:34.760045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.760023 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-data\") pod \"18c027be-26de-4cd2-bfcf-b353ccc68924\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " Oct 30 05:32:34.760261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.760242 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-kube-api-access-h5pzd\" (UniqueName: \"kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-kube-api-access-h5pzd\") pod \"18c027be-26de-4cd2-bfcf-b353ccc68924\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " Oct 30 05:32:34.760503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.760486 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-socket\") pod \"18c027be-26de-4cd2-bfcf-b353ccc68924\" (UID: \"18c027be-26de-4cd2-bfcf-b353ccc68924\") " Oct 30 05:32:34.761877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.761854 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-socket" pod "18c027be-26de-4cd2-bfcf-b353ccc68924" (UID: "18c027be-26de-4cd2-bfcf-b353ccc68924"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:34.762290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.762269 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-certs" pod "18c027be-26de-4cd2-bfcf-b353ccc68924" (UID: "18c027be-26de-4cd2-bfcf-b353ccc68924"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:34.765866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.765737 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-podinfo" pod "18c027be-26de-4cd2-bfcf-b353ccc68924" (UID: "18c027be-26de-4cd2-bfcf-b353ccc68924"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:32:34.766351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.766330 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c027be-26de-4cd2-bfcf-b353ccc68924-istiod-ca-cert" pod "18c027be-26de-4cd2-bfcf-b353ccc68924" (UID: "18c027be-26de-4cd2-bfcf-b353ccc68924"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:34.768150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.767878 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-credential-socket" pod "18c027be-26de-4cd2-bfcf-b353ccc68924" (UID: "18c027be-26de-4cd2-bfcf-b353ccc68924"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:34.799538 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.799273 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-data" pod "18c027be-26de-4cd2-bfcf-b353ccc68924" (UID: "18c027be-26de-4cd2-bfcf-b353ccc68924"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:34.809484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.809434 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw"] Oct 30 05:32:34.810900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.810857 6587 cpu_manager.go:414] "containerMap: removing container" podUID="18c027be-26de-4cd2-bfcf-b353ccc68924" containerName="istio-proxy" Oct 30 05:32:34.810900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.810887 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c027be-26de-4cd2-bfcf-b353ccc68924" containerName="istio-proxy" Oct 30 05:32:34.812068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.810957 6587 cpu_manager.go:414] "containerMap: removing container" podUID="79db524a-27ca-40a0-9599-a3a172b15ecb" containerName="main" Oct 30 05:32:34.812974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.812939 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-token" pod "18c027be-26de-4cd2-bfcf-b353ccc68924" (UID: "18c027be-26de-4cd2-bfcf-b353ccc68924"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:34.813572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.813542 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-envoy" pod "18c027be-26de-4cd2-bfcf-b353ccc68924" (UID: "18c027be-26de-4cd2-bfcf-b353ccc68924"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:34.826577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.814079 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79db524a-27ca-40a0-9599-a3a172b15ecb" containerName="main" Oct 30 05:32:34.826577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.814448 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-kube-api-access-h5pzd" pod "18c027be-26de-4cd2-bfcf-b353ccc68924" (UID: "18c027be-26de-4cd2-bfcf-b353ccc68924"). InnerVolumeSpecName "kube-api-access-h5pzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:34.826577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.814471 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c027be-26de-4cd2-bfcf-b353ccc68924" containerName="istio-proxy" Oct 30 05:32:34.826577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.814500 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="79db524a-27ca-40a0-9599-a3a172b15ecb" containerName="main" Oct 30 05:32:34.826577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.820289 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw"] Oct 30 05:32:34.826577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.820503 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:34.864676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.864577 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.864676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.864617 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5pzd\" (UniqueName: \"kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-kube-api-access-h5pzd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.864676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.864633 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.864676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.864651 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.864676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.864665 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.864676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.864677 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c027be-26de-4cd2-bfcf-b353ccc68924-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.865120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.864691 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.865120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.864703 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c027be-26de-4cd2-bfcf-b353ccc68924-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.865120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.864715 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c027be-26de-4cd2-bfcf-b353ccc68924-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:32:34.958064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.958028 6587 generic.go:356] "Generic (PLEG): container finished" podID="18c027be-26de-4cd2-bfcf-b353ccc68924" containerID="ea59fe5a92dd7b1a945e041e81920c594f8f0cc3a480310069e921ea44967a76" exitCode=0 Oct 30 05:32:34.958269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.958113 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" Oct 30 05:32:34.958269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.958157 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" event={"ID":"18c027be-26de-4cd2-bfcf-b353ccc68924","Type":"ContainerDied","Data":"ea59fe5a92dd7b1a945e041e81920c594f8f0cc3a480310069e921ea44967a76"} Oct 30 05:32:34.958269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.958244 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l" event={"ID":"18c027be-26de-4cd2-bfcf-b353ccc68924","Type":"ContainerDied","Data":"78c3dc316c1d0dc707024e206239f3bae3f22855d2bc12256ab20542ecb858ee"} Oct 30 05:32:34.959719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.958276 6587 scope.go:122] "RemoveContainer" containerID="ea59fe5a92dd7b1a945e041e81920c594f8f0cc3a480310069e921ea44967a76" Oct 30 05:32:34.965787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.965760 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:34.965889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.965823 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:34.966009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.965986 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:34.966194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.966174 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:34.966260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.966220 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/664e59b0-fa1f-498c-960e-94457fdcf3f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:34.966344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.966328 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf6w6\" (UniqueName: \"kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-kube-api-access-tf6w6\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:34.966476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.966456 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:34.966543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.966486 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:34.966543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.966516 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:34.973653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.972981 6587 scope.go:122] "RemoveContainer" containerID="ea59fe5a92dd7b1a945e041e81920c594f8f0cc3a480310069e921ea44967a76" Oct 30 05:32:34.973653 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:32:34.973288 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea59fe5a92dd7b1a945e041e81920c594f8f0cc3a480310069e921ea44967a76\": container with ID starting with ea59fe5a92dd7b1a945e041e81920c594f8f0cc3a480310069e921ea44967a76 not found: ID does not exist" containerID="ea59fe5a92dd7b1a945e041e81920c594f8f0cc3a480310069e921ea44967a76" Oct 30 05:32:34.973653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:34.973308 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea59fe5a92dd7b1a945e041e81920c594f8f0cc3a480310069e921ea44967a76"} err="failed to get container status \"ea59fe5a92dd7b1a945e041e81920c594f8f0cc3a480310069e921ea44967a76\": rpc error: code = NotFound desc = could not find container \"ea59fe5a92dd7b1a945e041e81920c594f8f0cc3a480310069e921ea44967a76\": container with ID starting with ea59fe5a92dd7b1a945e041e81920c594f8f0cc3a480310069e921ea44967a76 not found: ID does not exist" Oct 30 05:32:35.019599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.019560 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l"] Oct 30 05:32:35.025114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.025082 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l"] Oct 30 05:32:35.068074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.068030 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.068280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.068127 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.068280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.068202 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.068433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.068376 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.068433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.068417 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/664e59b0-fa1f-498c-960e-94457fdcf3f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.069269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.068484 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf6w6\" (UniqueName: \"kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-kube-api-access-tf6w6\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.069269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.068544 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.069269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.068600 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.069269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.068627 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.069269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.068672 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.069269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.068778 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.069269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.069259 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/664e59b0-fa1f-498c-960e-94457fdcf3f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.071161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.069475 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.071161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.069608 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.072140 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.071512 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.072140 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.071628 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.090303 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.090267 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.090751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.090727 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf6w6\" (UniqueName: \"kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-kube-api-access-tf6w6\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.148030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.147939 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:35.487878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.487844 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw"] Oct 30 05:32:35.488801 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:32:35.487890 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664e59b0_fa1f_498c_960e_94457fdcf3f1.slice/crio-b53a0a2dce17cbf62859c065261b6c85f2aada99005a235706f33917a2353b93 WatchSource:0}: Error finding container b53a0a2dce17cbf62859c065261b6c85f2aada99005a235706f33917a2353b93: Status 404 returned error can't find the container with id b53a0a2dce17cbf62859c065261b6c85f2aada99005a235706f33917a2353b93 Oct 30 05:32:35.492285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.492152 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:35.492285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.492236 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:35.492285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.492281 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:32:35.590833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.590801 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c027be-26de-4cd2-bfcf-b353ccc68924" path="/var/lib/kubelet/pods/18c027be-26de-4cd2-bfcf-b353ccc68924/volumes" Oct 30 05:32:35.972117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.972079 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" event={"ID":"664e59b0-fa1f-498c-960e-94457fdcf3f1","Type":"ContainerStarted","Data":"3fff937c00339736e5c0c4041c070b7f4c47d1f1febabe285f6c853c02bbb5c7"} Oct 30 05:32:35.972694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.972134 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" event={"ID":"664e59b0-fa1f-498c-960e-94457fdcf3f1","Type":"ContainerStarted","Data":"b53a0a2dce17cbf62859c065261b6c85f2aada99005a235706f33917a2353b93"} Oct 30 05:32:35.975409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:35.975353 6587 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podb966db7f-5185-48c0-80f4-4584735d9c89"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podb966db7f-5185-48c0-80f4-4584735d9c89] : Timed out while waiting for systemd to remove kubepods-burstable-podb966db7f_5185_48c0_80f4_4584735d9c89.slice" Oct 30 05:32:35.975542 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:32:35.975411 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable podb966db7f-5185-48c0-80f4-4584735d9c89] : unable to destroy cgroup paths for cgroup [kubepods burstable podb966db7f-5185-48c0-80f4-4584735d9c89] : Timed out while waiting for systemd to remove kubepods-burstable-podb966db7f_5185_48c0_80f4_4584735d9c89.slice" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" podUID="b966db7f-5185-48c0-80f4-4584735d9c89" Oct 30 05:32:36.055504 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:36.055434 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" podStartSLOduration=132.055409007 podStartE2EDuration="2m12.055409007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:36.047231703 +0000 UTC m=+2880.574998490" watchObservedRunningTime="2026-10-30 05:32:36.055409007 +0000 UTC m=+2880.583175794" Oct 30 05:32:36.148497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:36.148460 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:36.150299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:36.150267 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 05:32:36.150502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:36.150330 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" podUID="664e59b0-fa1f-498c-960e-94457fdcf3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 05:32:36.982997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:36.982964 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv" Oct 30 05:32:37.048708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:37.048668 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv"] Oct 30 05:32:37.061333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:37.061300 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv"] Oct 30 05:32:37.148621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:37.148587 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 05:32:37.148806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:37.148656 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" podUID="664e59b0-fa1f-498c-960e-94457fdcf3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 05:32:37.682473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:37.682035 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b966db7f-5185-48c0-80f4-4584735d9c89" path="/var/lib/kubelet/pods/b966db7f-5185-48c0-80f4-4584735d9c89/volumes" Oct 30 05:32:38.160388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:38.156695 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 05:32:38.160388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:38.156753 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" podUID="664e59b0-fa1f-498c-960e-94457fdcf3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 05:32:39.159923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:39.152770 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 05:32:39.159923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:39.152835 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" podUID="664e59b0-fa1f-498c-960e-94457fdcf3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 05:32:40.168616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:40.168574 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 05:32:40.169147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:40.168646 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" podUID="664e59b0-fa1f-498c-960e-94457fdcf3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 05:32:41.149091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:41.149050 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 05:32:41.149328 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:41.149119 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" podUID="664e59b0-fa1f-498c-960e-94457fdcf3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 05:32:42.148677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:42.148640 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 05:32:42.149485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:42.148709 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" podUID="664e59b0-fa1f-498c-960e-94457fdcf3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 05:32:42.675396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:42.675347 6587 scope.go:122] "RemoveContainer" containerID="901aa29d8026cc580053f05145635a39295436a52417fd8d84144646b39088fa" Oct 30 05:32:42.698990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:42.698964 6587 scope.go:122] "RemoveContainer" containerID="2517c8ab27190320bc13e0671a92d5603cb8713ed70bb341655c9b85a376e586" Oct 30 05:32:43.195483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:43.195448 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:43.197386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:43.197320 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:43.197886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:43.197868 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:32:55.989979 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:32:55.989898 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:32:59.623198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:59.623158 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h"] Oct 30 05:32:59.623639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:59.623411 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" podUID="82a13271-a884-4256-8ae5-f24d9a61c7c6" containerName="payload-processing" containerID="cri-o://4ec234737db6bd8e9f9ebdb77d66c3f7f04339304092fdb0a7ba36c1ce2ecac5" gracePeriod=30 Oct 30 05:32:59.654555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:59.654521 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj"] Oct 30 05:32:59.654843 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:32:59.654814 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" podUID="6ac662ee-f41a-4ffb-8d99-2e2806ec05db" containerName="payload-pre-processing" containerID="cri-o://3322dea5a4de53119c143ae77795e28ab6a9862de853ab4cf8e4cac244a2fc8a" gracePeriod=30 Oct 30 05:33:00.299888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.299525 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" Oct 30 05:33:00.385081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.383625 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a13271-a884-4256-8ae5-f24d9a61c7c6-kube-api-access-c9lgk\" (UniqueName: \"kubernetes.io/projected/82a13271-a884-4256-8ae5-f24d9a61c7c6-kube-api-access-c9lgk\") pod \"82a13271-a884-4256-8ae5-f24d9a61c7c6\" (UID: \"82a13271-a884-4256-8ae5-f24d9a61c7c6\") " Oct 30 05:33:00.385081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.383819 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a13271-a884-4256-8ae5-f24d9a61c7c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a13271-a884-4256-8ae5-f24d9a61c7c6-plugins-config-volume\") pod \"82a13271-a884-4256-8ae5-f24d9a61c7c6\" (UID: \"82a13271-a884-4256-8ae5-f24d9a61c7c6\") " Oct 30 05:33:00.396890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.396858 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a13271-a884-4256-8ae5-f24d9a61c7c6-plugins-config-volume" pod "82a13271-a884-4256-8ae5-f24d9a61c7c6" (UID: "82a13271-a884-4256-8ae5-f24d9a61c7c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:00.404014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.403982 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a13271-a884-4256-8ae5-f24d9a61c7c6-kube-api-access-c9lgk" pod "82a13271-a884-4256-8ae5-f24d9a61c7c6" (UID: "82a13271-a884-4256-8ae5-f24d9a61c7c6"). InnerVolumeSpecName "kube-api-access-c9lgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:00.429676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.429263 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g"] Oct 30 05:33:00.430800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.430772 6587 cpu_manager.go:414] "containerMap: removing container" podUID="82a13271-a884-4256-8ae5-f24d9a61c7c6" containerName="payload-processing" Oct 30 05:33:00.430906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.430805 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a13271-a884-4256-8ae5-f24d9a61c7c6" containerName="payload-processing" Oct 30 05:33:00.431083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.431062 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a13271-a884-4256-8ae5-f24d9a61c7c6" containerName="payload-processing" Oct 30 05:33:00.444612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.443008 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.481349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.477568 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" Oct 30 05:33:00.483076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.483039 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g"] Oct 30 05:33:00.488080 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.488050 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b5b27e-serving-cert\"" Oct 30 05:33:00.488940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.488225 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-grc2l\"" Oct 30 05:33:00.488940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.488320 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:33:00.488940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.488491 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 05:33:00.488940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.488575 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:33:00.488940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.488587 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee01eed-8cc5-4994-a5fc-7d860679fe86-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.488940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.488661 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spxr4\" (UniqueName: \"kubernetes.io/projected/aee01eed-8cc5-4994-a5fc-7d860679fe86-kube-api-access-spxr4\") pod \"maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.488940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.488814 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-metrics-certs\") pod \"maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.488940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.488897 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.506439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.489282 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9lgk\" (UniqueName: \"kubernetes.io/projected/82a13271-a884-4256-8ae5-f24d9a61c7c6-kube-api-access-c9lgk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:00.506439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.489306 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a13271-a884-4256-8ae5-f24d9a61c7c6-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:00.529346 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.528836 6587 generic.go:356] "Generic (PLEG): container finished" podID="82a13271-a884-4256-8ae5-f24d9a61c7c6" containerID="4ec234737db6bd8e9f9ebdb77d66c3f7f04339304092fdb0a7ba36c1ce2ecac5" exitCode=0 Oct 30 05:33:00.529346 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.528919 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" Oct 30 05:33:00.529346 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.529077 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" event={"ID":"82a13271-a884-4256-8ae5-f24d9a61c7c6","Type":"ContainerDied","Data":"4ec234737db6bd8e9f9ebdb77d66c3f7f04339304092fdb0a7ba36c1ce2ecac5"} Oct 30 05:33:00.529346 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.529115 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h" event={"ID":"82a13271-a884-4256-8ae5-f24d9a61c7c6","Type":"ContainerDied","Data":"5a7090d5e157d5fb5c90098c6a805ab92f38b4398acd73f9011fdd943e46de3c"} Oct 30 05:33:00.529346 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.529136 6587 scope.go:122] "RemoveContainer" containerID="4ec234737db6bd8e9f9ebdb77d66c3f7f04339304092fdb0a7ba36c1ce2ecac5" Oct 30 05:33:00.545738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.534079 6587 generic.go:356] "Generic (PLEG): container finished" podID="6ac662ee-f41a-4ffb-8d99-2e2806ec05db" containerID="3322dea5a4de53119c143ae77795e28ab6a9862de853ab4cf8e4cac244a2fc8a" exitCode=0 Oct 30 05:33:00.545738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.534233 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" event={"ID":"6ac662ee-f41a-4ffb-8d99-2e2806ec05db","Type":"ContainerDied","Data":"3322dea5a4de53119c143ae77795e28ab6a9862de853ab4cf8e4cac244a2fc8a"} Oct 30 05:33:00.545738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.534282 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" event={"ID":"6ac662ee-f41a-4ffb-8d99-2e2806ec05db","Type":"ContainerDied","Data":"c7e912b9108c0c152f24eab35ec3bd29af6e715eab5e5ae9a3a9594be88f5aa0"} Oct 30 05:33:00.545738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.534377 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj" Oct 30 05:33:00.545738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.545665 6587 scope.go:122] "RemoveContainer" containerID="4ec234737db6bd8e9f9ebdb77d66c3f7f04339304092fdb0a7ba36c1ce2ecac5" Oct 30 05:33:00.546190 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:00.546002 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec234737db6bd8e9f9ebdb77d66c3f7f04339304092fdb0a7ba36c1ce2ecac5\": container with ID starting with 4ec234737db6bd8e9f9ebdb77d66c3f7f04339304092fdb0a7ba36c1ce2ecac5 not found: ID does not exist" containerID="4ec234737db6bd8e9f9ebdb77d66c3f7f04339304092fdb0a7ba36c1ce2ecac5" Oct 30 05:33:00.546190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.546039 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec234737db6bd8e9f9ebdb77d66c3f7f04339304092fdb0a7ba36c1ce2ecac5"} err="failed to get container status \"4ec234737db6bd8e9f9ebdb77d66c3f7f04339304092fdb0a7ba36c1ce2ecac5\": rpc error: code = NotFound desc = could not find container \"4ec234737db6bd8e9f9ebdb77d66c3f7f04339304092fdb0a7ba36c1ce2ecac5\": container with ID starting with 4ec234737db6bd8e9f9ebdb77d66c3f7f04339304092fdb0a7ba36c1ce2ecac5 not found: ID does not exist" Oct 30 05:33:00.546190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.546063 6587 scope.go:122] "RemoveContainer" containerID="3322dea5a4de53119c143ae77795e28ab6a9862de853ab4cf8e4cac244a2fc8a" Oct 30 05:33:00.608701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.599549 6587 scope.go:122] "RemoveContainer" containerID="3322dea5a4de53119c143ae77795e28ab6a9862de853ab4cf8e4cac244a2fc8a" Oct 30 05:33:00.608701 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:00.599962 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3322dea5a4de53119c143ae77795e28ab6a9862de853ab4cf8e4cac244a2fc8a\": container with ID starting with 3322dea5a4de53119c143ae77795e28ab6a9862de853ab4cf8e4cac244a2fc8a not found: ID does not exist" containerID="3322dea5a4de53119c143ae77795e28ab6a9862de853ab4cf8e4cac244a2fc8a" Oct 30 05:33:00.608701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.599999 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3322dea5a4de53119c143ae77795e28ab6a9862de853ab4cf8e4cac244a2fc8a"} err="failed to get container status \"3322dea5a4de53119c143ae77795e28ab6a9862de853ab4cf8e4cac244a2fc8a\": rpc error: code = NotFound desc = could not find container \"3322dea5a4de53119c143ae77795e28ab6a9862de853ab4cf8e4cac244a2fc8a\": container with ID starting with 3322dea5a4de53119c143ae77795e28ab6a9862de853ab4cf8e4cac244a2fc8a not found: ID does not exist" Oct 30 05:33:00.633884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.628946 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-plugins-config-volume\") pod \"6ac662ee-f41a-4ffb-8d99-2e2806ec05db\" (UID: \"6ac662ee-f41a-4ffb-8d99-2e2806ec05db\") " Oct 30 05:33:00.633884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.629013 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-kube-api-access-67tmq\" (UniqueName: \"kubernetes.io/projected/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-kube-api-access-67tmq\") pod \"6ac662ee-f41a-4ffb-8d99-2e2806ec05db\" (UID: \"6ac662ee-f41a-4ffb-8d99-2e2806ec05db\") " Oct 30 05:33:00.633884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.629879 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-metrics-certs\") pod \"maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.633884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.629969 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.633884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.630419 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee01eed-8cc5-4994-a5fc-7d860679fe86-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.633884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.630494 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spxr4\" (UniqueName: \"kubernetes.io/projected/aee01eed-8cc5-4994-a5fc-7d860679fe86-kube-api-access-spxr4\") pod \"maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.669549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.664834 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-kube-api-access-67tmq" pod "6ac662ee-f41a-4ffb-8d99-2e2806ec05db" (UID: "6ac662ee-f41a-4ffb-8d99-2e2806ec05db"). InnerVolumeSpecName "kube-api-access-67tmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:00.669549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.666583 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-plugins-config-volume" pod "6ac662ee-f41a-4ffb-8d99-2e2806ec05db" (UID: "6ac662ee-f41a-4ffb-8d99-2e2806ec05db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:00.669549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.669291 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-metrics-certs\") pod \"maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.689996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.671027 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee01eed-8cc5-4994-a5fc-7d860679fe86-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.689996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.686564 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h"] Oct 30 05:33:00.689996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.687747 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spxr4\" (UniqueName: \"kubernetes.io/projected/aee01eed-8cc5-4994-a5fc-7d860679fe86-kube-api-access-spxr4\") pod \"maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.693418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.690080 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.693418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.693226 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h"] Oct 30 05:33:00.732761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.732727 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:00.732761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.732762 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67tmq\" (UniqueName: \"kubernetes.io/projected/6ac662ee-f41a-4ffb-8d99-2e2806ec05db-kube-api-access-67tmq\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:00.803642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.803531 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:00.910237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.909912 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj"] Oct 30 05:33:00.916463 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:00.915517 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj"] Oct 30 05:33:01.157782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:01.157752 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g"] Oct 30 05:33:01.159246 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:33:01.159220 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee01eed_8cc5_4994_a5fc_7d860679fe86.slice/crio-2e374326d1316d97d5233493024856a72587e366f21455952ceb3a244bc2c4cd WatchSource:0}: Error finding container 2e374326d1316d97d5233493024856a72587e366f21455952ceb3a244bc2c4cd: Status 404 returned error can't find the container with id 2e374326d1316d97d5233493024856a72587e366f21455952ceb3a244bc2c4cd Oct 30 05:33:01.161233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:01.161210 6587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:33:01.552425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:01.552344 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" event={"ID":"aee01eed-8cc5-4994-a5fc-7d860679fe86","Type":"ContainerStarted","Data":"2e374326d1316d97d5233493024856a72587e366f21455952ceb3a244bc2c4cd"} Oct 30 05:33:01.585260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:01.585220 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac662ee-f41a-4ffb-8d99-2e2806ec05db" path="/var/lib/kubelet/pods/6ac662ee-f41a-4ffb-8d99-2e2806ec05db/volumes" Oct 30 05:33:01.585763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:01.585747 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a13271-a884-4256-8ae5-f24d9a61c7c6" path="/var/lib/kubelet/pods/82a13271-a884-4256-8ae5-f24d9a61c7c6/volumes" Oct 30 05:33:04.785381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:04.784517 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" event={"ID":"aee01eed-8cc5-4994-a5fc-7d860679fe86","Type":"ContainerStarted","Data":"6920a24e0f4032f2ad768dce636d6386b8a59fc06e2580a8c989e3a17cfb1b6c"} Oct 30 05:33:04.833382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:04.785938 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:04.966384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:04.943565 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" podStartSLOduration=4.998300356 podStartE2EDuration="7.94354549s" totalImagesPullingTime="2.945245134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:04.895339383 +0000 UTC m=+2909.423106171" watchObservedRunningTime="2026-10-30 05:33:04.94354549 +0000 UTC m=+2909.471312277" Oct 30 05:33:05.288377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:05.287420 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" podUID="ae125d46-b3a6-4d61-b977-cecd363489be" containerName="main" probeResult="failure" output="Get \"https://10.134.1.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:05.342774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:05.341458 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" podUID="8e74f7bb-be64-439c-9991-9050a769b5cf" containerName="main" probeResult="failure" output="Get \"https://10.134.1.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:05.342774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:05.342026 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:05.618768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:05.606526 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:05.793500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:05.793452 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:33:05.811495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:05.811443 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:33:05.811718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:05.808464 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:05.826393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:05.811990 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:05.896439 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:05.845720 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:06.089382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.077449 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" podUID="8e74f7bb-be64-439c-9991-9050a769b5cf" containerName="main" probeResult="failure" output="Get \"https://10.134.1.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:06.290218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.290108 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" podUID="ae125d46-b3a6-4d61-b977-cecd363489be" containerName="main" probeResult="failure" output="Get \"https://10.134.1.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:06.295462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.294670 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:33:06.350599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.350554 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:06.352122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.352099 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:33:06.763392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.754420 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 30 05:33:06.847550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.844414 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:06.873873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.873827 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:33:06.913387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.881598 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:06.943147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.881652 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:06.943147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.881672 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:33:06.943147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.881691 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:33:06.943147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.881713 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:33:06.943147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.909513 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:33:06.943147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.928427 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:33:06.943147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.928473 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:33:06.943147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.928509 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:33:06.943147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:06.928545 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:33:07.253619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.253582 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7"] Oct 30 05:33:07.311433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.311390 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 05:33:07.312932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.312898 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" podUID="ae125d46-b3a6-4d61-b977-cecd363489be" containerName="main" probeResult="failure" output="Get \"https://10.134.1.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:07.381379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.359940 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:07.453379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.453000 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:07.720426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.703169 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" containerID="cri-o://227cbc21a0b522db94825bb9cb7c2a70cb860a54e774733c6726924aa11dd51b" gracePeriod=30 Oct 30 05:33:07.850383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.799187 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 05:33:07.855502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.855462 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:07.871721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.871672 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" podUID="ae125d46-b3a6-4d61-b977-cecd363489be" containerName="main" probeResult="failure" output="Get \"https://10.134.1.63:8000/health\": context deadline exceeded" Oct 30 05:33:07.895690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.894489 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:07.946886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.946846 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" podUID="e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded" Oct 30 05:33:07.947823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.947792 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:33:07.949419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.948547 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": context deadline exceeded" Oct 30 05:33:07.950298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.948606 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:07.958918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.948626 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 05:33:07.969486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:07.968856 6587 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:08.130380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.109661 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" podUID="8e74f7bb-be64-439c-9991-9050a769b5cf" containerName="main" probeResult="failure" output="Get \"https://10.134.1.62:8000/ready\": context deadline exceeded" Oct 30 05:33:08.186380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.113226 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:33:08.186380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.176707 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:33:08.338436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.338385 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" podUID="ae125d46-b3a6-4d61-b977-cecd363489be" containerName="main" probeResult="failure" output="Get \"https://10.134.1.63:8000/ready\": context deadline exceeded" Oct 30 05:33:08.410440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.406673 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:33:08.430608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.430426 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9"] Oct 30 05:33:08.468232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.448080 6587 cpu_manager.go:414] "containerMap: removing container" podUID="6ac662ee-f41a-4ffb-8d99-2e2806ec05db" containerName="payload-pre-processing" Oct 30 05:33:08.497161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.497125 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac662ee-f41a-4ffb-8d99-2e2806ec05db" containerName="payload-pre-processing" Oct 30 05:33:08.498148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.497921 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac662ee-f41a-4ffb-8d99-2e2806ec05db" containerName="payload-pre-processing" Oct 30 05:33:08.517682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.514472 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9"] Oct 30 05:33:08.517682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.514556 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:33:08.517682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.514596 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:33:08.517682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.514631 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:33:08.517682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.514662 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59" Oct 30 05:33:08.517682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.514693 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:33:08.517682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.514746 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:33:08.517682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.514780 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:33:08.517682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.514828 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:33:08.517682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.514931 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" Oct 30 05:33:08.517682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.516767 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:33:08.543459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.543435 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:33:08.543766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.543711 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:33:08.600585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.600545 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9\" (UID: \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" Oct 30 05:33:08.600748 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.600709 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql2hw\" (UniqueName: \"kubernetes.io/projected/eaa5728c-ac1b-47b6-a04c-e9423037ab25-kube-api-access-ql2hw\") pod \"maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9\" (UID: \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" Oct 30 05:33:08.601478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.600944 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9\" (UID: \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" Oct 30 05:33:08.703109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.703017 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql2hw\" (UniqueName: \"kubernetes.io/projected/eaa5728c-ac1b-47b6-a04c-e9423037ab25-kube-api-access-ql2hw\") pod \"maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9\" (UID: \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" Oct 30 05:33:08.703675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.703643 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9\" (UID: \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" Oct 30 05:33:08.703964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.703941 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9\" (UID: \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" Oct 30 05:33:08.704863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.704842 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9\" (UID: \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" Oct 30 05:33:08.722146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.722093 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9\" (UID: \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" Oct 30 05:33:08.728453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.728410 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql2hw\" (UniqueName: \"kubernetes.io/projected/eaa5728c-ac1b-47b6-a04c-e9423037ab25-kube-api-access-ql2hw\") pod \"maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9\" (UID: \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" Oct 30 05:33:08.846839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:08.846802 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" Oct 30 05:33:09.257481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:09.257451 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9"] Oct 30 05:33:09.258227 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:33:09.258193 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa5728c_ac1b_47b6_a04c_e9423037ab25.slice/crio-5f86ef3a455e963021fd778b44eec492aab6ff9fa620208a1273eda1c65d2f65 WatchSource:0}: Error finding container 5f86ef3a455e963021fd778b44eec492aab6ff9fa620208a1273eda1c65d2f65: Status 404 returned error can't find the container with id 5f86ef3a455e963021fd778b44eec492aab6ff9fa620208a1273eda1c65d2f65 Oct 30 05:33:09.707045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:09.707011 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" event={"ID":"eaa5728c-ac1b-47b6-a04c-e9423037ab25","Type":"ContainerStarted","Data":"5f86ef3a455e963021fd778b44eec492aab6ff9fa620208a1273eda1c65d2f65"} Oct 30 05:33:10.719921 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:10.719827 6587 generic.go:356] "Generic (PLEG): container finished" podID="eaa5728c-ac1b-47b6-a04c-e9423037ab25" containerID="2ee235afc90c614026b92f2464e128ecd2e053d7d3dfe4e43e513eb99148fc76" exitCode=0 Oct 30 05:33:10.719921 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:10.719898 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" event={"ID":"eaa5728c-ac1b-47b6-a04c-e9423037ab25","Type":"ContainerDied","Data":"2ee235afc90c614026b92f2464e128ecd2e053d7d3dfe4e43e513eb99148fc76"} Oct 30 05:33:10.854808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:10.854774 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm"] Oct 30 05:33:10.855030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:10.855003 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" containerID="cri-o://4ff405235d16322288fb6bc325830e7fc88a00682afe4462c15cebe05bc130fc" gracePeriod=30 Oct 30 05:33:10.861032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:10.861007 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd"] Oct 30 05:33:10.861234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:10.861211 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" podUID="fdb49b21-bb35-4f43-8ddd-2a8104a04509" containerName="istio-proxy" containerID="cri-o://dc48d6602fca7c1aaa44b533a74741ca899c766c1e1fde737603d731f8456539" gracePeriod=30 Oct 30 05:33:10.915352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:10.915303 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2"] Oct 30 05:33:10.915982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:10.915686 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" podUID="cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" containerName="istio-proxy" containerID="cri-o://551e859e4d30609a03548d99aa818ce0969bed5b0ef1ac984b7232c56937a361" gracePeriod=30 Oct 30 05:33:11.354185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.354151 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c"] Oct 30 05:33:11.364768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.364742 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" Oct 30 05:33:11.395546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.395380 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c"] Oct 30 05:33:11.513331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.513297 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c\" (UID: \"6678a054-bb47-4483-8ba3-5222da9e56e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" Oct 30 05:33:11.513537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.513476 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v4n7\" (UniqueName: \"kubernetes.io/projected/6678a054-bb47-4483-8ba3-5222da9e56e1-kube-api-access-2v4n7\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c\" (UID: \"6678a054-bb47-4483-8ba3-5222da9e56e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" Oct 30 05:33:11.513652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.513640 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c\" (UID: \"6678a054-bb47-4483-8ba3-5222da9e56e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" Oct 30 05:33:11.615429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.615332 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v4n7\" (UniqueName: \"kubernetes.io/projected/6678a054-bb47-4483-8ba3-5222da9e56e1-kube-api-access-2v4n7\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c\" (UID: \"6678a054-bb47-4483-8ba3-5222da9e56e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" Oct 30 05:33:11.616652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.615579 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c\" (UID: \"6678a054-bb47-4483-8ba3-5222da9e56e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" Oct 30 05:33:11.616652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.615824 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c\" (UID: \"6678a054-bb47-4483-8ba3-5222da9e56e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" Oct 30 05:33:11.616791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.616668 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c\" (UID: \"6678a054-bb47-4483-8ba3-5222da9e56e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" Oct 30 05:33:11.620053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.620009 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c\" (UID: \"6678a054-bb47-4483-8ba3-5222da9e56e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" Oct 30 05:33:11.640587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.640554 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v4n7\" (UniqueName: \"kubernetes.io/projected/6678a054-bb47-4483-8ba3-5222da9e56e1-kube-api-access-2v4n7\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c\" (UID: \"6678a054-bb47-4483-8ba3-5222da9e56e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" Oct 30 05:33:11.703678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.703644 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" Oct 30 05:33:11.990500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.990472 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" Oct 30 05:33:11.994213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:11.994189 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj"] Oct 30 05:33:12.024010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.023978 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-xks-ca\") pod \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\" (UID: \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\") " Oct 30 05:33:12.024333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.024308 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-openshift-ca\") pod \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\" (UID: \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\") " Oct 30 05:33:12.024480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.024405 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa5728c-ac1b-47b6-a04c-e9423037ab25-kube-api-access-ql2hw\" (UniqueName: \"kubernetes.io/projected/eaa5728c-ac1b-47b6-a04c-e9423037ab25-kube-api-access-ql2hw\") pod \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\" (UID: \"eaa5728c-ac1b-47b6-a04c-e9423037ab25\") " Oct 30 05:33:12.025193 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.024701 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-openshift-ca" pod "eaa5728c-ac1b-47b6-a04c-e9423037ab25" (UID: "eaa5728c-ac1b-47b6-a04c-e9423037ab25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:12.025295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.025205 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:12.026841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.026818 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-xks-ca" pod "eaa5728c-ac1b-47b6-a04c-e9423037ab25" (UID: "eaa5728c-ac1b-47b6-a04c-e9423037ab25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:12.027496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.027475 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa5728c-ac1b-47b6-a04c-e9423037ab25-kube-api-access-ql2hw" pod "eaa5728c-ac1b-47b6-a04c-e9423037ab25" (UID: "eaa5728c-ac1b-47b6-a04c-e9423037ab25"). InnerVolumeSpecName "kube-api-access-ql2hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:12.064676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.064646 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c"] Oct 30 05:33:12.085925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.085887 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg"] Oct 30 05:33:12.114790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.114756 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e19f2ea-cnnqg"] Oct 30 05:33:12.127418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.127210 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa5728c-ac1b-47b6-a04c-e9423037ab25-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:12.127418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.127245 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql2hw\" (UniqueName: \"kubernetes.io/projected/eaa5728c-ac1b-47b6-a04c-e9423037ab25-kube-api-access-ql2hw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:12.376475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.376443 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6"] Oct 30 05:33:12.377790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.377769 6587 cpu_manager.go:414] "containerMap: removing container" podUID="eaa5728c-ac1b-47b6-a04c-e9423037ab25" containerName="revoke-keys" Oct 30 05:33:12.377790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.377791 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa5728c-ac1b-47b6-a04c-e9423037ab25" containerName="revoke-keys" Oct 30 05:33:12.378628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.377941 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa5728c-ac1b-47b6-a04c-e9423037ab25" containerName="revoke-keys" Oct 30 05:33:12.383758 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.383734 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" Oct 30 05:33:12.403980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.403910 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6"] Oct 30 05:33:12.432805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.432770 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5873f3d-g75f6\" (UID: \"126f3871-104a-4cd8-99a2-00e114b38ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" Oct 30 05:33:12.433079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.433051 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfxnd\" (UniqueName: \"kubernetes.io/projected/126f3871-104a-4cd8-99a2-00e114b38ed5-kube-api-access-nfxnd\") pod \"maas-api-revoke-sub-c5873f3d-g75f6\" (UID: \"126f3871-104a-4cd8-99a2-00e114b38ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" Oct 30 05:33:12.433920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.433096 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5873f3d-g75f6\" (UID: \"126f3871-104a-4cd8-99a2-00e114b38ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" Oct 30 05:33:12.535229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.535194 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5873f3d-g75f6\" (UID: \"126f3871-104a-4cd8-99a2-00e114b38ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" Oct 30 05:33:12.535434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.535375 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfxnd\" (UniqueName: \"kubernetes.io/projected/126f3871-104a-4cd8-99a2-00e114b38ed5-kube-api-access-nfxnd\") pod \"maas-api-revoke-sub-c5873f3d-g75f6\" (UID: \"126f3871-104a-4cd8-99a2-00e114b38ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" Oct 30 05:33:12.535434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.535411 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5873f3d-g75f6\" (UID: \"126f3871-104a-4cd8-99a2-00e114b38ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" Oct 30 05:33:12.536395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.536327 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5873f3d-g75f6\" (UID: \"126f3871-104a-4cd8-99a2-00e114b38ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" Oct 30 05:33:12.538325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.538304 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5873f3d-g75f6\" (UID: \"126f3871-104a-4cd8-99a2-00e114b38ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" Oct 30 05:33:12.557904 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.557879 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfxnd\" (UniqueName: \"kubernetes.io/projected/126f3871-104a-4cd8-99a2-00e114b38ed5-kube-api-access-nfxnd\") pod \"maas-api-revoke-sub-c5873f3d-g75f6\" (UID: \"126f3871-104a-4cd8-99a2-00e114b38ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" Oct 30 05:33:12.684322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.684288 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:33:12.684498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.684349 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" podUID="cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:33:12.702586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.702555 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" Oct 30 05:33:12.710308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.710283 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:33:12.749880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.749843 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" Oct 30 05:33:12.750734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.749855 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9" event={"ID":"eaa5728c-ac1b-47b6-a04c-e9423037ab25","Type":"ContainerDied","Data":"5f86ef3a455e963021fd778b44eec492aab6ff9fa620208a1273eda1c65d2f65"} Oct 30 05:33:12.750734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.750001 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f86ef3a455e963021fd778b44eec492aab6ff9fa620208a1273eda1c65d2f65" Oct 30 05:33:12.759392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.757149 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" event={"ID":"6678a054-bb47-4483-8ba3-5222da9e56e1","Type":"ContainerStarted","Data":"6a6b29c328554169a39e679970bf56bedc577a1549c787b54f8991f6b712f03b"} Oct 30 05:33:12.759392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.757234 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" event={"ID":"6678a054-bb47-4483-8ba3-5222da9e56e1","Type":"ContainerStarted","Data":"bd7eb18a3633edeb9c6e3599885f74ba236a5782d0e911ace4654f618b00e1ee"} Oct 30 05:33:12.819829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.819500 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" podStartSLOduration=1.819476371 podStartE2EDuration="1.819476371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:12.806262749 +0000 UTC m=+2917.334029537" watchObservedRunningTime="2026-10-30 05:33:12.819476371 +0000 UTC m=+2917.347243159" Oct 30 05:33:12.971231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.970728 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:33:12.971231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:12.970788 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:33:13.053293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.053261 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6"] Oct 30 05:33:13.479141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.479113 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:33:13.547503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.547379 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk"] Oct 30 05:33:13.548859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.548829 6587 cpu_manager.go:414] "containerMap: removing container" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" Oct 30 05:33:13.548859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.548857 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" Oct 30 05:33:13.549089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.548889 6587 cpu_manager.go:414] "containerMap: removing container" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" Oct 30 05:33:13.549089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.548898 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" Oct 30 05:33:13.549190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.549164 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" Oct 30 05:33:13.549247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.549199 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerName="istio-proxy" Oct 30 05:33:13.554113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.554083 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.574751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.574716 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ec1091-serving-cert\"" Oct 30 05:33:13.664390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.663734 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027c5a3b-eabe-4622-a1e2-7294bfe4edda" path="/var/lib/kubelet/pods/027c5a3b-eabe-4622-a1e2-7294bfe4edda/volumes" Oct 30 05:33:13.666176 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.666145 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-credential-socket\") pod \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " Oct 30 05:33:13.667229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.667208 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istiod-ca-cert\") pod \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " Oct 30 05:33:13.667849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.667830 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-envoy\") pod \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " Oct 30 05:33:13.668343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.668325 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-token\" (UniqueName: \"kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-token\") pod \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " Oct 30 05:33:13.669662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.669642 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-data\") pod \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " Oct 30 05:33:13.670040 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.667137 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-credential-socket" pod "50a8b4fd-b32c-4097-bdeb-80afe0c5541b" (UID: "50a8b4fd-b32c-4097-bdeb-80afe0c5541b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:13.670182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.667738 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istiod-ca-cert" pod "50a8b4fd-b32c-4097-bdeb-80afe0c5541b" (UID: "50a8b4fd-b32c-4097-bdeb-80afe0c5541b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:13.670182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.669922 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-data" pod "50a8b4fd-b32c-4097-bdeb-80afe0c5541b" (UID: "50a8b4fd-b32c-4097-bdeb-80afe0c5541b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:13.670346 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.670327 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-podinfo\") pod \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " Oct 30 05:33:13.670554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.670534 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-kube-api-access-8skhn\" (UniqueName: \"kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-kube-api-access-8skhn\") pod \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " Oct 30 05:33:13.670807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.670788 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-socket\") pod \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " Oct 30 05:33:13.672089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.672072 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-certs\") pod \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\" (UID: \"50a8b4fd-b32c-4097-bdeb-80afe0c5541b\") " Oct 30 05:33:13.673425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.671312 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk"] Oct 30 05:33:13.673425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.671176 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-envoy" pod "50a8b4fd-b32c-4097-bdeb-80afe0c5541b" (UID: "50a8b4fd-b32c-4097-bdeb-80afe0c5541b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:13.673425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.671952 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-socket" pod "50a8b4fd-b32c-4097-bdeb-80afe0c5541b" (UID: "50a8b4fd-b32c-4097-bdeb-80afe0c5541b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:13.673425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.672428 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-certs" pod "50a8b4fd-b32c-4097-bdeb-80afe0c5541b" (UID: "50a8b4fd-b32c-4097-bdeb-80afe0c5541b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:13.673425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.673044 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-token" pod "50a8b4fd-b32c-4097-bdeb-80afe0c5541b" (UID: "50a8b4fd-b32c-4097-bdeb-80afe0c5541b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.673529 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-metrics-certs\") pod \"maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.673582 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdq6z\" (UniqueName: \"kubernetes.io/projected/13378ef4-ea56-4f17-8536-5ace5c724717-kube-api-access-hdq6z\") pod \"maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.673895 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.673960 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-kube-api-access-8skhn" pod "50a8b4fd-b32c-4097-bdeb-80afe0c5541b" (UID: "50a8b4fd-b32c-4097-bdeb-80afe0c5541b"). InnerVolumeSpecName "kube-api-access-8skhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.674048 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13378ef4-ea56-4f17-8536-5ace5c724717-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.674236 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.674257 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.674271 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.674284 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.674300 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.674312 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.674325 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.674337 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8skhn\" (UniqueName: \"kubernetes.io/projected/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-kube-api-access-8skhn\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:13.677513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.676214 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-podinfo" pod "50a8b4fd-b32c-4097-bdeb-80afe0c5541b" (UID: "50a8b4fd-b32c-4097-bdeb-80afe0c5541b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:33:13.768904 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.768864 6587 generic.go:356] "Generic (PLEG): container finished" podID="6678a054-bb47-4483-8ba3-5222da9e56e1" containerID="6a6b29c328554169a39e679970bf56bedc577a1549c787b54f8991f6b712f03b" exitCode=0 Oct 30 05:33:13.769163 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.768928 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" event={"ID":"6678a054-bb47-4483-8ba3-5222da9e56e1","Type":"ContainerDied","Data":"6a6b29c328554169a39e679970bf56bedc577a1549c787b54f8991f6b712f03b"} Oct 30 05:33:13.771247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.771221 6587 generic.go:356] "Generic (PLEG): container finished" podID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" containerID="227cbc21a0b522db94825bb9cb7c2a70cb860a54e774733c6726924aa11dd51b" exitCode=0 Oct 30 05:33:13.771415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.771276 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" Oct 30 05:33:13.771415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.771300 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" event={"ID":"50a8b4fd-b32c-4097-bdeb-80afe0c5541b","Type":"ContainerDied","Data":"227cbc21a0b522db94825bb9cb7c2a70cb860a54e774733c6726924aa11dd51b"} Oct 30 05:33:13.771415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.771341 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7" event={"ID":"50a8b4fd-b32c-4097-bdeb-80afe0c5541b","Type":"ContainerDied","Data":"91d303bb96b27d261ef4b7bcfeeec14f38bd77fb127f89e0854ad888871ff53f"} Oct 30 05:33:13.771415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.771377 6587 scope.go:122] "RemoveContainer" containerID="227cbc21a0b522db94825bb9cb7c2a70cb860a54e774733c6726924aa11dd51b" Oct 30 05:33:13.773578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.773555 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" event={"ID":"126f3871-104a-4cd8-99a2-00e114b38ed5","Type":"ContainerStarted","Data":"0f6ff9dada6f55fa0bf3f60769f80c0e8d219111303984ca878de3188f8bebaf"} Oct 30 05:33:13.773659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.773601 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" event={"ID":"126f3871-104a-4cd8-99a2-00e114b38ed5","Type":"ContainerStarted","Data":"543139616a6d695b10eae8b8714eba04603ee6fbccb03fae5bcc3894c5f3016c"} Oct 30 05:33:13.775580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.775553 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.775714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.775682 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13378ef4-ea56-4f17-8536-5ace5c724717-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.775899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.775873 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-metrics-certs\") pod \"maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.776406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.775921 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdq6z\" (UniqueName: \"kubernetes.io/projected/13378ef4-ea56-4f17-8536-5ace5c724717-kube-api-access-hdq6z\") pod \"maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.776406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.776231 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a8b4fd-b32c-4097-bdeb-80afe0c5541b-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:13.778019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.777986 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13378ef4-ea56-4f17-8536-5ace5c724717-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.778631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.778609 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.780294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.780274 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-metrics-certs\") pod \"maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.788501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.788479 6587 scope.go:122] "RemoveContainer" containerID="267dcf322ae60df01bc8acb9112bb2035f3d3a7546a56af2b4c2fb6617af97a1" Oct 30 05:33:13.798165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.798141 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdq6z\" (UniqueName: \"kubernetes.io/projected/13378ef4-ea56-4f17-8536-5ace5c724717-kube-api-access-hdq6z\") pod \"maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.821980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.821951 6587 scope.go:122] "RemoveContainer" containerID="227cbc21a0b522db94825bb9cb7c2a70cb860a54e774733c6726924aa11dd51b" Oct 30 05:33:13.823152 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:13.822416 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227cbc21a0b522db94825bb9cb7c2a70cb860a54e774733c6726924aa11dd51b\": container with ID starting with 227cbc21a0b522db94825bb9cb7c2a70cb860a54e774733c6726924aa11dd51b not found: ID does not exist" containerID="227cbc21a0b522db94825bb9cb7c2a70cb860a54e774733c6726924aa11dd51b" Oct 30 05:33:13.823152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.822497 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227cbc21a0b522db94825bb9cb7c2a70cb860a54e774733c6726924aa11dd51b"} err="failed to get container status \"227cbc21a0b522db94825bb9cb7c2a70cb860a54e774733c6726924aa11dd51b\": rpc error: code = NotFound desc = could not find container \"227cbc21a0b522db94825bb9cb7c2a70cb860a54e774733c6726924aa11dd51b\": container with ID starting with 227cbc21a0b522db94825bb9cb7c2a70cb860a54e774733c6726924aa11dd51b not found: ID does not exist" Oct 30 05:33:13.823152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.822523 6587 scope.go:122] "RemoveContainer" containerID="267dcf322ae60df01bc8acb9112bb2035f3d3a7546a56af2b4c2fb6617af97a1" Oct 30 05:33:13.823152 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:13.823033 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267dcf322ae60df01bc8acb9112bb2035f3d3a7546a56af2b4c2fb6617af97a1\": container with ID starting with 267dcf322ae60df01bc8acb9112bb2035f3d3a7546a56af2b4c2fb6617af97a1 not found: ID does not exist" containerID="267dcf322ae60df01bc8acb9112bb2035f3d3a7546a56af2b4c2fb6617af97a1" Oct 30 05:33:13.823152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.823067 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267dcf322ae60df01bc8acb9112bb2035f3d3a7546a56af2b4c2fb6617af97a1"} err="failed to get container status \"267dcf322ae60df01bc8acb9112bb2035f3d3a7546a56af2b4c2fb6617af97a1\": rpc error: code = NotFound desc = could not find container \"267dcf322ae60df01bc8acb9112bb2035f3d3a7546a56af2b4c2fb6617af97a1\": container with ID starting with 267dcf322ae60df01bc8acb9112bb2035f3d3a7546a56af2b4c2fb6617af97a1 not found: ID does not exist" Oct 30 05:33:13.879959 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.879855 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" podStartSLOduration=1.879834487 podStartE2EDuration="1.879834487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:13.875962128 +0000 UTC m=+2918.403728914" watchObservedRunningTime="2026-10-30 05:33:13.879834487 +0000 UTC m=+2918.407601276" Oct 30 05:33:13.895027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.894997 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:13.902648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.902615 6587 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:33:13.902829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.902671 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" podUID="fdb49b21-bb35-4f43-8ddd-2a8104a04509" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:33:13.923032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.922954 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7"] Oct 30 05:33:13.928395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:13.928368 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7"] Oct 30 05:33:14.222385 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:33:14.222336 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13378ef4_ea56_4f17_8536_5ace5c724717.slice/crio-2b7c9bf47dc8d1d9aaa646a1af6aaca80e35d616e0a20ec32ead83016c1f5131 WatchSource:0}: Error finding container 2b7c9bf47dc8d1d9aaa646a1af6aaca80e35d616e0a20ec32ead83016c1f5131: Status 404 returned error can't find the container with id 2b7c9bf47dc8d1d9aaa646a1af6aaca80e35d616e0a20ec32ead83016c1f5131 Oct 30 05:33:14.223883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:14.223860 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk"] Oct 30 05:33:14.791402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:14.791342 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" event={"ID":"13378ef4-ea56-4f17-8536-5ace5c724717","Type":"ContainerStarted","Data":"2b7c9bf47dc8d1d9aaa646a1af6aaca80e35d616e0a20ec32ead83016c1f5131"} Oct 30 05:33:14.794390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:14.793578 6587 generic.go:356] "Generic (PLEG): container finished" podID="126f3871-104a-4cd8-99a2-00e114b38ed5" containerID="0f6ff9dada6f55fa0bf3f60769f80c0e8d219111303984ca878de3188f8bebaf" exitCode=0 Oct 30 05:33:14.794390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:14.794319 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" event={"ID":"126f3871-104a-4cd8-99a2-00e114b38ed5","Type":"ContainerDied","Data":"0f6ff9dada6f55fa0bf3f60769f80c0e8d219111303984ca878de3188f8bebaf"} Oct 30 05:33:15.066852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.066821 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" Oct 30 05:33:15.131624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.131585 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-xks-ca\") pod \"6678a054-bb47-4483-8ba3-5222da9e56e1\" (UID: \"6678a054-bb47-4483-8ba3-5222da9e56e1\") " Oct 30 05:33:15.132706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.131645 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6678a054-bb47-4483-8ba3-5222da9e56e1-kube-api-access-2v4n7\" (UniqueName: \"kubernetes.io/projected/6678a054-bb47-4483-8ba3-5222da9e56e1-kube-api-access-2v4n7\") pod \"6678a054-bb47-4483-8ba3-5222da9e56e1\" (UID: \"6678a054-bb47-4483-8ba3-5222da9e56e1\") " Oct 30 05:33:15.132706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.132002 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-openshift-ca\") pod \"6678a054-bb47-4483-8ba3-5222da9e56e1\" (UID: \"6678a054-bb47-4483-8ba3-5222da9e56e1\") " Oct 30 05:33:15.133307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.133277 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-openshift-ca" pod "6678a054-bb47-4483-8ba3-5222da9e56e1" (UID: "6678a054-bb47-4483-8ba3-5222da9e56e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:15.135922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.135894 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-xks-ca" pod "6678a054-bb47-4483-8ba3-5222da9e56e1" (UID: "6678a054-bb47-4483-8ba3-5222da9e56e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:15.147577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.147546 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6678a054-bb47-4483-8ba3-5222da9e56e1-kube-api-access-2v4n7" pod "6678a054-bb47-4483-8ba3-5222da9e56e1" (UID: "6678a054-bb47-4483-8ba3-5222da9e56e1"). InnerVolumeSpecName "kube-api-access-2v4n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:15.234973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.234941 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:15.234973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.234973 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6678a054-bb47-4483-8ba3-5222da9e56e1-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:15.235444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.234986 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v4n7\" (UniqueName: \"kubernetes.io/projected/6678a054-bb47-4483-8ba3-5222da9e56e1-kube-api-access-2v4n7\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:15.587202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.587166 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a8b4fd-b32c-4097-bdeb-80afe0c5541b" path="/var/lib/kubelet/pods/50a8b4fd-b32c-4097-bdeb-80afe0c5541b/volumes" Oct 30 05:33:15.805952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.805921 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" Oct 30 05:33:15.806210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.805952 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c" event={"ID":"6678a054-bb47-4483-8ba3-5222da9e56e1","Type":"ContainerDied","Data":"bd7eb18a3633edeb9c6e3599885f74ba236a5782d0e911ace4654f618b00e1ee"} Oct 30 05:33:15.806210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.806014 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd7eb18a3633edeb9c6e3599885f74ba236a5782d0e911ace4654f618b00e1ee" Oct 30 05:33:15.807988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.807965 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" event={"ID":"13378ef4-ea56-4f17-8536-5ace5c724717","Type":"ContainerStarted","Data":"51e6a204c79d07b8a56c94c10985e07094685a82d32ea490a1fd416ce840dc6f"} Oct 30 05:33:15.808966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:15.808160 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:16.184781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.184401 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" Oct 30 05:33:16.231215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.231145 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" podStartSLOduration=3.793607176 podStartE2EDuration="4.231123888s" totalImagesPullingTime="437.516712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:15.870709017 +0000 UTC m=+2920.398475803" watchObservedRunningTime="2026-10-30 05:33:16.231123888 +0000 UTC m=+2920.758890675" Oct 30 05:33:16.293599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.293059 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:33:16.302586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.302546 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126f3871-104a-4cd8-99a2-00e114b38ed5-kube-api-access-nfxnd\" (UniqueName: \"kubernetes.io/projected/126f3871-104a-4cd8-99a2-00e114b38ed5-kube-api-access-nfxnd\") pod \"126f3871-104a-4cd8-99a2-00e114b38ed5\" (UID: \"126f3871-104a-4cd8-99a2-00e114b38ed5\") " Oct 30 05:33:16.302743 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.302661 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-openshift-ca\") pod \"126f3871-104a-4cd8-99a2-00e114b38ed5\" (UID: \"126f3871-104a-4cd8-99a2-00e114b38ed5\") " Oct 30 05:33:16.303017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.302994 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-xks-ca\") pod \"126f3871-104a-4cd8-99a2-00e114b38ed5\" (UID: \"126f3871-104a-4cd8-99a2-00e114b38ed5\") " Oct 30 05:33:16.304988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.304958 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-openshift-ca" pod "126f3871-104a-4cd8-99a2-00e114b38ed5" (UID: "126f3871-104a-4cd8-99a2-00e114b38ed5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:16.307754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.307724 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-xks-ca" pod "126f3871-104a-4cd8-99a2-00e114b38ed5" (UID: "126f3871-104a-4cd8-99a2-00e114b38ed5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:16.318994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.318964 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126f3871-104a-4cd8-99a2-00e114b38ed5-kube-api-access-nfxnd" pod "126f3871-104a-4cd8-99a2-00e114b38ed5" (UID: "126f3871-104a-4cd8-99a2-00e114b38ed5"). InnerVolumeSpecName "kube-api-access-nfxnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:16.398826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.398782 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:33:16.405095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.405062 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f131a03e-c992-40b1-a6e6-86d29073205b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f131a03e-c992-40b1-a6e6-86d29073205b-istio-podinfo\") pod \"f131a03e-c992-40b1-a6e6-86d29073205b\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " Oct 30 05:33:16.405292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.405178 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-certs\") pod \"f131a03e-c992-40b1-a6e6-86d29073205b\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " Oct 30 05:33:16.405970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.405322 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-kube-api-access-5rgqp\" (UniqueName: \"kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-kube-api-access-5rgqp\") pod \"f131a03e-c992-40b1-a6e6-86d29073205b\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " Oct 30 05:33:16.405970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.405510 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-credential-socket\") pod \"f131a03e-c992-40b1-a6e6-86d29073205b\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " Oct 30 05:33:16.405970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.405532 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-envoy\") pod \"f131a03e-c992-40b1-a6e6-86d29073205b\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " Oct 30 05:33:16.405970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.405563 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-socket\") pod \"f131a03e-c992-40b1-a6e6-86d29073205b\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " Oct 30 05:33:16.405970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.405651 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-data\") pod \"f131a03e-c992-40b1-a6e6-86d29073205b\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " Oct 30 05:33:16.405970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.405675 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-istio-token\" (UniqueName: \"kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-istio-token\") pod \"f131a03e-c992-40b1-a6e6-86d29073205b\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " Oct 30 05:33:16.405970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.405720 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f131a03e-c992-40b1-a6e6-86d29073205b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f131a03e-c992-40b1-a6e6-86d29073205b-istiod-ca-cert\") pod \"f131a03e-c992-40b1-a6e6-86d29073205b\" (UID: \"f131a03e-c992-40b1-a6e6-86d29073205b\") " Oct 30 05:33:16.407102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.407077 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.407211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.407106 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfxnd\" (UniqueName: \"kubernetes.io/projected/126f3871-104a-4cd8-99a2-00e114b38ed5-kube-api-access-nfxnd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.407211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.407121 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126f3871-104a-4cd8-99a2-00e114b38ed5-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.407480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.407451 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-data" pod "f131a03e-c992-40b1-a6e6-86d29073205b" (UID: "f131a03e-c992-40b1-a6e6-86d29073205b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.407554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.407495 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f131a03e-c992-40b1-a6e6-86d29073205b-istiod-ca-cert" pod "f131a03e-c992-40b1-a6e6-86d29073205b" (UID: "f131a03e-c992-40b1-a6e6-86d29073205b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:16.408584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.407667 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-socket" pod "f131a03e-c992-40b1-a6e6-86d29073205b" (UID: "f131a03e-c992-40b1-a6e6-86d29073205b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.408584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.407971 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-credential-socket" pod "f131a03e-c992-40b1-a6e6-86d29073205b" (UID: "f131a03e-c992-40b1-a6e6-86d29073205b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.410177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.410147 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-certs" pod "f131a03e-c992-40b1-a6e6-86d29073205b" (UID: "f131a03e-c992-40b1-a6e6-86d29073205b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.411877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.411850 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-istio-token" pod "f131a03e-c992-40b1-a6e6-86d29073205b" (UID: "f131a03e-c992-40b1-a6e6-86d29073205b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:16.413864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.413841 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-envoy" pod "f131a03e-c992-40b1-a6e6-86d29073205b" (UID: "f131a03e-c992-40b1-a6e6-86d29073205b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.416734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.416583 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-kube-api-access-5rgqp" pod "f131a03e-c992-40b1-a6e6-86d29073205b" (UID: "f131a03e-c992-40b1-a6e6-86d29073205b"). InnerVolumeSpecName "kube-api-access-5rgqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:16.416734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.416671 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f131a03e-c992-40b1-a6e6-86d29073205b-istio-podinfo" pod "f131a03e-c992-40b1-a6e6-86d29073205b" (UID: "f131a03e-c992-40b1-a6e6-86d29073205b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:33:16.443781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.443751 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:33:16.510924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.510886 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-data\") pod \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " Oct 30 05:33:16.511300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.511278 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-certs\") pod \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " Oct 30 05:33:16.511612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.511593 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-envoy\") pod \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " Oct 30 05:33:16.511801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.511784 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-kube-api-access-qn8qb\" (UniqueName: \"kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-kube-api-access-qn8qb\") pod \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " Oct 30 05:33:16.511957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.511943 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istiod-ca-cert\") pod \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " Oct 30 05:33:16.512537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.512514 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-credential-socket\") pod \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " Oct 30 05:33:16.513241 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.513219 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-socket\") pod \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " Oct 30 05:33:16.513736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.513693 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-podinfo\") pod \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " Oct 30 05:33:16.513871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.513763 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-token\" (UniqueName: \"kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-token\") pod \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\" (UID: \"fdb49b21-bb35-4f43-8ddd-2a8104a04509\") " Oct 30 05:33:16.514007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.511292 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-data" pod "fdb49b21-bb35-4f43-8ddd-2a8104a04509" (UID: "fdb49b21-bb35-4f43-8ddd-2a8104a04509"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.514110 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.511540 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-certs" pod "fdb49b21-bb35-4f43-8ddd-2a8104a04509" (UID: "fdb49b21-bb35-4f43-8ddd-2a8104a04509"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.514351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.512385 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istiod-ca-cert" pod "fdb49b21-bb35-4f43-8ddd-2a8104a04509" (UID: "fdb49b21-bb35-4f43-8ddd-2a8104a04509"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:16.514351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.512968 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-credential-socket" pod "fdb49b21-bb35-4f43-8ddd-2a8104a04509" (UID: "fdb49b21-bb35-4f43-8ddd-2a8104a04509"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.514510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.513618 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-socket" pod "fdb49b21-bb35-4f43-8ddd-2a8104a04509" (UID: "fdb49b21-bb35-4f43-8ddd-2a8104a04509"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.514871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.514851 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.514985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.514876 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rgqp\" (UniqueName: \"kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-kube-api-access-5rgqp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.514985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.514891 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.514985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.514905 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.514985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.514931 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.514985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.514945 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.514985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.514957 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.514985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.514970 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.514985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.514992 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.515548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.515004 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f131a03e-c992-40b1-a6e6-86d29073205b-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.515548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.515016 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f131a03e-c992-40b1-a6e6-86d29073205b-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.515548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.515029 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f131a03e-c992-40b1-a6e6-86d29073205b-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.515548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.515108 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.515548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.515126 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f131a03e-c992-40b1-a6e6-86d29073205b-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.530259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.530210 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-podinfo" pod "fdb49b21-bb35-4f43-8ddd-2a8104a04509" (UID: "fdb49b21-bb35-4f43-8ddd-2a8104a04509"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:33:16.530599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.530552 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-envoy" pod "fdb49b21-bb35-4f43-8ddd-2a8104a04509" (UID: "fdb49b21-bb35-4f43-8ddd-2a8104a04509"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.530738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.530621 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-kube-api-access-qn8qb" pod "fdb49b21-bb35-4f43-8ddd-2a8104a04509" (UID: "fdb49b21-bb35-4f43-8ddd-2a8104a04509"). InnerVolumeSpecName "kube-api-access-qn8qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:16.535064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.534999 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-token" pod "fdb49b21-bb35-4f43-8ddd-2a8104a04509" (UID: "fdb49b21-bb35-4f43-8ddd-2a8104a04509"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:16.616694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.616649 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istiod-ca-cert\") pod \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " Oct 30 05:33:16.616968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.616779 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-data\") pod \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " Oct 30 05:33:16.616968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.616834 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-credential-socket\") pod \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " Oct 30 05:33:16.616968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.616881 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-podinfo\") pod \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " Oct 30 05:33:16.616968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.616959 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-token\" (UniqueName: \"kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-token\") pod \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " Oct 30 05:33:16.617212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.617160 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-envoy\") pod \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " Oct 30 05:33:16.617212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.617160 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-data" pod "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" (UID: "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.617212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.617182 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-credential-socket" pod "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" (UID: "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.617212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.617078 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istiod-ca-cert" pod "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" (UID: "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:16.617458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.617194 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-socket\") pod \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " Oct 30 05:33:16.617458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.617287 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-certs\") pod \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " Oct 30 05:33:16.617458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.617445 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-socket" pod "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" (UID: "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.619256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.617493 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-kube-api-access-zgg4n\" (UniqueName: \"kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-kube-api-access-zgg4n\") pod \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\" (UID: \"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9\") " Oct 30 05:33:16.619256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.617519 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-certs" pod "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" (UID: "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.619256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.618572 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.619256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.618600 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn8qb\" (UniqueName: \"kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-kube-api-access-qn8qb\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.619256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.618617 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.619256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.618631 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.619256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.618643 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.619256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.618658 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.619256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.618671 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb49b21-bb35-4f43-8ddd-2a8104a04509-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.619256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.618684 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.619256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.618695 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.620999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.619294 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-podinfo" pod "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" (UID: "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:33:16.620999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.619558 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-envoy" pod "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" (UID: "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:16.620999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.619732 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-token" pod "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" (UID: "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:16.620999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.619852 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-kube-api-access-zgg4n" pod "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" (UID: "cd599f4e-76ee-4ac0-ac7b-95c766cca3f9"). InnerVolumeSpecName "kube-api-access-zgg4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:16.720814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.720754 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.720814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.720777 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.720814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.720786 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.720814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.720795 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgg4n\" (UniqueName: \"kubernetes.io/projected/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9-kube-api-access-zgg4n\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:16.820526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.820486 6587 generic.go:356] "Generic (PLEG): container finished" podID="fdb49b21-bb35-4f43-8ddd-2a8104a04509" containerID="dc48d6602fca7c1aaa44b533a74741ca899c766c1e1fde737603d731f8456539" exitCode=0 Oct 30 05:33:16.820702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.820535 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" Oct 30 05:33:16.820702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.820575 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" event={"ID":"fdb49b21-bb35-4f43-8ddd-2a8104a04509","Type":"ContainerDied","Data":"dc48d6602fca7c1aaa44b533a74741ca899c766c1e1fde737603d731f8456539"} Oct 30 05:33:16.820702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.820639 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd" event={"ID":"fdb49b21-bb35-4f43-8ddd-2a8104a04509","Type":"ContainerDied","Data":"d764f859b41bcf476256360a566b95f3e9e053b40c74d5257c6d958a998ba1ea"} Oct 30 05:33:16.820702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.820669 6587 scope.go:122] "RemoveContainer" containerID="dc48d6602fca7c1aaa44b533a74741ca899c766c1e1fde737603d731f8456539" Oct 30 05:33:16.822839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.822812 6587 generic.go:356] "Generic (PLEG): container finished" podID="cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" containerID="551e859e4d30609a03548d99aa818ce0969bed5b0ef1ac984b7232c56937a361" exitCode=0 Oct 30 05:33:16.822966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.822867 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" Oct 30 05:33:16.822966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.822899 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" event={"ID":"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9","Type":"ContainerDied","Data":"551e859e4d30609a03548d99aa818ce0969bed5b0ef1ac984b7232c56937a361"} Oct 30 05:33:16.822966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.822951 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2" event={"ID":"cd599f4e-76ee-4ac0-ac7b-95c766cca3f9","Type":"ContainerDied","Data":"e334320f8b3af8d225c85ff293f58db68a0dedb1cc40c2b1289f921656070824"} Oct 30 05:33:16.825631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.825605 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" Oct 30 05:33:16.826440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.825607 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6" event={"ID":"126f3871-104a-4cd8-99a2-00e114b38ed5","Type":"ContainerDied","Data":"543139616a6d695b10eae8b8714eba04603ee6fbccb03fae5bcc3894c5f3016c"} Oct 30 05:33:16.826440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.825739 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="543139616a6d695b10eae8b8714eba04603ee6fbccb03fae5bcc3894c5f3016c" Oct 30 05:33:16.827955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.827931 6587 generic.go:356] "Generic (PLEG): container finished" podID="f131a03e-c992-40b1-a6e6-86d29073205b" containerID="4ff405235d16322288fb6bc325830e7fc88a00682afe4462c15cebe05bc130fc" exitCode=0 Oct 30 05:33:16.828690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.828670 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" Oct 30 05:33:16.829541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.829442 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" event={"ID":"f131a03e-c992-40b1-a6e6-86d29073205b","Type":"ContainerDied","Data":"4ff405235d16322288fb6bc325830e7fc88a00682afe4462c15cebe05bc130fc"} Oct 30 05:33:16.829541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.829504 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm" event={"ID":"f131a03e-c992-40b1-a6e6-86d29073205b","Type":"ContainerDied","Data":"797fa622e7c099b4c6fa5a61632fbfcba591f0b81054b2981f8a4af0a00ed5cc"} Oct 30 05:33:16.836966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.836308 6587 scope.go:122] "RemoveContainer" containerID="dc48d6602fca7c1aaa44b533a74741ca899c766c1e1fde737603d731f8456539" Oct 30 05:33:16.836966 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:16.836624 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc48d6602fca7c1aaa44b533a74741ca899c766c1e1fde737603d731f8456539\": container with ID starting with dc48d6602fca7c1aaa44b533a74741ca899c766c1e1fde737603d731f8456539 not found: ID does not exist" containerID="dc48d6602fca7c1aaa44b533a74741ca899c766c1e1fde737603d731f8456539" Oct 30 05:33:16.836966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.836648 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc48d6602fca7c1aaa44b533a74741ca899c766c1e1fde737603d731f8456539"} err="failed to get container status \"dc48d6602fca7c1aaa44b533a74741ca899c766c1e1fde737603d731f8456539\": rpc error: code = NotFound desc = could not find container \"dc48d6602fca7c1aaa44b533a74741ca899c766c1e1fde737603d731f8456539\": container with ID starting with dc48d6602fca7c1aaa44b533a74741ca899c766c1e1fde737603d731f8456539 not found: ID does not exist" Oct 30 05:33:16.836966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.836667 6587 scope.go:122] "RemoveContainer" containerID="551e859e4d30609a03548d99aa818ce0969bed5b0ef1ac984b7232c56937a361" Oct 30 05:33:16.849886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.849867 6587 scope.go:122] "RemoveContainer" containerID="551e859e4d30609a03548d99aa818ce0969bed5b0ef1ac984b7232c56937a361" Oct 30 05:33:16.850176 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:16.850150 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551e859e4d30609a03548d99aa818ce0969bed5b0ef1ac984b7232c56937a361\": container with ID starting with 551e859e4d30609a03548d99aa818ce0969bed5b0ef1ac984b7232c56937a361 not found: ID does not exist" containerID="551e859e4d30609a03548d99aa818ce0969bed5b0ef1ac984b7232c56937a361" Oct 30 05:33:16.850274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.850183 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551e859e4d30609a03548d99aa818ce0969bed5b0ef1ac984b7232c56937a361"} err="failed to get container status \"551e859e4d30609a03548d99aa818ce0969bed5b0ef1ac984b7232c56937a361\": rpc error: code = NotFound desc = could not find container \"551e859e4d30609a03548d99aa818ce0969bed5b0ef1ac984b7232c56937a361\": container with ID starting with 551e859e4d30609a03548d99aa818ce0969bed5b0ef1ac984b7232c56937a361 not found: ID does not exist" Oct 30 05:33:16.850274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.850203 6587 scope.go:122] "RemoveContainer" containerID="4ff405235d16322288fb6bc325830e7fc88a00682afe4462c15cebe05bc130fc" Oct 30 05:33:16.862851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.862833 6587 scope.go:122] "RemoveContainer" containerID="6512ca1ab0e81c9d97c1839699c9f2e89cce37f7685c36c414d15c8cf626f5c1" Oct 30 05:33:16.875680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.875660 6587 scope.go:122] "RemoveContainer" containerID="4ff405235d16322288fb6bc325830e7fc88a00682afe4462c15cebe05bc130fc" Oct 30 05:33:16.876027 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:16.876002 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff405235d16322288fb6bc325830e7fc88a00682afe4462c15cebe05bc130fc\": container with ID starting with 4ff405235d16322288fb6bc325830e7fc88a00682afe4462c15cebe05bc130fc not found: ID does not exist" containerID="4ff405235d16322288fb6bc325830e7fc88a00682afe4462c15cebe05bc130fc" Oct 30 05:33:16.876747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.876037 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff405235d16322288fb6bc325830e7fc88a00682afe4462c15cebe05bc130fc"} err="failed to get container status \"4ff405235d16322288fb6bc325830e7fc88a00682afe4462c15cebe05bc130fc\": rpc error: code = NotFound desc = could not find container \"4ff405235d16322288fb6bc325830e7fc88a00682afe4462c15cebe05bc130fc\": container with ID starting with 4ff405235d16322288fb6bc325830e7fc88a00682afe4462c15cebe05bc130fc not found: ID does not exist" Oct 30 05:33:16.876747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.876059 6587 scope.go:122] "RemoveContainer" containerID="6512ca1ab0e81c9d97c1839699c9f2e89cce37f7685c36c414d15c8cf626f5c1" Oct 30 05:33:16.876747 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:16.876347 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6512ca1ab0e81c9d97c1839699c9f2e89cce37f7685c36c414d15c8cf626f5c1\": container with ID starting with 6512ca1ab0e81c9d97c1839699c9f2e89cce37f7685c36c414d15c8cf626f5c1 not found: ID does not exist" containerID="6512ca1ab0e81c9d97c1839699c9f2e89cce37f7685c36c414d15c8cf626f5c1" Oct 30 05:33:16.876747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.876419 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6512ca1ab0e81c9d97c1839699c9f2e89cce37f7685c36c414d15c8cf626f5c1"} err="failed to get container status \"6512ca1ab0e81c9d97c1839699c9f2e89cce37f7685c36c414d15c8cf626f5c1\": rpc error: code = NotFound desc = could not find container \"6512ca1ab0e81c9d97c1839699c9f2e89cce37f7685c36c414d15c8cf626f5c1\": container with ID starting with 6512ca1ab0e81c9d97c1839699c9f2e89cce37f7685c36c414d15c8cf626f5c1 not found: ID does not exist" Oct 30 05:33:16.882200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.882180 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd"] Oct 30 05:33:16.895438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.895414 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd"] Oct 30 05:33:16.946745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.946708 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2"] Oct 30 05:33:16.966521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:16.966490 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2"] Oct 30 05:33:17.032407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:17.032289 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm"] Oct 30 05:33:17.041710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:17.041681 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm"] Oct 30 05:33:17.587978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:17.587940 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" path="/var/lib/kubelet/pods/cd599f4e-76ee-4ac0-ac7b-95c766cca3f9/volumes" Oct 30 05:33:17.588862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:17.588840 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" path="/var/lib/kubelet/pods/f131a03e-c992-40b1-a6e6-86d29073205b/volumes" Oct 30 05:33:17.589558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:17.589540 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb49b21-bb35-4f43-8ddd-2a8104a04509" path="/var/lib/kubelet/pods/fdb49b21-bb35-4f43-8ddd-2a8104a04509/volumes" Oct 30 05:33:17.751735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:17.751700 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj"] Oct 30 05:33:17.787159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:17.787123 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk"] Oct 30 05:33:19.771576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.771538 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9"] Oct 30 05:33:19.772487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772465 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fdb49b21-bb35-4f43-8ddd-2a8104a04509" containerName="istio-proxy" Oct 30 05:33:19.772487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772484 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb49b21-bb35-4f43-8ddd-2a8104a04509" containerName="istio-proxy" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772508 6587 cpu_manager.go:414] "containerMap: removing container" podUID="126f3871-104a-4cd8-99a2-00e114b38ed5" containerName="revoke-keys" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772513 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126f3871-104a-4cd8-99a2-00e114b38ed5" containerName="revoke-keys" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772538 6587 cpu_manager.go:414] "containerMap: removing container" podUID="6678a054-bb47-4483-8ba3-5222da9e56e1" containerName="revoke-keys" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772544 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6678a054-bb47-4483-8ba3-5222da9e56e1" containerName="revoke-keys" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772556 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772561 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772590 6587 cpu_manager.go:414] "containerMap: removing container" podUID="cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" containerName="istio-proxy" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772595 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" containerName="istio-proxy" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772711 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="6678a054-bb47-4483-8ba3-5222da9e56e1" containerName="revoke-keys" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772721 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772731 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772737 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd599f4e-76ee-4ac0-ac7b-95c766cca3f9" containerName="istio-proxy" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772744 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb49b21-bb35-4f43-8ddd-2a8104a04509" containerName="istio-proxy" Oct 30 05:33:19.773774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.772751 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="126f3871-104a-4cd8-99a2-00e114b38ed5" containerName="revoke-keys" Oct 30 05:33:19.778720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.778700 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.794795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.794766 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.794914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.794876 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.794998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.794954 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.794998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.794981 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.795177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.795147 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.795276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.795259 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.795484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.795463 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.795632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.795614 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.796505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.795656 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjw24\" (UniqueName: \"kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-kube-api-access-xjw24\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.827678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.827643 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9"] Oct 30 05:33:19.897004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.896961 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.897215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.897068 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.897215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.897155 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.897215 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.897179 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.897350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.897232 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.897350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.897335 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.897500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.897458 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.897500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.897497 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.899526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.897648 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.899526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.897682 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjw24\" (UniqueName: \"kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-kube-api-access-xjw24\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.899526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.897857 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.899526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.897972 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.899526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.898165 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.899526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.898200 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.900309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.899884 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.900516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.900495 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.922652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.922619 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc"] Oct 30 05:33:19.924103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.924075 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" Oct 30 05:33:19.924103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.924101 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f131a03e-c992-40b1-a6e6-86d29073205b" containerName="istio-proxy" Oct 30 05:33:19.926533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.926504 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjw24\" (UniqueName: \"kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-kube-api-access-xjw24\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.926820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.926797 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:19.937817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.937795 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" Oct 30 05:33:19.952046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.952017 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc"] Oct 30 05:33:19.957576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.957545 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:33:19.957726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.957596 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:33:19.999587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.999558 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc91e775-2tffc\" (UID: \"c7fd2f46-2be0-483e-a865-af37dcea59be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" Oct 30 05:33:19.999768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.999719 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc91e775-2tffc\" (UID: \"c7fd2f46-2be0-483e-a865-af37dcea59be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" Oct 30 05:33:19.999916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:19.999900 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svsb4\" (UniqueName: \"kubernetes.io/projected/c7fd2f46-2be0-483e-a865-af37dcea59be-kube-api-access-svsb4\") pod \"maas-api-revoke-sub-dc91e775-2tffc\" (UID: \"c7fd2f46-2be0-483e-a865-af37dcea59be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" Oct 30 05:33:20.102003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.101965 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svsb4\" (UniqueName: \"kubernetes.io/projected/c7fd2f46-2be0-483e-a865-af37dcea59be-kube-api-access-svsb4\") pod \"maas-api-revoke-sub-dc91e775-2tffc\" (UID: \"c7fd2f46-2be0-483e-a865-af37dcea59be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" Oct 30 05:33:20.102207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.102187 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc91e775-2tffc\" (UID: \"c7fd2f46-2be0-483e-a865-af37dcea59be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" Oct 30 05:33:20.102376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.102344 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc91e775-2tffc\" (UID: \"c7fd2f46-2be0-483e-a865-af37dcea59be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" Oct 30 05:33:20.103039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.103013 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc91e775-2tffc\" (UID: \"c7fd2f46-2be0-483e-a865-af37dcea59be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" Oct 30 05:33:20.104976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.104951 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc91e775-2tffc\" (UID: \"c7fd2f46-2be0-483e-a865-af37dcea59be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" Oct 30 05:33:20.106989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.106966 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:20.133120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.133094 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svsb4\" (UniqueName: \"kubernetes.io/projected/c7fd2f46-2be0-483e-a865-af37dcea59be-kube-api-access-svsb4\") pod \"maas-api-revoke-sub-dc91e775-2tffc\" (UID: \"c7fd2f46-2be0-483e-a865-af37dcea59be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" Oct 30 05:33:20.288641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.288605 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" Oct 30 05:33:20.480609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.480579 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9"] Oct 30 05:33:20.488115 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:33:20.488085 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02ff6cc_a5bb_40de_a4e3_9cd3058abc64.slice/crio-eaaf2163e542bac75a91d78c68cb6af8c71ee7455016a5a091f84a6e86c5a252 WatchSource:0}: Error finding container eaaf2163e542bac75a91d78c68cb6af8c71ee7455016a5a091f84a6e86c5a252: Status 404 returned error can't find the container with id eaaf2163e542bac75a91d78c68cb6af8c71ee7455016a5a091f84a6e86c5a252 Oct 30 05:33:20.494556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.494515 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:20.494659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.494606 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:20.494735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.494671 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:20.626857 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:33:20.626824 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fd2f46_2be0_483e_a865_af37dcea59be.slice/crio-f8236e4d1b023123f84204b8e126577e7f49e4524cbdf088b2c5d62a0b836d44 WatchSource:0}: Error finding container f8236e4d1b023123f84204b8e126577e7f49e4524cbdf088b2c5d62a0b836d44: Status 404 returned error can't find the container with id f8236e4d1b023123f84204b8e126577e7f49e4524cbdf088b2c5d62a0b836d44 Oct 30 05:33:20.628789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.628753 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc"] Oct 30 05:33:20.881183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.881075 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" event={"ID":"c7fd2f46-2be0-483e-a865-af37dcea59be","Type":"ContainerStarted","Data":"f8236e4d1b023123f84204b8e126577e7f49e4524cbdf088b2c5d62a0b836d44"} Oct 30 05:33:20.887494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.883782 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" event={"ID":"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64","Type":"ContainerStarted","Data":"3878a63e317838cea37715cd6c8c7286cf4849d2fa2038ba086adfdae74ff79f"} Oct 30 05:33:20.887494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:20.883839 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" event={"ID":"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64","Type":"ContainerStarted","Data":"eaaf2163e542bac75a91d78c68cb6af8c71ee7455016a5a091f84a6e86c5a252"} Oct 30 05:33:21.107425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:21.107353 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:21.108820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:21.108795 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Oct 30 05:33:21.108931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:21.108846 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" podUID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Oct 30 05:33:21.870044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:21.870013 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:33:21.896347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:21.896304 6587 generic.go:356] "Generic (PLEG): container finished" podID="c7fd2f46-2be0-483e-a865-af37dcea59be" containerID="b809d44cbe25300f3b6546d43edbfad478e9223a051ea0d65ad4f8134aacc439" exitCode=0 Oct 30 05:33:21.907853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:21.897803 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" event={"ID":"c7fd2f46-2be0-483e-a865-af37dcea59be","Type":"ContainerDied","Data":"b809d44cbe25300f3b6546d43edbfad478e9223a051ea0d65ad4f8134aacc439"} Oct 30 05:33:21.924381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:21.923777 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" podStartSLOduration=2.9237259030000002 podStartE2EDuration="2.923725903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:20.935864539 +0000 UTC m=+2925.463631326" watchObservedRunningTime="2026-10-30 05:33:21.923725903 +0000 UTC m=+2926.451492690" Oct 30 05:33:22.052276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:22.052243 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw"] Oct 30 05:33:22.057199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:22.057169 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59916050-66bzw"] Oct 30 05:33:22.108138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:22.108061 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Oct 30 05:33:22.108138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:22.108114 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" podUID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Oct 30 05:33:22.789382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:22.777660 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" podUID="d591ba5b-cdc7-4f23-85c6-28418b0abefe" containerName="payload-processing" containerID="cri-o://e1cf2dd39fd53f68e4c349d1007fe123ce6e5fad721c1db4be4aa7a1e4b1a7e3" gracePeriod=25 Oct 30 05:33:22.817382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:22.795900 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" podUID="46d11379-6d13-4b4b-a36b-4d418b490323" containerName="payload-pre-processing" containerID="cri-o://70940e2c3d864d54730995d4b39b65836deabb4454d8d72e326b65aa8a5ad835" gracePeriod=25 Oct 30 05:33:23.114804 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:23.111268 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Oct 30 05:33:23.114804 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:23.111328 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" podUID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Oct 30 05:33:23.402775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:23.402748 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" Oct 30 05:33:23.477876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:23.477843 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7fd2f46-2be0-483e-a865-af37dcea59be-kube-api-access-svsb4\" (UniqueName: \"kubernetes.io/projected/c7fd2f46-2be0-483e-a865-af37dcea59be-kube-api-access-svsb4\") pod \"c7fd2f46-2be0-483e-a865-af37dcea59be\" (UID: \"c7fd2f46-2be0-483e-a865-af37dcea59be\") " Oct 30 05:33:23.478176 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:23.478155 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-xks-ca\") pod \"c7fd2f46-2be0-483e-a865-af37dcea59be\" (UID: \"c7fd2f46-2be0-483e-a865-af37dcea59be\") " Oct 30 05:33:23.478397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:23.478371 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-openshift-ca\") pod \"c7fd2f46-2be0-483e-a865-af37dcea59be\" (UID: \"c7fd2f46-2be0-483e-a865-af37dcea59be\") " Oct 30 05:33:23.478981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:23.478953 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-openshift-ca" pod "c7fd2f46-2be0-483e-a865-af37dcea59be" (UID: "c7fd2f46-2be0-483e-a865-af37dcea59be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:23.479266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:23.479243 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:23.481167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:23.480488 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fd2f46-2be0-483e-a865-af37dcea59be-kube-api-access-svsb4" pod "c7fd2f46-2be0-483e-a865-af37dcea59be" (UID: "c7fd2f46-2be0-483e-a865-af37dcea59be"). InnerVolumeSpecName "kube-api-access-svsb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:23.481167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:23.480547 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-xks-ca" pod "c7fd2f46-2be0-483e-a865-af37dcea59be" (UID: "c7fd2f46-2be0-483e-a865-af37dcea59be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:23.580570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:23.580537 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svsb4\" (UniqueName: \"kubernetes.io/projected/c7fd2f46-2be0-483e-a865-af37dcea59be-kube-api-access-svsb4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:23.580570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:23.580569 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd2f46-2be0-483e-a865-af37dcea59be-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:23.589779 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:23.589746 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d30d45-ade0-4df6-8fa2-d07894965147" path="/var/lib/kubelet/pods/f1d30d45-ade0-4df6-8fa2-d07894965147/volumes" Oct 30 05:33:24.113723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:24.113674 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Oct 30 05:33:24.113954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:24.113737 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" podUID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Oct 30 05:33:24.128390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:24.123565 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" event={"ID":"c7fd2f46-2be0-483e-a865-af37dcea59be","Type":"ContainerDied","Data":"f8236e4d1b023123f84204b8e126577e7f49e4524cbdf088b2c5d62a0b836d44"} Oct 30 05:33:24.129218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:24.129193 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8236e4d1b023123f84204b8e126577e7f49e4524cbdf088b2c5d62a0b836d44" Oct 30 05:33:24.136105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:24.129000 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc" Oct 30 05:33:25.107900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:25.107862 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Oct 30 05:33:25.108085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:25.107912 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" podUID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Oct 30 05:33:25.681285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:25.681238 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" podUID="46d11379-6d13-4b4b-a36b-4d418b490323" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:33:25.681877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:25.681646 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" podUID="d591ba5b-cdc7-4f23-85c6-28418b0abefe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:33:26.113380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:26.107791 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Oct 30 05:33:26.113380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:26.107873 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" podUID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Oct 30 05:33:26.332603 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:26.332530 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:33:26.596581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:26.596546 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c"] Oct 30 05:33:26.609047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:26.609015 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2f6fed-jmf6c"] Oct 30 05:33:27.050381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.048904 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" containerID="cri-o://f6a94b57388cc1058fefd5617f7e85408ffa91605d5fcda533a88d934be6cdcd" gracePeriod=45 Oct 30 05:33:27.108327 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.108301 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Oct 30 05:33:27.108980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.108349 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" podUID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Oct 30 05:33:27.308117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.308018 6587 generic.go:356] "Generic (PLEG): container finished" podID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerID="f6a94b57388cc1058fefd5617f7e85408ffa91605d5fcda533a88d934be6cdcd" exitCode=0 Oct 30 05:33:27.308117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.308096 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" event={"ID":"9889ac0a-17cf-41c2-b40b-b5ed537cc94c","Type":"ContainerDied","Data":"f6a94b57388cc1058fefd5617f7e85408ffa91605d5fcda533a88d934be6cdcd"} Oct 30 05:33:27.385618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.385588 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:33:27.498680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.498188 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-dshm\") pod \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " Oct 30 05:33:27.498924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.498897 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-home\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-home\") pod \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " Oct 30 05:33:27.499875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.498994 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tmp-dir\") pod \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " Oct 30 05:33:27.499875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.499089 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tls-certs\" (UniqueName: \"kubernetes.io/secret/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tls-certs\") pod \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " Oct 30 05:33:27.499875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.499181 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-model-cache\") pod \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " Oct 30 05:33:27.499875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.499452 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-kube-api-access-74l8c\" (UniqueName: \"kubernetes.io/projected/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-kube-api-access-74l8c\") pod \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\" (UID: \"9889ac0a-17cf-41c2-b40b-b5ed537cc94c\") " Oct 30 05:33:27.501534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.501494 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tmp-dir" pod "9889ac0a-17cf-41c2-b40b-b5ed537cc94c" (UID: "9889ac0a-17cf-41c2-b40b-b5ed537cc94c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:27.501722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.501699 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-dshm" pod "9889ac0a-17cf-41c2-b40b-b5ed537cc94c" (UID: "9889ac0a-17cf-41c2-b40b-b5ed537cc94c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:27.502132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.502102 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-model-cache" pod "9889ac0a-17cf-41c2-b40b-b5ed537cc94c" (UID: "9889ac0a-17cf-41c2-b40b-b5ed537cc94c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:27.502212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.502136 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-home" pod "9889ac0a-17cf-41c2-b40b-b5ed537cc94c" (UID: "9889ac0a-17cf-41c2-b40b-b5ed537cc94c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:27.503115 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.503091 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-kube-api-access-74l8c" pod "9889ac0a-17cf-41c2-b40b-b5ed537cc94c" (UID: "9889ac0a-17cf-41c2-b40b-b5ed537cc94c"). InnerVolumeSpecName "kube-api-access-74l8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:27.505751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.505725 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tls-certs" pod "9889ac0a-17cf-41c2-b40b-b5ed537cc94c" (UID: "9889ac0a-17cf-41c2-b40b-b5ed537cc94c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:27.591412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.591313 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6678a054-bb47-4483-8ba3-5222da9e56e1" path="/var/lib/kubelet/pods/6678a054-bb47-4483-8ba3-5222da9e56e1/volumes" Oct 30 05:33:27.603056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.602436 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74l8c\" (UniqueName: \"kubernetes.io/projected/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-kube-api-access-74l8c\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.603056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.602471 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.603056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.602488 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.603056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.602501 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.603056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.602515 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:27.603056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:27.602528 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9889ac0a-17cf-41c2-b40b-b5ed537cc94c-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:28.246934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:28.246899 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:28.247526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:28.247506 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:28.248496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:28.248474 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:33:28.321966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:28.321926 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" Oct 30 05:33:28.323705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:28.321958 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj" event={"ID":"9889ac0a-17cf-41c2-b40b-b5ed537cc94c","Type":"ContainerDied","Data":"934403b52864d11d81944389ecf798b90fbd7458b7d34299e4ec243b688a7314"} Oct 30 05:33:28.323705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:28.322012 6587 scope.go:122] "RemoveContainer" containerID="f6a94b57388cc1058fefd5617f7e85408ffa91605d5fcda533a88d934be6cdcd" Oct 30 05:33:28.427924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:28.426950 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj"] Oct 30 05:33:28.435102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:28.433826 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/simulator-w3-889ca9-kserve-64bd59c796-n58vj"] Oct 30 05:33:29.585852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:29.585816 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" path="/var/lib/kubelet/pods/9889ac0a-17cf-41c2-b40b-b5ed537cc94c/volumes" Oct 30 05:33:31.445664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:31.445623 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx"] Oct 30 05:33:31.446189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:31.445883 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" podUID="0e413931-a15a-4fb2-82ef-4e0096717c33" containerName="istio-proxy" containerID="cri-o://a8ddaa1951425ea4f8c5a558997284c641f84e111f873509902044c689ff91d1" gracePeriod=30 Oct 30 05:33:32.458345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:32.458312 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k"] Oct 30 05:33:34.503343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.503306 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4"] Oct 30 05:33:34.504265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.504246 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c7fd2f46-2be0-483e-a865-af37dcea59be" containerName="revoke-keys" Oct 30 05:33:34.505138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.504267 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7fd2f46-2be0-483e-a865-af37dcea59be" containerName="revoke-keys" Oct 30 05:33:34.505138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.504291 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" Oct 30 05:33:34.505138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.504296 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" Oct 30 05:33:34.505138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.504438 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7fd2f46-2be0-483e-a865-af37dcea59be" containerName="revoke-keys" Oct 30 05:33:34.505138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.504451 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9889ac0a-17cf-41c2-b40b-b5ed537cc94c" containerName="main" Oct 30 05:33:34.508782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.508762 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" Oct 30 05:33:34.533581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.533546 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4"] Oct 30 05:33:34.539416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.537726 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4bc875\"" Oct 30 05:33:34.539416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.537937 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4bc875-dockercfg-4n2kq\"" Oct 30 05:33:34.669452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.669410 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182c089e-21c6-470b-9dcd-85c1ced34758-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4\" (UID: \"182c089e-21c6-470b-9dcd-85c1ced34758\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" Oct 30 05:33:34.907907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.907862 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57dkz\" (UniqueName: \"kubernetes.io/projected/182c089e-21c6-470b-9dcd-85c1ced34758-kube-api-access-57dkz\") pod \"payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4\" (UID: \"182c089e-21c6-470b-9dcd-85c1ced34758\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" Oct 30 05:33:35.014272 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:34.915646 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4"] Oct 30 05:33:35.074681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.074646 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" Oct 30 05:33:35.371903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.371869 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182c089e-21c6-470b-9dcd-85c1ced34758-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4\" (UID: \"182c089e-21c6-470b-9dcd-85c1ced34758\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" Oct 30 05:33:35.373427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.373397 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b62bb7-1e0c-470c-b69e-fbd26617030d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4\" (UID: \"85b62bb7-1e0c-470c-b69e-fbd26617030d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" Oct 30 05:33:35.373899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.373876 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc4ps\" (UniqueName: \"kubernetes.io/projected/85b62bb7-1e0c-470c-b69e-fbd26617030d-kube-api-access-gc4ps\") pod \"payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4\" (UID: \"85b62bb7-1e0c-470c-b69e-fbd26617030d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" Oct 30 05:33:35.377273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.377240 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57dkz\" (UniqueName: \"kubernetes.io/projected/182c089e-21c6-470b-9dcd-85c1ced34758-kube-api-access-57dkz\") pod \"payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4\" (UID: \"182c089e-21c6-470b-9dcd-85c1ced34758\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" Oct 30 05:33:35.378074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.373160 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182c089e-21c6-470b-9dcd-85c1ced34758-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4\" (UID: \"182c089e-21c6-470b-9dcd-85c1ced34758\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" Oct 30 05:33:35.380369 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.380334 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4"] Oct 30 05:33:35.408635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.408308 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57dkz\" (UniqueName: \"kubernetes.io/projected/182c089e-21c6-470b-9dcd-85c1ced34758-kube-api-access-57dkz\") pod \"payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4\" (UID: \"182c089e-21c6-470b-9dcd-85c1ced34758\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" Oct 30 05:33:35.416913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.416884 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb"] Oct 30 05:33:35.422469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.422447 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" Oct 30 05:33:35.433827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.433799 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" Oct 30 05:33:35.440869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.440762 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:33:35.449246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.449063 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb"] Oct 30 05:33:35.450417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.450276 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:33:35.505299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.505265 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb\" (UID: \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" Oct 30 05:33:35.505757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.505526 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnshk\" (UniqueName: \"kubernetes.io/projected/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-kube-api-access-qnshk\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb\" (UID: \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" Oct 30 05:33:35.505757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.505600 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb\" (UID: \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" Oct 30 05:33:35.505757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.505654 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b62bb7-1e0c-470c-b69e-fbd26617030d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4\" (UID: \"85b62bb7-1e0c-470c-b69e-fbd26617030d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" Oct 30 05:33:35.505757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.505704 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc4ps\" (UniqueName: \"kubernetes.io/projected/85b62bb7-1e0c-470c-b69e-fbd26617030d-kube-api-access-gc4ps\") pod \"payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4\" (UID: \"85b62bb7-1e0c-470c-b69e-fbd26617030d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" Oct 30 05:33:35.509762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.509731 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b62bb7-1e0c-470c-b69e-fbd26617030d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4\" (UID: \"85b62bb7-1e0c-470c-b69e-fbd26617030d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" Oct 30 05:33:35.540337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.540305 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc4ps\" (UniqueName: \"kubernetes.io/projected/85b62bb7-1e0c-470c-b69e-fbd26617030d-kube-api-access-gc4ps\") pod \"payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4\" (UID: \"85b62bb7-1e0c-470c-b69e-fbd26617030d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" Oct 30 05:33:35.607687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.607236 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb\" (UID: \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" Oct 30 05:33:35.607687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.607580 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb\" (UID: \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" Oct 30 05:33:35.607920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.607833 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnshk\" (UniqueName: \"kubernetes.io/projected/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-kube-api-access-qnshk\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb\" (UID: \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" Oct 30 05:33:35.608344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.608005 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb\" (UID: \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" Oct 30 05:33:35.610526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.610499 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb\" (UID: \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" Oct 30 05:33:35.630195 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.630157 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnshk\" (UniqueName: \"kubernetes.io/projected/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-kube-api-access-qnshk\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb\" (UID: \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" Oct 30 05:33:35.682586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.682546 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" podUID="46d11379-6d13-4b4b-a36b-4d418b490323" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:33:35.683471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.682845 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" podUID="d591ba5b-cdc7-4f23-85c6-28418b0abefe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:33:35.737784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.737749 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" Oct 30 05:33:35.820736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.820706 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:33:35.825958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.824840 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" Oct 30 05:33:35.841294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:35.841246 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4"] Oct 30 05:33:36.057759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.057729 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4"] Oct 30 05:33:36.211067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.211039 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb"] Oct 30 05:33:36.213503 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:33:36.213474 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8d6cde_f5c2_4a82_90f3_c5782a03c8cd.slice/crio-83d28f4d31e63ec032b243cafe73c814c9c16a70e9ecba87c00d078740342b43 WatchSource:0}: Error finding container 83d28f4d31e63ec032b243cafe73c814c9c16a70e9ecba87c00d078740342b43: Status 404 returned error can't find the container with id 83d28f4d31e63ec032b243cafe73c814c9c16a70e9ecba87c00d078740342b43 Oct 30 05:33:36.472721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.472684 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" event={"ID":"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd","Type":"ContainerStarted","Data":"83d28f4d31e63ec032b243cafe73c814c9c16a70e9ecba87c00d078740342b43"} Oct 30 05:33:36.475432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.475283 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" event={"ID":"182c089e-21c6-470b-9dcd-85c1ced34758","Type":"ContainerStarted","Data":"2fa649156ae366cb6305078e4b009725f22b197d20fd8084447dc15a5a51d71a"} Oct 30 05:33:36.475432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.475340 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" event={"ID":"182c089e-21c6-470b-9dcd-85c1ced34758","Type":"ContainerStarted","Data":"14164e1d45d4edb443b0c02a64dc5b1607e8e54e5c8265fcb5b8a4124b8538af"} Oct 30 05:33:36.475708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.475671 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" Oct 30 05:33:36.477598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.477576 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" event={"ID":"85b62bb7-1e0c-470c-b69e-fbd26617030d","Type":"ContainerStarted","Data":"fcdebe8140172f7308ec79fb99ca67d6bcc702e7e5d498747a7db32174149dfa"} Oct 30 05:33:36.477702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.477611 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" event={"ID":"85b62bb7-1e0c-470c-b69e-fbd26617030d","Type":"ContainerStarted","Data":"52810d5fa5a0e8b6df1f4c01b57de7bb0e3feb8de3d123c50ecf40f875a110f1"} Oct 30 05:33:36.477768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.477722 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" Oct 30 05:33:36.536518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.536460 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" podStartSLOduration=2.536445379 podStartE2EDuration="2.536445379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:36.531677144 +0000 UTC m=+2941.059443931" watchObservedRunningTime="2026-10-30 05:33:36.536445379 +0000 UTC m=+2941.064212176" Oct 30 05:33:36.583270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.583214 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" podStartSLOduration=2.583198193 podStartE2EDuration="2.583198193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:36.581976451 +0000 UTC m=+2941.109743232" watchObservedRunningTime="2026-10-30 05:33:36.583198193 +0000 UTC m=+2941.110964999" Oct 30 05:33:36.892474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.892448 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:33:36.943186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.943156 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-token\" (UniqueName: \"kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-token\") pod \"0e413931-a15a-4fb2-82ef-4e0096717c33\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.943205 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-socket\") pod \"0e413931-a15a-4fb2-82ef-4e0096717c33\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.943245 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-data\") pod \"0e413931-a15a-4fb2-82ef-4e0096717c33\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.943299 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-envoy\") pod \"0e413931-a15a-4fb2-82ef-4e0096717c33\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.943350 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-kube-api-access-cfd2q\" (UniqueName: \"kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-kube-api-access-cfd2q\") pod \"0e413931-a15a-4fb2-82ef-4e0096717c33\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.943408 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-certs\") pod \"0e413931-a15a-4fb2-82ef-4e0096717c33\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.943479 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-podinfo\") pod \"0e413931-a15a-4fb2-82ef-4e0096717c33\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.943682 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-credential-socket\") pod \"0e413931-a15a-4fb2-82ef-4e0096717c33\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.943801 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e413931-a15a-4fb2-82ef-4e0096717c33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e413931-a15a-4fb2-82ef-4e0096717c33-istiod-ca-cert\") pod \"0e413931-a15a-4fb2-82ef-4e0096717c33\" (UID: \"0e413931-a15a-4fb2-82ef-4e0096717c33\") " Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.944589 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-certs" pod "0e413931-a15a-4fb2-82ef-4e0096717c33" (UID: "0e413931-a15a-4fb2-82ef-4e0096717c33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.944830 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-data" pod "0e413931-a15a-4fb2-82ef-4e0096717c33" (UID: "0e413931-a15a-4fb2-82ef-4e0096717c33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.944891 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e413931-a15a-4fb2-82ef-4e0096717c33-istiod-ca-cert" pod "0e413931-a15a-4fb2-82ef-4e0096717c33" (UID: "0e413931-a15a-4fb2-82ef-4e0096717c33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.945076 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-socket" pod "0e413931-a15a-4fb2-82ef-4e0096717c33" (UID: "0e413931-a15a-4fb2-82ef-4e0096717c33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.945157 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-credential-socket" pod "0e413931-a15a-4fb2-82ef-4e0096717c33" (UID: "0e413931-a15a-4fb2-82ef-4e0096717c33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.945476 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.945498 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.945512 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e413931-a15a-4fb2-82ef-4e0096717c33-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.945527 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.945542 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:36.959526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.948996 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-envoy" pod "0e413931-a15a-4fb2-82ef-4e0096717c33" (UID: "0e413931-a15a-4fb2-82ef-4e0096717c33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:36.960823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.960762 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-kube-api-access-cfd2q" pod "0e413931-a15a-4fb2-82ef-4e0096717c33" (UID: "0e413931-a15a-4fb2-82ef-4e0096717c33"). InnerVolumeSpecName "kube-api-access-cfd2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:36.960823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.960771 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-podinfo" pod "0e413931-a15a-4fb2-82ef-4e0096717c33" (UID: "0e413931-a15a-4fb2-82ef-4e0096717c33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:33:36.961154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:36.961122 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-token" pod "0e413931-a15a-4fb2-82ef-4e0096717c33" (UID: "0e413931-a15a-4fb2-82ef-4e0096717c33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:37.047619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.047538 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:37.047619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.047577 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:37.047619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.047591 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfd2q\" (UniqueName: \"kubernetes.io/projected/0e413931-a15a-4fb2-82ef-4e0096717c33-kube-api-access-cfd2q\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:37.047619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.047606 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e413931-a15a-4fb2-82ef-4e0096717c33-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:37.499277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.499240 6587 generic.go:356] "Generic (PLEG): container finished" podID="0e413931-a15a-4fb2-82ef-4e0096717c33" containerID="a8ddaa1951425ea4f8c5a558997284c641f84e111f873509902044c689ff91d1" exitCode=0 Oct 30 05:33:37.499501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.499330 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" event={"ID":"0e413931-a15a-4fb2-82ef-4e0096717c33","Type":"ContainerDied","Data":"a8ddaa1951425ea4f8c5a558997284c641f84e111f873509902044c689ff91d1"} Oct 30 05:33:37.499501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.499436 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" event={"ID":"0e413931-a15a-4fb2-82ef-4e0096717c33","Type":"ContainerDied","Data":"4f5d19c0ffdae0bb92941653baad9068c9a94d154f50a258b42e2ee5092f8f49"} Oct 30 05:33:37.499501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.499485 6587 scope.go:122] "RemoveContainer" containerID="a8ddaa1951425ea4f8c5a558997284c641f84e111f873509902044c689ff91d1" Oct 30 05:33:37.499720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.499700 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx" Oct 30 05:33:37.507446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.507413 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" event={"ID":"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd","Type":"ContainerStarted","Data":"585389339da7e137c4eceac0928f52d5422622baae1aaec679eb2dc78b8e6b88"} Oct 30 05:33:37.572107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.572083 6587 scope.go:122] "RemoveContainer" containerID="a8ddaa1951425ea4f8c5a558997284c641f84e111f873509902044c689ff91d1" Oct 30 05:33:37.572521 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:37.572495 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ddaa1951425ea4f8c5a558997284c641f84e111f873509902044c689ff91d1\": container with ID starting with a8ddaa1951425ea4f8c5a558997284c641f84e111f873509902044c689ff91d1 not found: ID does not exist" containerID="a8ddaa1951425ea4f8c5a558997284c641f84e111f873509902044c689ff91d1" Oct 30 05:33:37.572574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.572530 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ddaa1951425ea4f8c5a558997284c641f84e111f873509902044c689ff91d1"} err="failed to get container status \"a8ddaa1951425ea4f8c5a558997284c641f84e111f873509902044c689ff91d1\": rpc error: code = NotFound desc = could not find container \"a8ddaa1951425ea4f8c5a558997284c641f84e111f873509902044c689ff91d1\": container with ID starting with a8ddaa1951425ea4f8c5a558997284c641f84e111f873509902044c689ff91d1 not found: ID does not exist" Oct 30 05:33:37.590223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.590171 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" podStartSLOduration=2.5901578130000003 podStartE2EDuration="2.590157813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:37.585555569 +0000 UTC m=+2942.113322357" watchObservedRunningTime="2026-10-30 05:33:37.590157813 +0000 UTC m=+2942.117924599" Oct 30 05:33:37.638674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.638638 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx"] Oct 30 05:33:37.650827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:37.650798 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx"] Oct 30 05:33:38.290413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.290377 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w"] Oct 30 05:33:38.291854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.291825 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0e413931-a15a-4fb2-82ef-4e0096717c33" containerName="istio-proxy" Oct 30 05:33:38.291854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.291855 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e413931-a15a-4fb2-82ef-4e0096717c33" containerName="istio-proxy" Oct 30 05:33:38.292147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.292132 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e413931-a15a-4fb2-82ef-4e0096717c33" containerName="istio-proxy" Oct 30 05:33:38.297781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.297760 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.337384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.337321 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7226a8-openshift-default-dockercfg-xgvc8\"" Oct 30 05:33:38.359854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.357548 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w"] Oct 30 05:33:38.382433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.382397 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-envoy\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.382648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.382615 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-certs\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.382724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.382685 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-credential-socket\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.382811 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.382792 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-socket\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.382877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.382823 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-data\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.382938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.382882 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjmgz\" (UniqueName: \"kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-kube-api-access-qjmgz\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.382938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.382921 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f360336e-96e3-41c3-b483-aecf620a959f-istiod-ca-cert\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.383187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.383159 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-istio-token\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.383293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.383272 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f360336e-96e3-41c3-b483-aecf620a959f-istio-podinfo\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.514588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.490321 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-socket\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.514588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.490386 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-data\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.514588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.490452 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjmgz\" (UniqueName: \"kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-kube-api-access-qjmgz\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.514588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.490494 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f360336e-96e3-41c3-b483-aecf620a959f-istiod-ca-cert\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.514588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.490629 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-istio-token\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.514588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.490718 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f360336e-96e3-41c3-b483-aecf620a959f-istio-podinfo\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.514588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.490807 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-envoy\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.514588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.490975 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-certs\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.514588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.491036 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-credential-socket\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.514588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.491576 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-credential-socket\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.516343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.516313 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-socket\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.525766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.516759 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-data\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.525977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.517430 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f360336e-96e3-41c3-b483-aecf620a959f-istiod-ca-cert\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.526076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.520669 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-certs\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.632390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.632160 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f360336e-96e3-41c3-b483-aecf620a959f-istio-podinfo\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.650382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.649108 6587 generic.go:356] "Generic (PLEG): container finished" podID="bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd" containerID="585389339da7e137c4eceac0928f52d5422622baae1aaec679eb2dc78b8e6b88" exitCode=0 Oct 30 05:33:38.650382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.649430 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" event={"ID":"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd","Type":"ContainerDied","Data":"585389339da7e137c4eceac0928f52d5422622baae1aaec679eb2dc78b8e6b88"} Oct 30 05:33:38.668404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.663704 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-envoy\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.668404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.664419 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjmgz\" (UniqueName: \"kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-kube-api-access-qjmgz\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.668404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.664649 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-istio-token\") pod \"e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:38.918878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:38.918787 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:39.251853 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:33:39.251821 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf360336e_96e3_41c3_b483_aecf620a959f.slice/crio-928050fd4fd1667822b4fb15f8fde6cde08cd74ad025926c07b88c94c2725a8f WatchSource:0}: Error finding container 928050fd4fd1667822b4fb15f8fde6cde08cd74ad025926c07b88c94c2725a8f: Status 404 returned error can't find the container with id 928050fd4fd1667822b4fb15f8fde6cde08cd74ad025926c07b88c94c2725a8f Oct 30 05:33:39.260755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:39.255765 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:39.260755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:39.255910 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:39.260755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:39.255959 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:39.260755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:39.258215 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w"] Oct 30 05:33:39.587915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:39.587873 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e413931-a15a-4fb2-82ef-4e0096717c33" path="/var/lib/kubelet/pods/0e413931-a15a-4fb2-82ef-4e0096717c33/volumes" Oct 30 05:33:39.664628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:39.664592 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" event={"ID":"f360336e-96e3-41c3-b483-aecf620a959f","Type":"ContainerStarted","Data":"fcbc9da06e47375d5ca044c36e00ab5d6c3306744c2444d01175dcd759dea8e9"} Oct 30 05:33:39.665104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:39.664656 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" event={"ID":"f360336e-96e3-41c3-b483-aecf620a959f","Type":"ContainerStarted","Data":"928050fd4fd1667822b4fb15f8fde6cde08cd74ad025926c07b88c94c2725a8f"} Oct 30 05:33:39.716623 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:39.716569 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" podStartSLOduration=1.716554148 podStartE2EDuration="1.716554148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:39.71223288 +0000 UTC m=+2944.239999665" watchObservedRunningTime="2026-10-30 05:33:39.716554148 +0000 UTC m=+2944.244320933" Oct 30 05:33:39.923501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:39.923472 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:39.965332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:39.965297 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" Oct 30 05:33:40.022281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:40.022245 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-openshift-ca\") pod \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\" (UID: \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\") " Oct 30 05:33:40.022482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:40.022313 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-kube-api-access-qnshk\" (UniqueName: \"kubernetes.io/projected/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-kube-api-access-qnshk\") pod \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\" (UID: \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\") " Oct 30 05:33:40.022547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:40.022483 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-xks-ca\") pod \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\" (UID: \"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd\") " Oct 30 05:33:40.023289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:40.022970 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-openshift-ca" pod "bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd" (UID: "bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:40.024970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:40.023663 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:40.025977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:40.025951 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-xks-ca" pod "bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd" (UID: "bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:40.027124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:40.027091 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-kube-api-access-qnshk" pod "bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd" (UID: "bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd"). InnerVolumeSpecName "kube-api-access-qnshk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:40.126136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:40.126097 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnshk\" (UniqueName: \"kubernetes.io/projected/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-kube-api-access-qnshk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:40.126136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:40.126134 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:40.777246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:40.772626 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" Oct 30 05:33:40.783431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:40.780813 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb" event={"ID":"bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd","Type":"ContainerDied","Data":"83d28f4d31e63ec032b243cafe73c814c9c16a70e9ecba87c00d078740342b43"} Oct 30 05:33:40.783431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:40.780892 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83d28f4d31e63ec032b243cafe73c814c9c16a70e9ecba87c00d078740342b43" Oct 30 05:33:40.974743 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:40.974704 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:41.782548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:41.782519 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:41.783725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:41.783702 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:33:42.950378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:42.947697 6587 scope.go:122] "RemoveContainer" containerID="966d35b84119d5597bf8402afb586458fe9aad99f91500da47a28753fb30c258" Oct 30 05:33:43.066025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.065626 6587 scope.go:122] "RemoveContainer" containerID="2a88aba419aaaaeec25cd83733da58d911aa9c139ae44f085c5369e2b6b717fc" Oct 30 05:33:43.456624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.456134 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" Oct 30 05:33:43.556025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.555989 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" Oct 30 05:33:43.593416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.592906 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d591ba5b-cdc7-4f23-85c6-28418b0abefe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d591ba5b-cdc7-4f23-85c6-28418b0abefe-plugins-config-volume\") pod \"d591ba5b-cdc7-4f23-85c6-28418b0abefe\" (UID: \"d591ba5b-cdc7-4f23-85c6-28418b0abefe\") " Oct 30 05:33:43.593416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.593159 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d591ba5b-cdc7-4f23-85c6-28418b0abefe-kube-api-access-vc6qs\" (UniqueName: \"kubernetes.io/projected/d591ba5b-cdc7-4f23-85c6-28418b0abefe-kube-api-access-vc6qs\") pod \"d591ba5b-cdc7-4f23-85c6-28418b0abefe\" (UID: \"d591ba5b-cdc7-4f23-85c6-28418b0abefe\") " Oct 30 05:33:43.594848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.594789 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d591ba5b-cdc7-4f23-85c6-28418b0abefe-plugins-config-volume" pod "d591ba5b-cdc7-4f23-85c6-28418b0abefe" (UID: "d591ba5b-cdc7-4f23-85c6-28418b0abefe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:43.601425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.601392 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d591ba5b-cdc7-4f23-85c6-28418b0abefe-kube-api-access-vc6qs" pod "d591ba5b-cdc7-4f23-85c6-28418b0abefe" (UID: "d591ba5b-cdc7-4f23-85c6-28418b0abefe"). InnerVolumeSpecName "kube-api-access-vc6qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:43.695976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.695938 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d11379-6d13-4b4b-a36b-4d418b490323-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d11379-6d13-4b4b-a36b-4d418b490323-plugins-config-volume\") pod \"46d11379-6d13-4b4b-a36b-4d418b490323\" (UID: \"46d11379-6d13-4b4b-a36b-4d418b490323\") " Oct 30 05:33:43.696176 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.696106 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d11379-6d13-4b4b-a36b-4d418b490323-kube-api-access-pbmvk\" (UniqueName: \"kubernetes.io/projected/46d11379-6d13-4b4b-a36b-4d418b490323-kube-api-access-pbmvk\") pod \"46d11379-6d13-4b4b-a36b-4d418b490323\" (UID: \"46d11379-6d13-4b4b-a36b-4d418b490323\") " Oct 30 05:33:43.696764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.696727 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d11379-6d13-4b4b-a36b-4d418b490323-plugins-config-volume" pod "46d11379-6d13-4b4b-a36b-4d418b490323" (UID: "46d11379-6d13-4b4b-a36b-4d418b490323"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:43.697280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.697241 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc6qs\" (UniqueName: \"kubernetes.io/projected/d591ba5b-cdc7-4f23-85c6-28418b0abefe-kube-api-access-vc6qs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:43.697280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.697268 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d11379-6d13-4b4b-a36b-4d418b490323-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:43.697529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.697283 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d591ba5b-cdc7-4f23-85c6-28418b0abefe-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:43.699070 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.699036 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d11379-6d13-4b4b-a36b-4d418b490323-kube-api-access-pbmvk" pod "46d11379-6d13-4b4b-a36b-4d418b490323" (UID: "46d11379-6d13-4b4b-a36b-4d418b490323"). InnerVolumeSpecName "kube-api-access-pbmvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:43.798908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.798871 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbmvk\" (UniqueName: \"kubernetes.io/projected/46d11379-6d13-4b4b-a36b-4d418b490323-kube-api-access-pbmvk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:43.961012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.960978 6587 generic.go:356] "Generic (PLEG): container finished" podID="46d11379-6d13-4b4b-a36b-4d418b490323" containerID="70940e2c3d864d54730995d4b39b65836deabb4454d8d72e326b65aa8a5ad835" exitCode=0 Oct 30 05:33:43.961603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.961043 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" Oct 30 05:33:43.961768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.961706 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" event={"ID":"46d11379-6d13-4b4b-a36b-4d418b490323","Type":"ContainerDied","Data":"70940e2c3d864d54730995d4b39b65836deabb4454d8d72e326b65aa8a5ad835"} Oct 30 05:33:43.961974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.961947 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk" event={"ID":"46d11379-6d13-4b4b-a36b-4d418b490323","Type":"ContainerDied","Data":"b5c9b8e51974fff9840c69c8cdeab46a8ab4a32715b4c2430dda3785c9b35b6f"} Oct 30 05:33:43.963333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.962005 6587 scope.go:122] "RemoveContainer" containerID="70940e2c3d864d54730995d4b39b65836deabb4454d8d72e326b65aa8a5ad835" Oct 30 05:33:43.963333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.962941 6587 generic.go:356] "Generic (PLEG): container finished" podID="d591ba5b-cdc7-4f23-85c6-28418b0abefe" containerID="e1cf2dd39fd53f68e4c349d1007fe123ce6e5fad721c1db4be4aa7a1e4b1a7e3" exitCode=0 Oct 30 05:33:43.963333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.963034 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" Oct 30 05:33:43.963333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.963044 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" event={"ID":"d591ba5b-cdc7-4f23-85c6-28418b0abefe","Type":"ContainerDied","Data":"e1cf2dd39fd53f68e4c349d1007fe123ce6e5fad721c1db4be4aa7a1e4b1a7e3"} Oct 30 05:33:43.963333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.963096 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj" event={"ID":"d591ba5b-cdc7-4f23-85c6-28418b0abefe","Type":"ContainerDied","Data":"ae1cd1abd3bef7971cec0101eee4479ac00e7e106a813fbf36cf92153553eb2a"} Oct 30 05:33:43.977426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.977325 6587 scope.go:122] "RemoveContainer" containerID="70940e2c3d864d54730995d4b39b65836deabb4454d8d72e326b65aa8a5ad835" Oct 30 05:33:43.977656 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:43.977633 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70940e2c3d864d54730995d4b39b65836deabb4454d8d72e326b65aa8a5ad835\": container with ID starting with 70940e2c3d864d54730995d4b39b65836deabb4454d8d72e326b65aa8a5ad835 not found: ID does not exist" containerID="70940e2c3d864d54730995d4b39b65836deabb4454d8d72e326b65aa8a5ad835" Oct 30 05:33:43.977736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.977662 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70940e2c3d864d54730995d4b39b65836deabb4454d8d72e326b65aa8a5ad835"} err="failed to get container status \"70940e2c3d864d54730995d4b39b65836deabb4454d8d72e326b65aa8a5ad835\": rpc error: code = NotFound desc = could not find container \"70940e2c3d864d54730995d4b39b65836deabb4454d8d72e326b65aa8a5ad835\": container with ID starting with 70940e2c3d864d54730995d4b39b65836deabb4454d8d72e326b65aa8a5ad835 not found: ID does not exist" Oct 30 05:33:43.977736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.977689 6587 scope.go:122] "RemoveContainer" containerID="e1cf2dd39fd53f68e4c349d1007fe123ce6e5fad721c1db4be4aa7a1e4b1a7e3" Oct 30 05:33:43.991880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.991859 6587 scope.go:122] "RemoveContainer" containerID="e1cf2dd39fd53f68e4c349d1007fe123ce6e5fad721c1db4be4aa7a1e4b1a7e3" Oct 30 05:33:43.992179 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:43.992154 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1cf2dd39fd53f68e4c349d1007fe123ce6e5fad721c1db4be4aa7a1e4b1a7e3\": container with ID starting with e1cf2dd39fd53f68e4c349d1007fe123ce6e5fad721c1db4be4aa7a1e4b1a7e3 not found: ID does not exist" containerID="e1cf2dd39fd53f68e4c349d1007fe123ce6e5fad721c1db4be4aa7a1e4b1a7e3" Oct 30 05:33:43.992237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:43.992182 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1cf2dd39fd53f68e4c349d1007fe123ce6e5fad721c1db4be4aa7a1e4b1a7e3"} err="failed to get container status \"e1cf2dd39fd53f68e4c349d1007fe123ce6e5fad721c1db4be4aa7a1e4b1a7e3\": rpc error: code = NotFound desc = could not find container \"e1cf2dd39fd53f68e4c349d1007fe123ce6e5fad721c1db4be4aa7a1e4b1a7e3\": container with ID starting with e1cf2dd39fd53f68e4c349d1007fe123ce6e5fad721c1db4be4aa7a1e4b1a7e3 not found: ID does not exist" Oct 30 05:33:44.102060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.102025 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj"] Oct 30 05:33:44.105565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.105527 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj"] Oct 30 05:33:44.150941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.150832 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk"] Oct 30 05:33:44.158520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.158494 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk"] Oct 30 05:33:44.600614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.600583 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn"] Oct 30 05:33:44.601642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.601622 6587 cpu_manager.go:414] "containerMap: removing container" podUID="bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd" containerName="revoke-keys" Oct 30 05:33:44.601705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.601643 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd" containerName="revoke-keys" Oct 30 05:33:44.601705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.601667 6587 cpu_manager.go:414] "containerMap: removing container" podUID="46d11379-6d13-4b4b-a36b-4d418b490323" containerName="payload-pre-processing" Oct 30 05:33:44.601705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.601672 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d11379-6d13-4b4b-a36b-4d418b490323" containerName="payload-pre-processing" Oct 30 05:33:44.601705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.601697 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d591ba5b-cdc7-4f23-85c6-28418b0abefe" containerName="payload-processing" Oct 30 05:33:44.601705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.601703 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d591ba5b-cdc7-4f23-85c6-28418b0abefe" containerName="payload-processing" Oct 30 05:33:44.601871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.601832 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd" containerName="revoke-keys" Oct 30 05:33:44.601871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.601845 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d11379-6d13-4b4b-a36b-4d418b490323" containerName="payload-pre-processing" Oct 30 05:33:44.601871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.601855 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d591ba5b-cdc7-4f23-85c6-28418b0abefe" containerName="payload-processing" Oct 30 05:33:44.607454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.607432 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" Oct 30 05:33:44.614886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.614861 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn\" (UID: \"1bb9624f-f9d9-4b85-89d6-87e470ff40a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" Oct 30 05:33:44.615207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.615184 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgszn\" (UniqueName: \"kubernetes.io/projected/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-kube-api-access-pgszn\") pod \"payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn\" (UID: \"1bb9624f-f9d9-4b85-89d6-87e470ff40a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" Oct 30 05:33:44.625723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.625692 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7226a8-dockercfg-dzr7t\"" Oct 30 05:33:44.641380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.641144 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7226a8\"" Oct 30 05:33:44.659099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.659060 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn"] Oct 30 05:33:44.789383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.788400 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgszn\" (UniqueName: \"kubernetes.io/projected/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-kube-api-access-pgszn\") pod \"payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn\" (UID: \"1bb9624f-f9d9-4b85-89d6-87e470ff40a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" Oct 30 05:33:44.789383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.788605 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn\" (UID: \"1bb9624f-f9d9-4b85-89d6-87e470ff40a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" Oct 30 05:33:44.801380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.801067 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn\" (UID: \"1bb9624f-f9d9-4b85-89d6-87e470ff40a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" Oct 30 05:33:44.969466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:44.959557 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8"] Oct 30 05:33:45.020384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.018421 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" Oct 30 05:33:45.111381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.101999 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8\" (UID: \"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" Oct 30 05:33:45.111381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.102294 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p58g\" (UniqueName: \"kubernetes.io/projected/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-kube-api-access-7p58g\") pod \"payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8\" (UID: \"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" Oct 30 05:33:45.164463 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.163724 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgszn\" (UniqueName: \"kubernetes.io/projected/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-kube-api-access-pgszn\") pod \"payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn\" (UID: \"1bb9624f-f9d9-4b85-89d6-87e470ff40a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" Oct 30 05:33:45.186662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.186582 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8"] Oct 30 05:33:45.215068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.215032 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8\" (UID: \"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" Oct 30 05:33:45.215348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.215323 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p58g\" (UniqueName: \"kubernetes.io/projected/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-kube-api-access-7p58g\") pod \"payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8\" (UID: \"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" Oct 30 05:33:45.228244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.228175 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8\" (UID: \"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" Oct 30 05:33:45.255651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.241029 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p58g\" (UniqueName: \"kubernetes.io/projected/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-kube-api-access-7p58g\") pod \"payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8\" (UID: \"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" Oct 30 05:33:45.259786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.259756 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" Oct 30 05:33:45.504639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.504106 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" Oct 30 05:33:45.592476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.592442 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d11379-6d13-4b4b-a36b-4d418b490323" path="/var/lib/kubelet/pods/46d11379-6d13-4b4b-a36b-4d418b490323/volumes" Oct 30 05:33:45.593383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.593347 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d591ba5b-cdc7-4f23-85c6-28418b0abefe" path="/var/lib/kubelet/pods/d591ba5b-cdc7-4f23-85c6-28418b0abefe/volumes" Oct 30 05:33:45.661416 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:33:45.661176 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb9624f_f9d9_4b85_89d6_87e470ff40a1.slice/crio-8cbbe3d1a1095490a2347b8b8a858e1b70d9d2c67552d702522521f00c1414b9 WatchSource:0}: Error finding container 8cbbe3d1a1095490a2347b8b8a858e1b70d9d2c67552d702522521f00c1414b9: Status 404 returned error can't find the container with id 8cbbe3d1a1095490a2347b8b8a858e1b70d9d2c67552d702522521f00c1414b9 Oct 30 05:33:45.673427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.673314 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn"] Oct 30 05:33:45.789605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.789521 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs"] Oct 30 05:33:45.796926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.796902 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" Oct 30 05:33:45.814338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.814313 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:33:45.814531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.814398 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:33:45.862293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.862261 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs"] Oct 30 05:33:45.899589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.899562 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8"] Oct 30 05:33:45.977749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.977718 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2690b27e-vdxcs\" (UID: \"009cdc21-8578-4132-bdfa-55d149c7984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" Oct 30 05:33:45.978180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.977866 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twhsw\" (UniqueName: \"kubernetes.io/projected/009cdc21-8578-4132-bdfa-55d149c7984b-kube-api-access-twhsw\") pod \"maas-api-revoke-sub-2690b27e-vdxcs\" (UID: \"009cdc21-8578-4132-bdfa-55d149c7984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" Oct 30 05:33:45.978180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:45.977965 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2690b27e-vdxcs\" (UID: \"009cdc21-8578-4132-bdfa-55d149c7984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" Oct 30 05:33:46.079741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.079697 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2690b27e-vdxcs\" (UID: \"009cdc21-8578-4132-bdfa-55d149c7984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" Oct 30 05:33:46.079939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.079809 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twhsw\" (UniqueName: \"kubernetes.io/projected/009cdc21-8578-4132-bdfa-55d149c7984b-kube-api-access-twhsw\") pod \"maas-api-revoke-sub-2690b27e-vdxcs\" (UID: \"009cdc21-8578-4132-bdfa-55d149c7984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" Oct 30 05:33:46.079939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.079881 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2690b27e-vdxcs\" (UID: \"009cdc21-8578-4132-bdfa-55d149c7984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" Oct 30 05:33:46.080587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.080559 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2690b27e-vdxcs\" (UID: \"009cdc21-8578-4132-bdfa-55d149c7984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" Oct 30 05:33:46.082540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.082515 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2690b27e-vdxcs\" (UID: \"009cdc21-8578-4132-bdfa-55d149c7984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" Oct 30 05:33:46.106711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.106688 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twhsw\" (UniqueName: \"kubernetes.io/projected/009cdc21-8578-4132-bdfa-55d149c7984b-kube-api-access-twhsw\") pod \"maas-api-revoke-sub-2690b27e-vdxcs\" (UID: \"009cdc21-8578-4132-bdfa-55d149c7984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" Oct 30 05:33:46.114751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.114729 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" Oct 30 05:33:46.172426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.172395 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm"] Oct 30 05:33:46.205983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.205816 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z"] Oct 30 05:33:46.263971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.232010 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" event={"ID":"1bb9624f-f9d9-4b85-89d6-87e470ff40a1","Type":"ContainerStarted","Data":"3ad58e74672e4c7a11179547eaf170b50cc04f1a045e38d9c911587a651b65a0"} Oct 30 05:33:46.263971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.232081 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" event={"ID":"1bb9624f-f9d9-4b85-89d6-87e470ff40a1","Type":"ContainerStarted","Data":"8cbbe3d1a1095490a2347b8b8a858e1b70d9d2c67552d702522521f00c1414b9"} Oct 30 05:33:46.263971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.233139 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" Oct 30 05:33:46.263971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.236973 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" event={"ID":"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669","Type":"ContainerStarted","Data":"caa86687cd6557f309cb5b4f613ed9ad64927db02a49daf4d3ff67bab39aeb5d"} Oct 30 05:33:46.263971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.237023 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" event={"ID":"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669","Type":"ContainerStarted","Data":"92ce856f3afd4adca3a62ea72d8157df0ea2a987002dae80efbf47462d62896a"} Oct 30 05:33:46.263971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.237995 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" Oct 30 05:33:46.311481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.309822 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" podStartSLOduration=2.309803791 podStartE2EDuration="2.309803791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:46.309602112 +0000 UTC m=+2950.837368910" watchObservedRunningTime="2026-10-30 05:33:46.309803791 +0000 UTC m=+2950.837570578" Oct 30 05:33:46.463441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.460790 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" podStartSLOduration=2.460769405 podStartE2EDuration="2.460769405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:46.415150924 +0000 UTC m=+2950.942917712" watchObservedRunningTime="2026-10-30 05:33:46.460769405 +0000 UTC m=+2950.988536192" Oct 30 05:33:46.508875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:46.508842 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs"] Oct 30 05:33:47.292573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:47.292485 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" event={"ID":"009cdc21-8578-4132-bdfa-55d149c7984b","Type":"ContainerStarted","Data":"08e74218719dc6984bd36665130d8c74a24c56aaab7ba29588e50fcf69178419"} Oct 30 05:33:47.486243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:47.486196 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" podUID="ae125d46-b3a6-4d61-b977-cecd363489be" containerName="main" containerID="cri-o://30e6532df37a2a495beb674bd7c42fc427e481dd7a86805b3b87852a19c60cd2" gracePeriod=45 Oct 30 05:33:47.510222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:47.510196 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" Oct 30 05:33:47.510328 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:47.510266 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" Oct 30 05:33:48.057703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.056850 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:33:48.083921 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.083887 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-home\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-home\") pod \"ae125d46-b3a6-4d61-b977-cecd363489be\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " Oct 30 05:33:48.084076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.083948 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae125d46-b3a6-4d61-b977-cecd363489be-kube-api-access-rvl74\" (UniqueName: \"kubernetes.io/projected/ae125d46-b3a6-4d61-b977-cecd363489be-kube-api-access-rvl74\") pod \"ae125d46-b3a6-4d61-b977-cecd363489be\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " Oct 30 05:33:48.084076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.084021 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-model-cache\") pod \"ae125d46-b3a6-4d61-b977-cecd363489be\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " Oct 30 05:33:48.084265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.084103 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae125d46-b3a6-4d61-b977-cecd363489be-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae125d46-b3a6-4d61-b977-cecd363489be-tls-certs\") pod \"ae125d46-b3a6-4d61-b977-cecd363489be\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " Oct 30 05:33:48.084824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.084781 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-model-cache" pod "ae125d46-b3a6-4d61-b977-cecd363489be" (UID: "ae125d46-b3a6-4d61-b977-cecd363489be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:48.084928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.084862 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-dshm\") pod \"ae125d46-b3a6-4d61-b977-cecd363489be\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " Oct 30 05:33:48.085157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.085060 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-tmp-dir\") pod \"ae125d46-b3a6-4d61-b977-cecd363489be\" (UID: \"ae125d46-b3a6-4d61-b977-cecd363489be\") " Oct 30 05:33:48.086595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.085622 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-tmp-dir" pod "ae125d46-b3a6-4d61-b977-cecd363489be" (UID: "ae125d46-b3a6-4d61-b977-cecd363489be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:48.086595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.086480 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:48.086595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.086507 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:48.086595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.086505 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-home" pod "ae125d46-b3a6-4d61-b977-cecd363489be" (UID: "ae125d46-b3a6-4d61-b977-cecd363489be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:48.088784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.088762 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae125d46-b3a6-4d61-b977-cecd363489be-tls-certs" pod "ae125d46-b3a6-4d61-b977-cecd363489be" (UID: "ae125d46-b3a6-4d61-b977-cecd363489be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:48.088887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.088803 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-dshm" pod "ae125d46-b3a6-4d61-b977-cecd363489be" (UID: "ae125d46-b3a6-4d61-b977-cecd363489be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:48.089109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.089076 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae125d46-b3a6-4d61-b977-cecd363489be-kube-api-access-rvl74" pod "ae125d46-b3a6-4d61-b977-cecd363489be" (UID: "ae125d46-b3a6-4d61-b977-cecd363489be"). InnerVolumeSpecName "kube-api-access-rvl74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:48.189029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.188995 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae125d46-b3a6-4d61-b977-cecd363489be-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:48.189029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.189026 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:48.189029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.189035 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae125d46-b3a6-4d61-b977-cecd363489be-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:48.189288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.189044 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvl74\" (UniqueName: \"kubernetes.io/projected/ae125d46-b3a6-4d61-b977-cecd363489be-kube-api-access-rvl74\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:48.302514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.302480 6587 generic.go:356] "Generic (PLEG): container finished" podID="ae125d46-b3a6-4d61-b977-cecd363489be" containerID="30e6532df37a2a495beb674bd7c42fc427e481dd7a86805b3b87852a19c60cd2" exitCode=0 Oct 30 05:33:48.302891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.302536 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" Oct 30 05:33:48.302891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.302566 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" event={"ID":"ae125d46-b3a6-4d61-b977-cecd363489be","Type":"ContainerDied","Data":"30e6532df37a2a495beb674bd7c42fc427e481dd7a86805b3b87852a19c60cd2"} Oct 30 05:33:48.302891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.302610 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k" event={"ID":"ae125d46-b3a6-4d61-b977-cecd363489be","Type":"ContainerDied","Data":"75959c1acfc84358d405b3b1cffa769a3d9da4464869d433846b49550c69663d"} Oct 30 05:33:48.302891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.302628 6587 scope.go:122] "RemoveContainer" containerID="30e6532df37a2a495beb674bd7c42fc427e481dd7a86805b3b87852a19c60cd2" Oct 30 05:33:48.304969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.304948 6587 generic.go:356] "Generic (PLEG): container finished" podID="009cdc21-8578-4132-bdfa-55d149c7984b" containerID="a50e8f21b0ff89224d0e6fefb7b563a49f0a06ed52d8f05e935c929efa7e9f8a" exitCode=0 Oct 30 05:33:48.305139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.305117 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" event={"ID":"009cdc21-8578-4132-bdfa-55d149c7984b","Type":"ContainerDied","Data":"a50e8f21b0ff89224d0e6fefb7b563a49f0a06ed52d8f05e935c929efa7e9f8a"} Oct 30 05:33:48.344615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.344586 6587 scope.go:122] "RemoveContainer" containerID="30e6532df37a2a495beb674bd7c42fc427e481dd7a86805b3b87852a19c60cd2" Oct 30 05:33:48.346737 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:48.346705 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e6532df37a2a495beb674bd7c42fc427e481dd7a86805b3b87852a19c60cd2\": container with ID starting with 30e6532df37a2a495beb674bd7c42fc427e481dd7a86805b3b87852a19c60cd2 not found: ID does not exist" containerID="30e6532df37a2a495beb674bd7c42fc427e481dd7a86805b3b87852a19c60cd2" Oct 30 05:33:48.350442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.350404 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e6532df37a2a495beb674bd7c42fc427e481dd7a86805b3b87852a19c60cd2"} err="failed to get container status \"30e6532df37a2a495beb674bd7c42fc427e481dd7a86805b3b87852a19c60cd2\": rpc error: code = NotFound desc = could not find container \"30e6532df37a2a495beb674bd7c42fc427e481dd7a86805b3b87852a19c60cd2\": container with ID starting with 30e6532df37a2a495beb674bd7c42fc427e481dd7a86805b3b87852a19c60cd2 not found: ID does not exist" Oct 30 05:33:48.383284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.383243 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k"] Oct 30 05:33:48.389477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:48.389448 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-35a94e/auth-test-model-35a94e-kserve-665d6fd8d9-kbg9k"] Oct 30 05:33:49.532527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:49.532493 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" Oct 30 05:33:49.596331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:49.596240 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae125d46-b3a6-4d61-b977-cecd363489be" path="/var/lib/kubelet/pods/ae125d46-b3a6-4d61-b977-cecd363489be/volumes" Oct 30 05:33:49.631183 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:49.631146 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-xks-ca\") pod \"009cdc21-8578-4132-bdfa-55d149c7984b\" (UID: \"009cdc21-8578-4132-bdfa-55d149c7984b\") " Oct 30 05:33:49.631568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:49.631545 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009cdc21-8578-4132-bdfa-55d149c7984b-kube-api-access-twhsw\" (UniqueName: \"kubernetes.io/projected/009cdc21-8578-4132-bdfa-55d149c7984b-kube-api-access-twhsw\") pod \"009cdc21-8578-4132-bdfa-55d149c7984b\" (UID: \"009cdc21-8578-4132-bdfa-55d149c7984b\") " Oct 30 05:33:49.631828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:49.631812 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-openshift-ca\") pod \"009cdc21-8578-4132-bdfa-55d149c7984b\" (UID: \"009cdc21-8578-4132-bdfa-55d149c7984b\") " Oct 30 05:33:49.633159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:49.632261 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-openshift-ca" pod "009cdc21-8578-4132-bdfa-55d149c7984b" (UID: "009cdc21-8578-4132-bdfa-55d149c7984b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:49.641208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:49.641183 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-xks-ca" pod "009cdc21-8578-4132-bdfa-55d149c7984b" (UID: "009cdc21-8578-4132-bdfa-55d149c7984b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:49.643303 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:49.643274 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009cdc21-8578-4132-bdfa-55d149c7984b-kube-api-access-twhsw" pod "009cdc21-8578-4132-bdfa-55d149c7984b" (UID: "009cdc21-8578-4132-bdfa-55d149c7984b"). InnerVolumeSpecName "kube-api-access-twhsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:49.735484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:49.735445 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twhsw\" (UniqueName: \"kubernetes.io/projected/009cdc21-8578-4132-bdfa-55d149c7984b-kube-api-access-twhsw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:49.735484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:49.735484 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:49.735766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:49.735500 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009cdc21-8578-4132-bdfa-55d149c7984b-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:50.358380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:50.354750 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" event={"ID":"009cdc21-8578-4132-bdfa-55d149c7984b","Type":"ContainerDied","Data":"08e74218719dc6984bd36665130d8c74a24c56aaab7ba29588e50fcf69178419"} Oct 30 05:33:50.358380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:50.354817 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08e74218719dc6984bd36665130d8c74a24c56aaab7ba29588e50fcf69178419" Oct 30 05:33:50.358380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:50.354887 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs" Oct 30 05:33:51.176769 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:51.176725 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" podUID="e92a12c9-eb28-4851-b02b-f574892dfb49" containerName="payload-processing" containerID="cri-o://9797ceaef15b7f674b9aca7453396fc1caf944aacbc7a4b5eff184bb18662635" gracePeriod=25 Oct 30 05:33:51.206466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:51.206400 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" podUID="2492f676-52a6-4199-8401-36f35dd25a66" containerName="payload-pre-processing" containerID="cri-o://981c84e58bc070fc84bde94ea43ad2bcc53a41082006150e8287148b3fe2896f" gracePeriod=25 Oct 30 05:33:55.057239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:55.057195 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" podUID="e92a12c9-eb28-4851-b02b-f574892dfb49" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:33:55.057793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:55.057263 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" podUID="2492f676-52a6-4199-8401-36f35dd25a66" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:33:56.111867 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:33:56.111822 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:33:56.198768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.198731 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6"] Oct 30 05:33:56.199801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.199775 6587 cpu_manager.go:414] "containerMap: removing container" podUID="ae125d46-b3a6-4d61-b977-cecd363489be" containerName="main" Oct 30 05:33:56.199801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.199800 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae125d46-b3a6-4d61-b977-cecd363489be" containerName="main" Oct 30 05:33:56.221738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.199863 6587 cpu_manager.go:414] "containerMap: removing container" podUID="009cdc21-8578-4132-bdfa-55d149c7984b" containerName="revoke-keys" Oct 30 05:33:56.221738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.199872 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009cdc21-8578-4132-bdfa-55d149c7984b" containerName="revoke-keys" Oct 30 05:33:56.221738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.200117 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae125d46-b3a6-4d61-b977-cecd363489be" containerName="main" Oct 30 05:33:56.221738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.200141 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="009cdc21-8578-4132-bdfa-55d149c7984b" containerName="revoke-keys" Oct 30 05:33:56.323559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.323520 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6"] Oct 30 05:33:56.323755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.323645 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" Oct 30 05:33:56.342775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.342745 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:33:56.342937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.342752 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:33:56.396506 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.396432 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50ba2b02-2qqs6\" (UID: \"3e33da2d-1ccb-42af-8326-3762acfb079e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" Oct 30 05:33:56.396662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.396619 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk42b\" (UniqueName: \"kubernetes.io/projected/3e33da2d-1ccb-42af-8326-3762acfb079e-kube-api-access-fk42b\") pod \"maas-api-revoke-sub-50ba2b02-2qqs6\" (UID: \"3e33da2d-1ccb-42af-8326-3762acfb079e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" Oct 30 05:33:56.396773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.396755 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50ba2b02-2qqs6\" (UID: \"3e33da2d-1ccb-42af-8326-3762acfb079e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" Oct 30 05:33:56.498727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.498684 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50ba2b02-2qqs6\" (UID: \"3e33da2d-1ccb-42af-8326-3762acfb079e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" Oct 30 05:33:56.498926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.498837 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk42b\" (UniqueName: \"kubernetes.io/projected/3e33da2d-1ccb-42af-8326-3762acfb079e-kube-api-access-fk42b\") pod \"maas-api-revoke-sub-50ba2b02-2qqs6\" (UID: \"3e33da2d-1ccb-42af-8326-3762acfb079e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" Oct 30 05:33:56.499003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.498988 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50ba2b02-2qqs6\" (UID: \"3e33da2d-1ccb-42af-8326-3762acfb079e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" Oct 30 05:33:56.499651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.499630 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50ba2b02-2qqs6\" (UID: \"3e33da2d-1ccb-42af-8326-3762acfb079e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" Oct 30 05:33:56.553962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.553918 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50ba2b02-2qqs6\" (UID: \"3e33da2d-1ccb-42af-8326-3762acfb079e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" Oct 30 05:33:56.554497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.554470 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk42b\" (UniqueName: \"kubernetes.io/projected/3e33da2d-1ccb-42af-8326-3762acfb079e-kube-api-access-fk42b\") pod \"maas-api-revoke-sub-50ba2b02-2qqs6\" (UID: \"3e33da2d-1ccb-42af-8326-3762acfb079e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" Oct 30 05:33:56.785957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:56.785887 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" Oct 30 05:33:57.252784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:57.243157 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6"] Oct 30 05:33:57.266858 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:33:57.266780 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e33da2d_1ccb_42af_8326_3762acfb079e.slice/crio-208497a7aecd4e715ddcc87666a7b59caca4c13e421aa5f97e1566cd50171308 WatchSource:0}: Error finding container 208497a7aecd4e715ddcc87666a7b59caca4c13e421aa5f97e1566cd50171308: Status 404 returned error can't find the container with id 208497a7aecd4e715ddcc87666a7b59caca4c13e421aa5f97e1566cd50171308 Oct 30 05:33:57.434269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:57.434235 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" event={"ID":"3e33da2d-1ccb-42af-8326-3762acfb079e","Type":"ContainerStarted","Data":"208497a7aecd4e715ddcc87666a7b59caca4c13e421aa5f97e1566cd50171308"} Oct 30 05:33:58.309695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:58.309667 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" Oct 30 05:33:58.310189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:58.309735 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" Oct 30 05:33:58.457380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:58.456927 6587 generic.go:356] "Generic (PLEG): container finished" podID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerID="2e57348e3f31ad7d3b65738d60f17caddbb1ed44c802e1e3679ceac58dfca93f" exitCode=0 Oct 30 05:33:58.457380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:58.457010 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" event={"ID":"c5530d72-64d2-48b1-9f51-362f4a41b916","Type":"ContainerDied","Data":"2e57348e3f31ad7d3b65738d60f17caddbb1ed44c802e1e3679ceac58dfca93f"} Oct 30 05:33:58.466535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:58.457643 6587 scope.go:122] "RemoveContainer" containerID="2e57348e3f31ad7d3b65738d60f17caddbb1ed44c802e1e3679ceac58dfca93f" Oct 30 05:33:58.470188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:58.469953 6587 generic.go:356] "Generic (PLEG): container finished" podID="3e33da2d-1ccb-42af-8326-3762acfb079e" containerID="3d00a673f3f95c5a443fdea3bdb8b092ae1f798bc74c296b55f3f2bc49b64ed2" exitCode=0 Oct 30 05:33:58.470420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:58.470339 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" event={"ID":"3e33da2d-1ccb-42af-8326-3762acfb079e","Type":"ContainerDied","Data":"3d00a673f3f95c5a443fdea3bdb8b092ae1f798bc74c296b55f3f2bc49b64ed2"} Oct 30 05:33:58.471618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:58.471581 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:58.471813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:58.471663 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:58.471813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:58.471712 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:33:59.482539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:59.482505 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" event={"ID":"c5530d72-64d2-48b1-9f51-362f4a41b916","Type":"ContainerStarted","Data":"26120fe2eb888a7e7ba3da4f8dcec046228edd7dc692d46290c2fa82779a3335"} Oct 30 05:33:59.796486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:59.796458 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:33:59.797627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:59.797530 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Oct 30 05:33:59.797627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:59.797597 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Oct 30 05:33:59.857601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:59.857570 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" Oct 30 05:33:59.904065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:59.904031 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-openshift-ca\") pod \"3e33da2d-1ccb-42af-8326-3762acfb079e\" (UID: \"3e33da2d-1ccb-42af-8326-3762acfb079e\") " Oct 30 05:33:59.904258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:59.904152 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-xks-ca\") pod \"3e33da2d-1ccb-42af-8326-3762acfb079e\" (UID: \"3e33da2d-1ccb-42af-8326-3762acfb079e\") " Oct 30 05:33:59.904334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:59.904281 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e33da2d-1ccb-42af-8326-3762acfb079e-kube-api-access-fk42b\" (UniqueName: \"kubernetes.io/projected/3e33da2d-1ccb-42af-8326-3762acfb079e-kube-api-access-fk42b\") pod \"3e33da2d-1ccb-42af-8326-3762acfb079e\" (UID: \"3e33da2d-1ccb-42af-8326-3762acfb079e\") " Oct 30 05:33:59.904412 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:59.904391 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-openshift-ca" pod "3e33da2d-1ccb-42af-8326-3762acfb079e" (UID: "3e33da2d-1ccb-42af-8326-3762acfb079e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:59.904970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:59.904944 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:33:59.907103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:59.907078 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e33da2d-1ccb-42af-8326-3762acfb079e-kube-api-access-fk42b" pod "3e33da2d-1ccb-42af-8326-3762acfb079e" (UID: "3e33da2d-1ccb-42af-8326-3762acfb079e"). InnerVolumeSpecName "kube-api-access-fk42b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:59.907210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:33:59.907121 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-xks-ca" pod "3e33da2d-1ccb-42af-8326-3762acfb079e" (UID: "3e33da2d-1ccb-42af-8326-3762acfb079e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:34:00.006311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:00.006277 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e33da2d-1ccb-42af-8326-3762acfb079e-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:00.006311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:00.006308 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk42b\" (UniqueName: \"kubernetes.io/projected/3e33da2d-1ccb-42af-8326-3762acfb079e-kube-api-access-fk42b\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:00.494127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:00.494100 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" Oct 30 05:34:00.494655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:00.494134 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6" event={"ID":"3e33da2d-1ccb-42af-8326-3762acfb079e","Type":"ContainerDied","Data":"208497a7aecd4e715ddcc87666a7b59caca4c13e421aa5f97e1566cd50171308"} Oct 30 05:34:00.494655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:00.494185 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="208497a7aecd4e715ddcc87666a7b59caca4c13e421aa5f97e1566cd50171308" Oct 30 05:34:00.795977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:00.795895 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Oct 30 05:34:00.795977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:00.795946 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Oct 30 05:34:01.796119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:01.796082 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Oct 30 05:34:01.796575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:01.796137 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Oct 30 05:34:02.265549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:02.263517 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:34:02.795754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:02.795721 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Oct 30 05:34:02.795945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:02.795769 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Oct 30 05:34:03.795909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:03.795872 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Oct 30 05:34:03.796313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:03.795925 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Oct 30 05:34:04.883124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:04.883092 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:34:04.886723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:04.886694 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:34:05.061769 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:05.061727 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" podUID="e92a12c9-eb28-4851-b02b-f574892dfb49" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:34:05.069974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:05.069940 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" podUID="2492f676-52a6-4199-8401-36f35dd25a66" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:34:09.531210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.531170 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj"] Oct 30 05:34:09.532111 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.532095 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3e33da2d-1ccb-42af-8326-3762acfb079e" containerName="revoke-keys" Oct 30 05:34:09.532175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.532112 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e33da2d-1ccb-42af-8326-3762acfb079e" containerName="revoke-keys" Oct 30 05:34:09.532278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.532267 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e33da2d-1ccb-42af-8326-3762acfb079e" containerName="revoke-keys" Oct 30 05:34:09.536427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.536409 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" Oct 30 05:34:09.555293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.555266 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b"] Oct 30 05:34:09.561018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.560990 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" Oct 30 05:34:09.564121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.564098 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:34:09.564448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.564404 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:34:09.564621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.564596 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj"] Oct 30 05:34:09.592809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.592774 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b"] Oct 30 05:34:09.665843 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.665797 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dh2g\" (UniqueName: \"kubernetes.io/projected/8c6bfca0-77af-43f1-97be-bef4862c4c23-kube-api-access-7dh2g\") pod \"maas-api-revoke-sub-41b1538d-fd76b\" (UID: \"8c6bfca0-77af-43f1-97be-bef4862c4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" Oct 30 05:34:09.665843 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.665844 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6r78\" (UniqueName: \"kubernetes.io/projected/a84f548e-8a0f-4c30-90ba-b486f657e0b4-kube-api-access-z6r78\") pod \"maas-api-revoke-sub-0486f7c4-dmhbj\" (UID: \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" Oct 30 05:34:09.666091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.666001 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b1538d-fd76b\" (UID: \"8c6bfca0-77af-43f1-97be-bef4862c4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" Oct 30 05:34:09.666091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.666059 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b1538d-fd76b\" (UID: \"8c6bfca0-77af-43f1-97be-bef4862c4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" Oct 30 05:34:09.666218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.666206 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0486f7c4-dmhbj\" (UID: \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" Oct 30 05:34:09.666276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.666264 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0486f7c4-dmhbj\" (UID: \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" Oct 30 05:34:09.768000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.767962 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dh2g\" (UniqueName: \"kubernetes.io/projected/8c6bfca0-77af-43f1-97be-bef4862c4c23-kube-api-access-7dh2g\") pod \"maas-api-revoke-sub-41b1538d-fd76b\" (UID: \"8c6bfca0-77af-43f1-97be-bef4862c4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" Oct 30 05:34:09.768000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.768002 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6r78\" (UniqueName: \"kubernetes.io/projected/a84f548e-8a0f-4c30-90ba-b486f657e0b4-kube-api-access-z6r78\") pod \"maas-api-revoke-sub-0486f7c4-dmhbj\" (UID: \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" Oct 30 05:34:09.768286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.768131 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b1538d-fd76b\" (UID: \"8c6bfca0-77af-43f1-97be-bef4862c4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" Oct 30 05:34:09.768286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.768203 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b1538d-fd76b\" (UID: \"8c6bfca0-77af-43f1-97be-bef4862c4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" Oct 30 05:34:09.768520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.768343 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0486f7c4-dmhbj\" (UID: \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" Oct 30 05:34:09.768520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.768486 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0486f7c4-dmhbj\" (UID: \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" Oct 30 05:34:09.769325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.769300 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0486f7c4-dmhbj\" (UID: \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" Oct 30 05:34:09.769512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.769322 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b1538d-fd76b\" (UID: \"8c6bfca0-77af-43f1-97be-bef4862c4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" Oct 30 05:34:09.771133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.771108 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b1538d-fd76b\" (UID: \"8c6bfca0-77af-43f1-97be-bef4862c4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" Oct 30 05:34:09.771923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.771898 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0486f7c4-dmhbj\" (UID: \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" Oct 30 05:34:09.790582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.790523 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6r78\" (UniqueName: \"kubernetes.io/projected/a84f548e-8a0f-4c30-90ba-b486f657e0b4-kube-api-access-z6r78\") pod \"maas-api-revoke-sub-0486f7c4-dmhbj\" (UID: \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" Oct 30 05:34:09.790681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.790642 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dh2g\" (UniqueName: \"kubernetes.io/projected/8c6bfca0-77af-43f1-97be-bef4862c4c23-kube-api-access-7dh2g\") pod \"maas-api-revoke-sub-41b1538d-fd76b\" (UID: \"8c6bfca0-77af-43f1-97be-bef4862c4c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" Oct 30 05:34:09.852012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.851975 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" Oct 30 05:34:09.888890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:09.888852 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" Oct 30 05:34:10.334262 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:34:10.334225 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6bfca0_77af_43f1_97be_bef4862c4c23.slice/crio-2eb9bcc6925b5dea84cc82e1fb361e8c5a6521fd2fc9a3c71aaf071ef56b6731 WatchSource:0}: Error finding container 2eb9bcc6925b5dea84cc82e1fb361e8c5a6521fd2fc9a3c71aaf071ef56b6731: Status 404 returned error can't find the container with id 2eb9bcc6925b5dea84cc82e1fb361e8c5a6521fd2fc9a3c71aaf071ef56b6731 Oct 30 05:34:10.335321 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:10.335294 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b"] Oct 30 05:34:10.385455 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:34:10.385424 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84f548e_8a0f_4c30_90ba_b486f657e0b4.slice/crio-2728db09d6e75e5a3971fe5db748c97e3e57e08366e73ac73d7eff6e1753d2cc WatchSource:0}: Error finding container 2728db09d6e75e5a3971fe5db748c97e3e57e08366e73ac73d7eff6e1753d2cc: Status 404 returned error can't find the container with id 2728db09d6e75e5a3971fe5db748c97e3e57e08366e73ac73d7eff6e1753d2cc Oct 30 05:34:10.387879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:10.387852 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj"] Oct 30 05:34:10.652116 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:10.652031 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" event={"ID":"a84f548e-8a0f-4c30-90ba-b486f657e0b4","Type":"ContainerStarted","Data":"2728db09d6e75e5a3971fe5db748c97e3e57e08366e73ac73d7eff6e1753d2cc"} Oct 30 05:34:10.653940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:10.653912 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" event={"ID":"8c6bfca0-77af-43f1-97be-bef4862c4c23","Type":"ContainerStarted","Data":"2eb9bcc6925b5dea84cc82e1fb361e8c5a6521fd2fc9a3c71aaf071ef56b6731"} Oct 30 05:34:11.666957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.666918 6587 generic.go:356] "Generic (PLEG): container finished" podID="8c6bfca0-77af-43f1-97be-bef4862c4c23" containerID="ab206525956691618afe6f00447677c3e7a6a80e94fde926a86bc01a2fd6122f" exitCode=0 Oct 30 05:34:11.667409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.667161 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" event={"ID":"8c6bfca0-77af-43f1-97be-bef4862c4c23","Type":"ContainerDied","Data":"ab206525956691618afe6f00447677c3e7a6a80e94fde926a86bc01a2fd6122f"} Oct 30 05:34:11.669958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.669922 6587 generic.go:356] "Generic (PLEG): container finished" podID="e92a12c9-eb28-4851-b02b-f574892dfb49" containerID="9797ceaef15b7f674b9aca7453396fc1caf944aacbc7a4b5eff184bb18662635" exitCode=0 Oct 30 05:34:11.670478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.670005 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" event={"ID":"e92a12c9-eb28-4851-b02b-f574892dfb49","Type":"ContainerDied","Data":"9797ceaef15b7f674b9aca7453396fc1caf944aacbc7a4b5eff184bb18662635"} Oct 30 05:34:11.670580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.670513 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" event={"ID":"e92a12c9-eb28-4851-b02b-f574892dfb49","Type":"ContainerDied","Data":"24d30ba4d20773cd7cc628dabaa0d473c90020f8b9767f87137a39fdcc770295"} Oct 30 05:34:11.670580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.670538 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24d30ba4d20773cd7cc628dabaa0d473c90020f8b9767f87137a39fdcc770295" Oct 30 05:34:11.672500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.672457 6587 generic.go:356] "Generic (PLEG): container finished" podID="2492f676-52a6-4199-8401-36f35dd25a66" containerID="981c84e58bc070fc84bde94ea43ad2bcc53a41082006150e8287148b3fe2896f" exitCode=0 Oct 30 05:34:11.672624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.672568 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" event={"ID":"2492f676-52a6-4199-8401-36f35dd25a66","Type":"ContainerDied","Data":"981c84e58bc070fc84bde94ea43ad2bcc53a41082006150e8287148b3fe2896f"} Oct 30 05:34:11.674984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.674752 6587 generic.go:356] "Generic (PLEG): container finished" podID="a84f548e-8a0f-4c30-90ba-b486f657e0b4" containerID="2b03bfaba61d21185de28acccfdfb5e7bcbb6017fe0fb8d352ccfed2cd82386d" exitCode=0 Oct 30 05:34:11.674984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.674786 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" event={"ID":"a84f548e-8a0f-4c30-90ba-b486f657e0b4","Type":"ContainerDied","Data":"2b03bfaba61d21185de28acccfdfb5e7bcbb6017fe0fb8d352ccfed2cd82386d"} Oct 30 05:34:11.745869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.745789 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" Oct 30 05:34:11.754047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.754025 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" Oct 30 05:34:11.812323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.812290 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92a12c9-eb28-4851-b02b-f574892dfb49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92a12c9-eb28-4851-b02b-f574892dfb49-plugins-config-volume\") pod \"e92a12c9-eb28-4851-b02b-f574892dfb49\" (UID: \"e92a12c9-eb28-4851-b02b-f574892dfb49\") " Oct 30 05:34:11.812544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.812526 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92a12c9-eb28-4851-b02b-f574892dfb49-kube-api-access-cnw2v\" (UniqueName: \"kubernetes.io/projected/e92a12c9-eb28-4851-b02b-f574892dfb49-kube-api-access-cnw2v\") pod \"e92a12c9-eb28-4851-b02b-f574892dfb49\" (UID: \"e92a12c9-eb28-4851-b02b-f574892dfb49\") " Oct 30 05:34:11.812672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.812573 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2492f676-52a6-4199-8401-36f35dd25a66-kube-api-access-xhzlj\" (UniqueName: \"kubernetes.io/projected/2492f676-52a6-4199-8401-36f35dd25a66-kube-api-access-xhzlj\") pod \"2492f676-52a6-4199-8401-36f35dd25a66\" (UID: \"2492f676-52a6-4199-8401-36f35dd25a66\") " Oct 30 05:34:11.812738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.812685 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2492f676-52a6-4199-8401-36f35dd25a66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2492f676-52a6-4199-8401-36f35dd25a66-plugins-config-volume\") pod \"2492f676-52a6-4199-8401-36f35dd25a66\" (UID: \"2492f676-52a6-4199-8401-36f35dd25a66\") " Oct 30 05:34:11.812795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.812728 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92a12c9-eb28-4851-b02b-f574892dfb49-plugins-config-volume" pod "e92a12c9-eb28-4851-b02b-f574892dfb49" (UID: "e92a12c9-eb28-4851-b02b-f574892dfb49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:11.813117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.813092 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2492f676-52a6-4199-8401-36f35dd25a66-plugins-config-volume" pod "2492f676-52a6-4199-8401-36f35dd25a66" (UID: "2492f676-52a6-4199-8401-36f35dd25a66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:11.813543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.813522 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92a12c9-eb28-4851-b02b-f574892dfb49-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:11.813627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.813546 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2492f676-52a6-4199-8401-36f35dd25a66-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:11.814954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.814930 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2492f676-52a6-4199-8401-36f35dd25a66-kube-api-access-xhzlj" pod "2492f676-52a6-4199-8401-36f35dd25a66" (UID: "2492f676-52a6-4199-8401-36f35dd25a66"). InnerVolumeSpecName "kube-api-access-xhzlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:11.815058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.814970 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92a12c9-eb28-4851-b02b-f574892dfb49-kube-api-access-cnw2v" pod "e92a12c9-eb28-4851-b02b-f574892dfb49" (UID: "e92a12c9-eb28-4851-b02b-f574892dfb49"). InnerVolumeSpecName "kube-api-access-cnw2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:11.915877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.915833 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnw2v\" (UniqueName: \"kubernetes.io/projected/e92a12c9-eb28-4851-b02b-f574892dfb49-kube-api-access-cnw2v\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:11.915877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:11.915871 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhzlj\" (UniqueName: \"kubernetes.io/projected/2492f676-52a6-4199-8401-36f35dd25a66-kube-api-access-xhzlj\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:12.688008 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:12.687968 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" Oct 30 05:34:12.688509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:12.688012 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z" event={"ID":"2492f676-52a6-4199-8401-36f35dd25a66","Type":"ContainerDied","Data":"9c752ea4ba19c9a760b61ae8b16ddd7b75e60fa4b5196a1d2d83beae854d8d84"} Oct 30 05:34:12.688509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:12.688089 6587 scope.go:122] "RemoveContainer" containerID="981c84e58bc070fc84bde94ea43ad2bcc53a41082006150e8287148b3fe2896f" Oct 30 05:34:12.688509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:12.688298 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm" Oct 30 05:34:12.753776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:12.753744 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm"] Oct 30 05:34:12.770954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:12.770922 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm"] Oct 30 05:34:12.802170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:12.802131 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z"] Oct 30 05:34:12.807404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:12.807352 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z"] Oct 30 05:34:12.929738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:12.929712 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" Oct 30 05:34:12.978126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:12.978082 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" Oct 30 05:34:13.036918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.036886 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-xks-ca\") pod \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\" (UID: \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\") " Oct 30 05:34:13.037112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.036989 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84f548e-8a0f-4c30-90ba-b486f657e0b4-kube-api-access-z6r78\" (UniqueName: \"kubernetes.io/projected/a84f548e-8a0f-4c30-90ba-b486f657e0b4-kube-api-access-z6r78\") pod \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\" (UID: \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\") " Oct 30 05:34:13.037184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.037150 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-openshift-ca\") pod \"8c6bfca0-77af-43f1-97be-bef4862c4c23\" (UID: \"8c6bfca0-77af-43f1-97be-bef4862c4c23\") " Oct 30 05:34:13.037670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.037194 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-openshift-ca\") pod \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\" (UID: \"a84f548e-8a0f-4c30-90ba-b486f657e0b4\") " Oct 30 05:34:13.037670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.037474 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-xks-ca\") pod \"8c6bfca0-77af-43f1-97be-bef4862c4c23\" (UID: \"8c6bfca0-77af-43f1-97be-bef4862c4c23\") " Oct 30 05:34:13.037670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.037594 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-openshift-ca" pod "8c6bfca0-77af-43f1-97be-bef4862c4c23" (UID: "8c6bfca0-77af-43f1-97be-bef4862c4c23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:13.038284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.037929 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6bfca0-77af-43f1-97be-bef4862c4c23-kube-api-access-7dh2g\" (UniqueName: \"kubernetes.io/projected/8c6bfca0-77af-43f1-97be-bef4862c4c23-kube-api-access-7dh2g\") pod \"8c6bfca0-77af-43f1-97be-bef4862c4c23\" (UID: \"8c6bfca0-77af-43f1-97be-bef4862c4c23\") " Oct 30 05:34:13.038381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.038306 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-openshift-ca" pod "a84f548e-8a0f-4c30-90ba-b486f657e0b4" (UID: "a84f548e-8a0f-4c30-90ba-b486f657e0b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:13.039297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.039270 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:13.039461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.039301 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:13.044820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.044792 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6bfca0-77af-43f1-97be-bef4862c4c23-kube-api-access-7dh2g" pod "8c6bfca0-77af-43f1-97be-bef4862c4c23" (UID: "8c6bfca0-77af-43f1-97be-bef4862c4c23"). InnerVolumeSpecName "kube-api-access-7dh2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:13.044941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.044905 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-xks-ca" pod "8c6bfca0-77af-43f1-97be-bef4862c4c23" (UID: "8c6bfca0-77af-43f1-97be-bef4862c4c23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:34:13.045845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.045817 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-xks-ca" pod "a84f548e-8a0f-4c30-90ba-b486f657e0b4" (UID: "a84f548e-8a0f-4c30-90ba-b486f657e0b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:34:13.046696 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.046669 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84f548e-8a0f-4c30-90ba-b486f657e0b4-kube-api-access-z6r78" pod "a84f548e-8a0f-4c30-90ba-b486f657e0b4" (UID: "a84f548e-8a0f-4c30-90ba-b486f657e0b4"). InnerVolumeSpecName "kube-api-access-z6r78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:13.047345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.047323 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq"] Oct 30 05:34:13.053092 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.053067 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4437a4e7-j4pkq"] Oct 30 05:34:13.140838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.140807 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dh2g\" (UniqueName: \"kubernetes.io/projected/8c6bfca0-77af-43f1-97be-bef4862c4c23-kube-api-access-7dh2g\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:13.140838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.140836 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84f548e-8a0f-4c30-90ba-b486f657e0b4-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:13.141041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.140849 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6r78\" (UniqueName: \"kubernetes.io/projected/a84f548e-8a0f-4c30-90ba-b486f657e0b4-kube-api-access-z6r78\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:13.141041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.140858 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6bfca0-77af-43f1-97be-bef4862c4c23-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:13.585135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.585102 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2492f676-52a6-4199-8401-36f35dd25a66" path="/var/lib/kubelet/pods/2492f676-52a6-4199-8401-36f35dd25a66/volumes" Oct 30 05:34:13.585958 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.585712 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c237656b-07e0-44e4-acd8-5d3b3d6da565" path="/var/lib/kubelet/pods/c237656b-07e0-44e4-acd8-5d3b3d6da565/volumes" Oct 30 05:34:13.586212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.586197 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92a12c9-eb28-4851-b02b-f574892dfb49" path="/var/lib/kubelet/pods/e92a12c9-eb28-4851-b02b-f574892dfb49/volumes" Oct 30 05:34:13.701125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.701092 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" Oct 30 05:34:13.701125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.701121 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b" event={"ID":"8c6bfca0-77af-43f1-97be-bef4862c4c23","Type":"ContainerDied","Data":"2eb9bcc6925b5dea84cc82e1fb361e8c5a6521fd2fc9a3c71aaf071ef56b6731"} Oct 30 05:34:13.701752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.701171 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eb9bcc6925b5dea84cc82e1fb361e8c5a6521fd2fc9a3c71aaf071ef56b6731" Oct 30 05:34:13.704405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.704381 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" Oct 30 05:34:13.704405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.704386 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj" event={"ID":"a84f548e-8a0f-4c30-90ba-b486f657e0b4","Type":"ContainerDied","Data":"2728db09d6e75e5a3971fe5db748c97e3e57e08366e73ac73d7eff6e1753d2cc"} Oct 30 05:34:13.704665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:13.704434 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2728db09d6e75e5a3971fe5db748c97e3e57e08366e73ac73d7eff6e1753d2cc" Oct 30 05:34:25.990678 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:34:25.990639 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:34:26.036491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:26.036454 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg"] Oct 30 05:34:26.049669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:26.049640 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59046973-fnjkg"] Oct 30 05:34:27.585039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:27.585001 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2451f7cd-4642-4831-8739-78d9390298e3" path="/var/lib/kubelet/pods/2451f7cd-4642-4831-8739-78d9390298e3/volumes" Oct 30 05:34:29.051240 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:29.051198 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh"] Oct 30 05:34:29.437804 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:29.436348 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb"] Oct 30 05:34:29.444885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:29.444852 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-a914e9-x8jjb"] Oct 30 05:34:29.589627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:29.589593 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd" path="/var/lib/kubelet/pods/bd8d6cde-f5c2-4a82-90f3-c5782a03c8cd/volumes" Oct 30 05:34:36.276263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:36.276230 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:34:36.276789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:36.276607 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:34:37.569708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:37.569673 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc"] Oct 30 05:34:37.570139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:37.569880 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" podUID="ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" containerName="istio-proxy" containerID="cri-o://db841bef681b0ca4818facec32f08f8ddf866facb1ee76aae5adc30aea45ceb3" gracePeriod=30 Oct 30 05:34:38.138464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.138427 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc"] Oct 30 05:34:38.139764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.139739 6587 cpu_manager.go:414] "containerMap: removing container" podUID="8c6bfca0-77af-43f1-97be-bef4862c4c23" containerName="revoke-keys" Oct 30 05:34:38.139764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.139763 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c6bfca0-77af-43f1-97be-bef4862c4c23" containerName="revoke-keys" Oct 30 05:34:38.139947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.139836 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a84f548e-8a0f-4c30-90ba-b486f657e0b4" containerName="revoke-keys" Oct 30 05:34:38.139947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.139845 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a84f548e-8a0f-4c30-90ba-b486f657e0b4" containerName="revoke-keys" Oct 30 05:34:38.139947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.139868 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e92a12c9-eb28-4851-b02b-f574892dfb49" containerName="payload-processing" Oct 30 05:34:38.139947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.139875 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92a12c9-eb28-4851-b02b-f574892dfb49" containerName="payload-processing" Oct 30 05:34:38.139947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.139900 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2492f676-52a6-4199-8401-36f35dd25a66" containerName="payload-pre-processing" Oct 30 05:34:38.139947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.139907 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2492f676-52a6-4199-8401-36f35dd25a66" containerName="payload-pre-processing" Oct 30 05:34:38.140228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.140089 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a84f548e-8a0f-4c30-90ba-b486f657e0b4" containerName="revoke-keys" Oct 30 05:34:38.140228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.140107 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92a12c9-eb28-4851-b02b-f574892dfb49" containerName="payload-processing" Oct 30 05:34:38.140228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.140119 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2492f676-52a6-4199-8401-36f35dd25a66" containerName="payload-pre-processing" Oct 30 05:34:38.140228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.140135 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c6bfca0-77af-43f1-97be-bef4862c4c23" containerName="revoke-keys" Oct 30 05:34:38.175728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.175667 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" Oct 30 05:34:38.195072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.195045 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:34:38.209739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.209608 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:34:38.214465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.214413 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc"] Oct 30 05:34:38.216773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.216736 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc\" (UID: \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" Oct 30 05:34:38.216911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.216844 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhthm\" (UniqueName: \"kubernetes.io/projected/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-kube-api-access-dhthm\") pod \"maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc\" (UID: \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" Oct 30 05:34:38.217071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.217054 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc\" (UID: \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" Oct 30 05:34:38.319665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.319634 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc\" (UID: \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" Oct 30 05:34:38.319855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.319696 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhthm\" (UniqueName: \"kubernetes.io/projected/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-kube-api-access-dhthm\") pod \"maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc\" (UID: \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" Oct 30 05:34:38.319855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.319819 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc\" (UID: \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" Oct 30 05:34:38.320514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.320488 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc\" (UID: \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" Oct 30 05:34:38.322379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.322343 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc\" (UID: \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" Oct 30 05:34:38.343080 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.343057 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhthm\" (UniqueName: \"kubernetes.io/projected/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-kube-api-access-dhthm\") pod \"maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc\" (UID: \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" Oct 30 05:34:38.520136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.520060 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" Oct 30 05:34:38.860136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:38.860102 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc"] Oct 30 05:34:38.863791 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:34:38.863763 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca89ca8_f51a_4f71_ac27_eb7d714aaa1a.slice/crio-f881e4eb45bf32fd1d0c69ca945a8d4ce331c862b7f4658254e844014d3d001e WatchSource:0}: Error finding container f881e4eb45bf32fd1d0c69ca945a8d4ce331c862b7f4658254e844014d3d001e: Status 404 returned error can't find the container with id f881e4eb45bf32fd1d0c69ca945a8d4ce331c862b7f4658254e844014d3d001e Oct 30 05:34:39.234386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.232480 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44"] Oct 30 05:34:39.241879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.241142 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" Oct 30 05:34:39.257401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.257320 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" event={"ID":"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a","Type":"ContainerStarted","Data":"f881e4eb45bf32fd1d0c69ca945a8d4ce331c862b7f4658254e844014d3d001e"} Oct 30 05:34:39.257575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.257458 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44"] Oct 30 05:34:39.343606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.343573 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44\" (UID: \"9092860c-8c05-411c-b377-a5482e6b6d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" Oct 30 05:34:39.343876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.343851 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44\" (UID: \"9092860c-8c05-411c-b377-a5482e6b6d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" Oct 30 05:34:39.343962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.343948 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2wr6\" (UniqueName: \"kubernetes.io/projected/9092860c-8c05-411c-b377-a5482e6b6d95-kube-api-access-q2wr6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44\" (UID: \"9092860c-8c05-411c-b377-a5482e6b6d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" Oct 30 05:34:39.446234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.446204 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44\" (UID: \"9092860c-8c05-411c-b377-a5482e6b6d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" Oct 30 05:34:39.446477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.446307 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2wr6\" (UniqueName: \"kubernetes.io/projected/9092860c-8c05-411c-b377-a5482e6b6d95-kube-api-access-q2wr6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44\" (UID: \"9092860c-8c05-411c-b377-a5482e6b6d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" Oct 30 05:34:39.446572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.446551 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44\" (UID: \"9092860c-8c05-411c-b377-a5482e6b6d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" Oct 30 05:34:39.447347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.447320 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44\" (UID: \"9092860c-8c05-411c-b377-a5482e6b6d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" Oct 30 05:34:39.448943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.448917 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44\" (UID: \"9092860c-8c05-411c-b377-a5482e6b6d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" Oct 30 05:34:39.471045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.471022 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2wr6\" (UniqueName: \"kubernetes.io/projected/9092860c-8c05-411c-b377-a5482e6b6d95-kube-api-access-q2wr6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44\" (UID: \"9092860c-8c05-411c-b377-a5482e6b6d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" Oct 30 05:34:39.563372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.563321 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" Oct 30 05:34:39.891749 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:34:39.891717 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9092860c_8c05_411c_b377_a5482e6b6d95.slice/crio-80fac5c4d1645d6a2e6bad00be8054d23756efd7d0fc08e1e25f8c8a733a83ec WatchSource:0}: Error finding container 80fac5c4d1645d6a2e6bad00be8054d23756efd7d0fc08e1e25f8c8a733a83ec: Status 404 returned error can't find the container with id 80fac5c4d1645d6a2e6bad00be8054d23756efd7d0fc08e1e25f8c8a733a83ec Oct 30 05:34:39.892870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:39.892837 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44"] Oct 30 05:34:40.268237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:40.268196 6587 generic.go:356] "Generic (PLEG): container finished" podID="3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a" containerID="2ae9b87260dec2eaee468d8e01115514d3f5fc534a336ecfc0cce15bc2d9274c" exitCode=0 Oct 30 05:34:40.268469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:40.268277 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" event={"ID":"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a","Type":"ContainerDied","Data":"2ae9b87260dec2eaee468d8e01115514d3f5fc534a336ecfc0cce15bc2d9274c"} Oct 30 05:34:40.270029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:40.270005 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" event={"ID":"9092860c-8c05-411c-b377-a5482e6b6d95","Type":"ContainerStarted","Data":"80fac5c4d1645d6a2e6bad00be8054d23756efd7d0fc08e1e25f8c8a733a83ec"} Oct 30 05:34:40.959508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:40.959468 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:34:40.960000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:40.959533 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" podUID="ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:34:41.281513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:41.281477 6587 generic.go:356] "Generic (PLEG): container finished" podID="9092860c-8c05-411c-b377-a5482e6b6d95" containerID="f6febf50708ca8eb30653c0ee7955fc3e508a7db7e95c62a70cbf008ff780d7a" exitCode=0 Oct 30 05:34:41.281724 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:41.281559 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" event={"ID":"9092860c-8c05-411c-b377-a5482e6b6d95","Type":"ContainerDied","Data":"f6febf50708ca8eb30653c0ee7955fc3e508a7db7e95c62a70cbf008ff780d7a"} Oct 30 05:34:41.534664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:41.534636 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" Oct 30 05:34:41.602735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:41.602701 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-xks-ca\") pod \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\" (UID: \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\") " Oct 30 05:34:41.602932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:41.602913 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-kube-api-access-dhthm\" (UniqueName: \"kubernetes.io/projected/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-kube-api-access-dhthm\") pod \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\" (UID: \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\") " Oct 30 05:34:41.603643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:41.603001 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-openshift-ca\") pod \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\" (UID: \"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a\") " Oct 30 05:34:41.604839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:41.604807 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-openshift-ca" pod "3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a" (UID: "3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:41.605913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:41.605886 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-xks-ca" pod "3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a" (UID: "3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:34:41.614466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:41.611468 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-kube-api-access-dhthm" pod "3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a" (UID: "3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a"). InnerVolumeSpecName "kube-api-access-dhthm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:41.706531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:41.706496 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:41.706531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:41.706525 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhthm\" (UniqueName: \"kubernetes.io/projected/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-kube-api-access-dhthm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:41.706531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:41.706536 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:42.305622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.305589 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" Oct 30 05:34:42.305622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.305607 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc" event={"ID":"3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a","Type":"ContainerDied","Data":"f881e4eb45bf32fd1d0c69ca945a8d4ce331c862b7f4658254e844014d3d001e"} Oct 30 05:34:42.306142 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.305677 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f881e4eb45bf32fd1d0c69ca945a8d4ce331c862b7f4658254e844014d3d001e" Oct 30 05:34:42.520353 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.520321 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" Oct 30 05:34:42.622297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.622263 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-openshift-ca\") pod \"9092860c-8c05-411c-b377-a5482e6b6d95\" (UID: \"9092860c-8c05-411c-b377-a5482e6b6d95\") " Oct 30 05:34:42.622553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.622529 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-xks-ca\") pod \"9092860c-8c05-411c-b377-a5482e6b6d95\" (UID: \"9092860c-8c05-411c-b377-a5482e6b6d95\") " Oct 30 05:34:42.622715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.622688 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-openshift-ca" pod "9092860c-8c05-411c-b377-a5482e6b6d95" (UID: "9092860c-8c05-411c-b377-a5482e6b6d95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:42.622784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.622764 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9092860c-8c05-411c-b377-a5482e6b6d95-kube-api-access-q2wr6\" (UniqueName: \"kubernetes.io/projected/9092860c-8c05-411c-b377-a5482e6b6d95-kube-api-access-q2wr6\") pod \"9092860c-8c05-411c-b377-a5482e6b6d95\" (UID: \"9092860c-8c05-411c-b377-a5482e6b6d95\") " Oct 30 05:34:42.623727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.623699 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:42.625641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.625607 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-xks-ca" pod "9092860c-8c05-411c-b377-a5482e6b6d95" (UID: "9092860c-8c05-411c-b377-a5482e6b6d95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:34:42.625901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.625885 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9092860c-8c05-411c-b377-a5482e6b6d95-kube-api-access-q2wr6" pod "9092860c-8c05-411c-b377-a5482e6b6d95" (UID: "9092860c-8c05-411c-b377-a5482e6b6d95"). InnerVolumeSpecName "kube-api-access-q2wr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:42.726170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.726137 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9092860c-8c05-411c-b377-a5482e6b6d95-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:42.726170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.726168 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2wr6\" (UniqueName: \"kubernetes.io/projected/9092860c-8c05-411c-b377-a5482e6b6d95-kube-api-access-q2wr6\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:42.881990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.881963 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:34:42.917650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.917621 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw"] Oct 30 05:34:42.935171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:42.935140 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw"] Oct 30 05:34:43.034903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.034869 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-token\" (UniqueName: \"kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-token\") pod \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " Oct 30 05:34:43.035100 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.034931 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istiod-ca-cert\") pod \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " Oct 30 05:34:43.035170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.035100 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-certs\") pod \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " Oct 30 05:34:43.035170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.035150 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-kube-api-access-2zq5d\" (UniqueName: \"kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-kube-api-access-2zq5d\") pod \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " Oct 30 05:34:43.035290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.035188 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-socket\") pod \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " Oct 30 05:34:43.035397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.035294 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-credential-socket\") pod \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " Oct 30 05:34:43.035519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.035393 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istiod-ca-cert" pod "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" (UID: "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:43.035519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.035416 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-data\") pod \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " Oct 30 05:34:43.035519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.035481 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-certs" pod "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" (UID: "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:43.035708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.035518 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-envoy\") pod \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " Oct 30 05:34:43.035708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.035581 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-podinfo\") pod \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\" (UID: \"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484\") " Oct 30 05:34:43.035708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.035622 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-socket" pod "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" (UID: "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:43.035708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.035697 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-credential-socket" pod "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" (UID: "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:43.036505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.035735 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-data" pod "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" (UID: "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:43.036599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.036524 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.036599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.036544 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.036599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.036555 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.036599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.036563 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.036599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.036571 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.037932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.037914 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-kube-api-access-2zq5d" pod "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" (UID: "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484"). InnerVolumeSpecName "kube-api-access-2zq5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:43.038146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.038116 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-token" pod "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" (UID: "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:43.038245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.038182 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-podinfo" pod "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" (UID: "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:34:43.038245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.038189 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-envoy" pod "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" (UID: "ec8ac8d3-efa9-4d17-aca4-d5c44e7de484"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:43.138616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.138534 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.138616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.138568 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.138616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.138581 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.138616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.138596 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zq5d\" (UniqueName: \"kubernetes.io/projected/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484-kube-api-access-2zq5d\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:43.319572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.319545 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" Oct 30 05:34:43.319991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.319580 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44" event={"ID":"9092860c-8c05-411c-b377-a5482e6b6d95","Type":"ContainerDied","Data":"80fac5c4d1645d6a2e6bad00be8054d23756efd7d0fc08e1e25f8c8a733a83ec"} Oct 30 05:34:43.319991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.319634 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80fac5c4d1645d6a2e6bad00be8054d23756efd7d0fc08e1e25f8c8a733a83ec" Oct 30 05:34:43.321555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.321527 6587 generic.go:356] "Generic (PLEG): container finished" podID="ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" containerID="db841bef681b0ca4818facec32f08f8ddf866facb1ee76aae5adc30aea45ceb3" exitCode=0 Oct 30 05:34:43.321720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.321577 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" Oct 30 05:34:43.321720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.321613 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" event={"ID":"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484","Type":"ContainerDied","Data":"db841bef681b0ca4818facec32f08f8ddf866facb1ee76aae5adc30aea45ceb3"} Oct 30 05:34:43.321720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.321680 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc" event={"ID":"ec8ac8d3-efa9-4d17-aca4-d5c44e7de484","Type":"ContainerDied","Data":"d50191f105ae06c6386e1f1bd519137459b081efb9e8f47c4ee631cb84d6d0dd"} Oct 30 05:34:43.321720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.321712 6587 scope.go:122] "RemoveContainer" containerID="db841bef681b0ca4818facec32f08f8ddf866facb1ee76aae5adc30aea45ceb3" Oct 30 05:34:43.337677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.337656 6587 scope.go:122] "RemoveContainer" containerID="db841bef681b0ca4818facec32f08f8ddf866facb1ee76aae5adc30aea45ceb3" Oct 30 05:34:43.338057 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:34:43.338030 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db841bef681b0ca4818facec32f08f8ddf866facb1ee76aae5adc30aea45ceb3\": container with ID starting with db841bef681b0ca4818facec32f08f8ddf866facb1ee76aae5adc30aea45ceb3 not found: ID does not exist" containerID="db841bef681b0ca4818facec32f08f8ddf866facb1ee76aae5adc30aea45ceb3" Oct 30 05:34:43.338164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.338064 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db841bef681b0ca4818facec32f08f8ddf866facb1ee76aae5adc30aea45ceb3"} err="failed to get container status \"db841bef681b0ca4818facec32f08f8ddf866facb1ee76aae5adc30aea45ceb3\": rpc error: code = NotFound desc = could not find container \"db841bef681b0ca4818facec32f08f8ddf866facb1ee76aae5adc30aea45ceb3\": container with ID starting with db841bef681b0ca4818facec32f08f8ddf866facb1ee76aae5adc30aea45ceb3 not found: ID does not exist" Oct 30 05:34:43.345840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.345812 6587 scope.go:122] "RemoveContainer" containerID="ffc48516b28ace24a2ef02c5ddc64c8ddb4740e5d66e580072d8dcedd9babec1" Oct 30 05:34:43.357917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.357899 6587 scope.go:122] "RemoveContainer" containerID="9797ceaef15b7f674b9aca7453396fc1caf944aacbc7a4b5eff184bb18662635" Oct 30 05:34:43.370393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.370371 6587 scope.go:122] "RemoveContainer" containerID="fdfff254fd06b517eb9146efb2c802fb30305e6a4afa9d69568d2aa76f0a4994" Oct 30 05:34:43.383509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.383488 6587 scope.go:122] "RemoveContainer" containerID="4cd9f819255ea7b060e860a3be8f74dc9ae020477a3c06a847a90d9b28b327ac" Oct 30 05:34:43.395263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.395118 6587 scope.go:122] "RemoveContainer" containerID="729d1aac14e1a36f4994282f755f9a0461c4987cf09813c38b4b63355d05d243" Oct 30 05:34:43.406916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.406894 6587 scope.go:122] "RemoveContainer" containerID="32bfd6dead1de4d9af55f224071ae8f15b5df48506e100837687f06307d3e34e" Oct 30 05:34:43.408344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.408320 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc"] Oct 30 05:34:43.414335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.414312 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc"] Oct 30 05:34:43.584872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:43.584842 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" path="/var/lib/kubelet/pods/ec8ac8d3-efa9-4d17-aca4-d5c44e7de484/volumes" Oct 30 05:34:44.088493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.088453 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" podUID="8e74f7bb-be64-439c-9991-9050a769b5cf" containerName="main" containerID="cri-o://df53b99829fb7613f1bd6e788cb7946b6a9ef63e3c52ab51aee83c435186f71c" gracePeriod=45 Oct 30 05:34:44.348458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.340966 6587 generic.go:356] "Generic (PLEG): container finished" podID="8e74f7bb-be64-439c-9991-9050a769b5cf" containerID="df53b99829fb7613f1bd6e788cb7946b6a9ef63e3c52ab51aee83c435186f71c" exitCode=0 Oct 30 05:34:44.348458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.341088 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" event={"ID":"8e74f7bb-be64-439c-9991-9050a769b5cf","Type":"ContainerDied","Data":"df53b99829fb7613f1bd6e788cb7946b6a9ef63e3c52ab51aee83c435186f71c"} Oct 30 05:34:44.411972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.411943 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:34:44.574882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.574851 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-tmp-dir\") pod \"8e74f7bb-be64-439c-9991-9050a769b5cf\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " Oct 30 05:34:44.575113 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.574931 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-model-cache\") pod \"8e74f7bb-be64-439c-9991-9050a769b5cf\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " Oct 30 05:34:44.575200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.575189 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e74f7bb-be64-439c-9991-9050a769b5cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e74f7bb-be64-439c-9991-9050a769b5cf-tls-certs\") pod \"8e74f7bb-be64-439c-9991-9050a769b5cf\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " Oct 30 05:34:44.575272 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.575222 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-dshm\") pod \"8e74f7bb-be64-439c-9991-9050a769b5cf\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " Oct 30 05:34:44.575339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.575277 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-model-cache" pod "8e74f7bb-be64-439c-9991-9050a769b5cf" (UID: "8e74f7bb-be64-439c-9991-9050a769b5cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:44.575339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.575294 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-tmp-dir" pod "8e74f7bb-be64-439c-9991-9050a769b5cf" (UID: "8e74f7bb-be64-439c-9991-9050a769b5cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:44.575339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.575304 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-home\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-home\") pod \"8e74f7bb-be64-439c-9991-9050a769b5cf\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " Oct 30 05:34:44.575529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.575483 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e74f7bb-be64-439c-9991-9050a769b5cf-kube-api-access-d7kg2\" (UniqueName: \"kubernetes.io/projected/8e74f7bb-be64-439c-9991-9050a769b5cf-kube-api-access-d7kg2\") pod \"8e74f7bb-be64-439c-9991-9050a769b5cf\" (UID: \"8e74f7bb-be64-439c-9991-9050a769b5cf\") " Oct 30 05:34:44.575597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.575583 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-home" pod "8e74f7bb-be64-439c-9991-9050a769b5cf" (UID: "8e74f7bb-be64-439c-9991-9050a769b5cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:44.576341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.576318 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:44.576484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.576343 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:44.576484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.576371 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:44.577614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.577589 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-dshm" pod "8e74f7bb-be64-439c-9991-9050a769b5cf" (UID: "8e74f7bb-be64-439c-9991-9050a769b5cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:44.577950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.577925 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e74f7bb-be64-439c-9991-9050a769b5cf-kube-api-access-d7kg2" pod "8e74f7bb-be64-439c-9991-9050a769b5cf" (UID: "8e74f7bb-be64-439c-9991-9050a769b5cf"). InnerVolumeSpecName "kube-api-access-d7kg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:44.578005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.577932 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e74f7bb-be64-439c-9991-9050a769b5cf-tls-certs" pod "8e74f7bb-be64-439c-9991-9050a769b5cf" (UID: "8e74f7bb-be64-439c-9991-9050a769b5cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:34:44.679178 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.679139 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7kg2\" (UniqueName: \"kubernetes.io/projected/8e74f7bb-be64-439c-9991-9050a769b5cf-kube-api-access-d7kg2\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:44.679178 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.679179 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e74f7bb-be64-439c-9991-9050a769b5cf-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:44.679481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:44.679193 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e74f7bb-be64-439c-9991-9050a769b5cf-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:45.545384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:45.543838 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" event={"ID":"8e74f7bb-be64-439c-9991-9050a769b5cf","Type":"ContainerDied","Data":"c3224561727d1989e2bad3389048b7c0e2b2b679caa41ae993774de34383c9e7"} Oct 30 05:34:45.545384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:45.543901 6587 scope.go:122] "RemoveContainer" containerID="df53b99829fb7613f1bd6e788cb7946b6a9ef63e3c52ab51aee83c435186f71c" Oct 30 05:34:45.545384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:45.544020 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh" Oct 30 05:34:45.762737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:45.762704 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh"] Oct 30 05:34:45.774385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:45.774342 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-262c10/rate-test-model-262c10-kserve-648694d885-p2rhh"] Oct 30 05:34:47.583492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:47.583460 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e74f7bb-be64-439c-9991-9050a769b5cf" path="/var/lib/kubelet/pods/8e74f7bb-be64-439c-9991-9050a769b5cf/volumes" Oct 30 05:34:48.424414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.424370 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v"] Oct 30 05:34:48.425775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.425749 6587 cpu_manager.go:414] "containerMap: removing container" podUID="ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" containerName="istio-proxy" Oct 30 05:34:48.425775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.425773 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" containerName="istio-proxy" Oct 30 05:34:48.425969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.425833 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9092860c-8c05-411c-b377-a5482e6b6d95" containerName="revoke-keys" Oct 30 05:34:48.425969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.425842 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9092860c-8c05-411c-b377-a5482e6b6d95" containerName="revoke-keys" Oct 30 05:34:48.425969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.425873 6587 cpu_manager.go:414] "containerMap: removing container" podUID="8e74f7bb-be64-439c-9991-9050a769b5cf" containerName="main" Oct 30 05:34:48.425969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.425880 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e74f7bb-be64-439c-9991-9050a769b5cf" containerName="main" Oct 30 05:34:48.425969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.425898 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a" containerName="revoke-keys" Oct 30 05:34:48.425969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.425905 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a" containerName="revoke-keys" Oct 30 05:34:48.427200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.426088 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9092860c-8c05-411c-b377-a5482e6b6d95" containerName="revoke-keys" Oct 30 05:34:48.427200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.426102 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e74f7bb-be64-439c-9991-9050a769b5cf" containerName="main" Oct 30 05:34:48.427200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.426113 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a" containerName="revoke-keys" Oct 30 05:34:48.427200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.426134 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8ac8d3-efa9-4d17-aca4-d5c44e7de484" containerName="istio-proxy" Oct 30 05:34:48.432573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.432546 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.452510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.452477 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v"] Oct 30 05:34:48.558819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.558783 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.559034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.558845 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-podinfo\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.559034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.558955 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-data\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.559034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.559004 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-envoy\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.559173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.559080 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-certs\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.559220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.559178 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-token\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.559259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.559232 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjzvw\" (UniqueName: \"kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-kube-api-access-sjzvw\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.559309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.559263 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istiod-ca-cert\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.559465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.559445 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-credential-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.652293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.652257 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5"] Oct 30 05:34:48.660949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.660920 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.661125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.661037 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-podinfo\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.661125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.661088 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-data\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.661295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.661135 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-envoy\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.661295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.661180 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-certs\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.661481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.661353 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-token\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.661481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.661407 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.661603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.661477 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjzvw\" (UniqueName: \"kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-kube-api-access-sjzvw\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.661603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.661537 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istiod-ca-cert\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.661792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.661768 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-credential-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.661873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.661817 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-certs\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.662105 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.662081 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-credential-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.662795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.662773 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istiod-ca-cert\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.662994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.662975 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-data\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.663910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.663886 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-envoy\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.664438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.664417 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-podinfo\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.683983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.683923 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z"] Oct 30 05:34:48.684110 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.683983 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjzvw\" (UniqueName: \"kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-kube-api-access-sjzvw\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.684368 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.684336 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-token\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:48.752542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:48.752508 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:49.116449 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:34:49.116406 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0879dd79_1117_46f7_b6e4_8b6f94e9cc67.slice/crio-4ce253ce1131dabf85c75c0fea2790f09ab511179415b35cc12d7800ed9b170c WatchSource:0}: Error finding container 4ce253ce1131dabf85c75c0fea2790f09ab511179415b35cc12d7800ed9b170c: Status 404 returned error can't find the container with id 4ce253ce1131dabf85c75c0fea2790f09ab511179415b35cc12d7800ed9b170c Oct 30 05:34:49.118258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:49.118236 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v"] Oct 30 05:34:49.119780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:49.119749 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:34:49.119876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:49.119809 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:34:49.119876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:49.119838 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:34:49.594665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:49.594629 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" event={"ID":"0879dd79-1117-46f7-b6e4-8b6f94e9cc67","Type":"ContainerStarted","Data":"7addf003c8a19630177de95436891b985663ec6ad2874a62d5d48ad7ffa97e0c"} Oct 30 05:34:49.594665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:49.594694 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" event={"ID":"0879dd79-1117-46f7-b6e4-8b6f94e9cc67","Type":"ContainerStarted","Data":"4ce253ce1131dabf85c75c0fea2790f09ab511179415b35cc12d7800ed9b170c"} Oct 30 05:34:49.663189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:49.663116 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" podStartSLOduration=1.663097255 podStartE2EDuration="1.663097255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:34:49.656014218 +0000 UTC m=+3014.183781007" watchObservedRunningTime="2026-10-30 05:34:49.663097255 +0000 UTC m=+3014.190864042" Oct 30 05:34:49.706612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:49.706576 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm"] Oct 30 05:34:49.739585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:49.739540 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h"] Oct 30 05:34:49.753156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:49.752888 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:50.754324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:50.754289 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 05:34:50.754768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:50.754409 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" podUID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 05:34:51.672605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.672565 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f"] Oct 30 05:34:51.677899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.677867 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" Oct 30 05:34:51.685823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.685795 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt"] Oct 30 05:34:51.692374 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.692336 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" Oct 30 05:34:51.696016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.695991 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:34:51.696334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.696308 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:34:51.700587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.700560 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f"] Oct 30 05:34:51.757393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.753899 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 05:34:51.757393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.753953 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" podUID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 05:34:51.763461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.758425 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt"] Oct 30 05:34:51.785503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.785458 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7lbh\" (UniqueName: \"kubernetes.io/projected/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-kube-api-access-w7lbh\") pod \"maas-api-revoke-sub-372fea20-wxtgt\" (UID: \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" Oct 30 05:34:51.785671 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.785633 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-372fea20-wxtgt\" (UID: \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" Oct 30 05:34:51.785727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.785684 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d98b018-fbq6f\" (UID: \"5e40fbdc-a209-4661-8315-6bee6a146ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" Oct 30 05:34:51.785841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.785812 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc8bd\" (UniqueName: \"kubernetes.io/projected/5e40fbdc-a209-4661-8315-6bee6a146ffb-kube-api-access-kc8bd\") pod \"maas-api-revoke-sub-1d98b018-fbq6f\" (UID: \"5e40fbdc-a209-4661-8315-6bee6a146ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" Oct 30 05:34:51.785928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.785911 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d98b018-fbq6f\" (UID: \"5e40fbdc-a209-4661-8315-6bee6a146ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" Oct 30 05:34:51.786046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.785990 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-372fea20-wxtgt\" (UID: \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" Oct 30 05:34:51.887864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.887821 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7lbh\" (UniqueName: \"kubernetes.io/projected/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-kube-api-access-w7lbh\") pod \"maas-api-revoke-sub-372fea20-wxtgt\" (UID: \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" Oct 30 05:34:51.888085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.887932 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-372fea20-wxtgt\" (UID: \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" Oct 30 05:34:51.888085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.887987 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d98b018-fbq6f\" (UID: \"5e40fbdc-a209-4661-8315-6bee6a146ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" Oct 30 05:34:51.888085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.888033 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc8bd\" (UniqueName: \"kubernetes.io/projected/5e40fbdc-a209-4661-8315-6bee6a146ffb-kube-api-access-kc8bd\") pod \"maas-api-revoke-sub-1d98b018-fbq6f\" (UID: \"5e40fbdc-a209-4661-8315-6bee6a146ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" Oct 30 05:34:51.888264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.888208 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d98b018-fbq6f\" (UID: \"5e40fbdc-a209-4661-8315-6bee6a146ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" Oct 30 05:34:51.888420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.888342 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-372fea20-wxtgt\" (UID: \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" Oct 30 05:34:51.889192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.889161 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-372fea20-wxtgt\" (UID: \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" Oct 30 05:34:51.889399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.889340 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d98b018-fbq6f\" (UID: \"5e40fbdc-a209-4661-8315-6bee6a146ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" Oct 30 05:34:51.890856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.890831 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-372fea20-wxtgt\" (UID: \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" Oct 30 05:34:51.891303 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.891281 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d98b018-fbq6f\" (UID: \"5e40fbdc-a209-4661-8315-6bee6a146ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" Oct 30 05:34:51.910966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.910942 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc8bd\" (UniqueName: \"kubernetes.io/projected/5e40fbdc-a209-4661-8315-6bee6a146ffb-kube-api-access-kc8bd\") pod \"maas-api-revoke-sub-1d98b018-fbq6f\" (UID: \"5e40fbdc-a209-4661-8315-6bee6a146ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" Oct 30 05:34:51.911517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.911495 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7lbh\" (UniqueName: \"kubernetes.io/projected/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-kube-api-access-w7lbh\") pod \"maas-api-revoke-sub-372fea20-wxtgt\" (UID: \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" Oct 30 05:34:51.999920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:51.999828 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" Oct 30 05:34:52.076435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:52.075581 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" Oct 30 05:34:52.504661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:52.504322 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f"] Oct 30 05:34:52.506346 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:34:52.506313 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e40fbdc_a209_4661_8315_6bee6a146ffb.slice/crio-ab1b4c2ada1861478949960292af501308573182b64fde825efeecc43a4480b9 WatchSource:0}: Error finding container ab1b4c2ada1861478949960292af501308573182b64fde825efeecc43a4480b9: Status 404 returned error can't find the container with id ab1b4c2ada1861478949960292af501308573182b64fde825efeecc43a4480b9 Oct 30 05:34:52.611120 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:34:52.611085 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbe4fa6_2b31_4b1d_867b_2a2c26c87f55.slice/crio-1a9e73c6170a2f78e9ac23643f7f54edbadd7d9e8e7fcb1c949a36a4a86a14dd WatchSource:0}: Error finding container 1a9e73c6170a2f78e9ac23643f7f54edbadd7d9e8e7fcb1c949a36a4a86a14dd: Status 404 returned error can't find the container with id 1a9e73c6170a2f78e9ac23643f7f54edbadd7d9e8e7fcb1c949a36a4a86a14dd Oct 30 05:34:52.613587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:52.613563 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt"] Oct 30 05:34:52.757054 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:52.757012 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 05:34:52.757288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:52.757075 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" podUID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 05:34:52.763850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:52.763814 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" event={"ID":"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55","Type":"ContainerStarted","Data":"1a9e73c6170a2f78e9ac23643f7f54edbadd7d9e8e7fcb1c949a36a4a86a14dd"} Oct 30 05:34:52.766147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:52.766114 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" event={"ID":"5e40fbdc-a209-4661-8315-6bee6a146ffb","Type":"ContainerStarted","Data":"ab1b4c2ada1861478949960292af501308573182b64fde825efeecc43a4480b9"} Oct 30 05:34:53.657994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:53.654677 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" podUID="b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc" containerName="payload-processing" containerID="cri-o://f3bf091bde286d02bf2ea48d77d6baf32ded0e349c09825f5935b16269d7a09b" gracePeriod=25 Oct 30 05:34:53.684218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:53.684183 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" podUID="82c7bf20-a8c4-426e-b692-41e564879c82" containerName="payload-pre-processing" containerID="cri-o://1a195695ce00996f05c6d3bd2cd7a0843043f2f61972e773352e231857e244ce" gracePeriod=25 Oct 30 05:34:53.755490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:53.755103 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 05:34:53.755490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:53.755163 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" podUID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 05:34:53.783084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:53.783045 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" event={"ID":"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55","Type":"ContainerStarted","Data":"17917854a53a29fdc86d92df1a90880b3fc5fa801dbf353cb75d73128a3a1869"} Oct 30 05:34:53.785579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:53.785553 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" event={"ID":"5e40fbdc-a209-4661-8315-6bee6a146ffb","Type":"ContainerStarted","Data":"2b1d9bcb5d3d89ad7a8180d271d4116381574cd60c04ee1e67ee6d5d7ff419c2"} Oct 30 05:34:53.796098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:53.796068 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" podUID="b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:34:53.796448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:53.796310 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" podUID="82c7bf20-a8c4-426e-b692-41e564879c82" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:34:53.862631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:53.858969 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" podStartSLOduration=2.858949153 podStartE2EDuration="2.858949153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:34:53.838682962 +0000 UTC m=+3018.366449749" watchObservedRunningTime="2026-10-30 05:34:53.858949153 +0000 UTC m=+3018.386715941" Oct 30 05:34:53.893378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:53.889616 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" podStartSLOduration=2.889597285 podStartE2EDuration="2.889597285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:34:53.886828092 +0000 UTC m=+3018.414594878" watchObservedRunningTime="2026-10-30 05:34:53.889597285 +0000 UTC m=+3018.417364073" Oct 30 05:34:54.707261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:54.707191 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" podUID="00485499-909c-4e26-8206-edb1483727ce" containerName="payload-processing" containerID="cri-o://fa141672bc6b8a6c589b34da7fd732178444548712641360a092a93b7e2b58a2" gracePeriod=25 Oct 30 05:34:54.757647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:54.757163 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" podUID="e360f2a8-ce2f-47ae-8e56-a265de776937" containerName="payload-pre-processing" containerID="cri-o://e218bdb67d883c7e2b607f467c91b8c9758236f1f6d2f52eb8a565733c8ef0cb" gracePeriod=25 Oct 30 05:34:54.766716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:54.762571 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 05:34:54.766716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:54.762624 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" podUID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 05:34:54.796760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:54.796728 6587 generic.go:356] "Generic (PLEG): container finished" podID="7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55" containerID="17917854a53a29fdc86d92df1a90880b3fc5fa801dbf353cb75d73128a3a1869" exitCode=0 Oct 30 05:34:54.797243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:54.796864 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" event={"ID":"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55","Type":"ContainerDied","Data":"17917854a53a29fdc86d92df1a90880b3fc5fa801dbf353cb75d73128a3a1869"} Oct 30 05:34:54.798738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:54.798711 6587 generic.go:356] "Generic (PLEG): container finished" podID="5e40fbdc-a209-4661-8315-6bee6a146ffb" containerID="2b1d9bcb5d3d89ad7a8180d271d4116381574cd60c04ee1e67ee6d5d7ff419c2" exitCode=0 Oct 30 05:34:54.798867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:54.798755 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" event={"ID":"5e40fbdc-a209-4661-8315-6bee6a146ffb","Type":"ContainerDied","Data":"2b1d9bcb5d3d89ad7a8180d271d4116381574cd60c04ee1e67ee6d5d7ff419c2"} Oct 30 05:34:55.783238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:55.782756 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 05:34:55.783238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:55.782828 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" podUID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 05:34:56.072834 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.072782 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8"] Oct 30 05:34:56.099386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.083621 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" Oct 30 05:34:56.204160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.204119 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8"] Oct 30 05:34:56.233438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.226871 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ec1091-dockercfg-rrzfr\"" Oct 30 05:34:56.233438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.226983 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ec1091\"" Oct 30 05:34:56.233438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.229179 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973098f5-b7c1-42dc-934d-4cb248f7695b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8\" (UID: \"973098f5-b7c1-42dc-934d-4cb248f7695b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" Oct 30 05:34:56.233438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.229480 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzjdv\" (UniqueName: \"kubernetes.io/projected/973098f5-b7c1-42dc-934d-4cb248f7695b-kube-api-access-hzjdv\") pod \"payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8\" (UID: \"973098f5-b7c1-42dc-934d-4cb248f7695b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" Oct 30 05:34:56.301086 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:34:56.300990 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:34:56.356702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.332034 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzjdv\" (UniqueName: \"kubernetes.io/projected/973098f5-b7c1-42dc-934d-4cb248f7695b-kube-api-access-hzjdv\") pod \"payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8\" (UID: \"973098f5-b7c1-42dc-934d-4cb248f7695b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" Oct 30 05:34:56.360939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.360903 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973098f5-b7c1-42dc-934d-4cb248f7695b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8\" (UID: \"973098f5-b7c1-42dc-934d-4cb248f7695b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" Oct 30 05:34:56.362159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.362122 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973098f5-b7c1-42dc-934d-4cb248f7695b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8\" (UID: \"973098f5-b7c1-42dc-934d-4cb248f7695b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" Oct 30 05:34:56.391654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.387232 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzjdv\" (UniqueName: \"kubernetes.io/projected/973098f5-b7c1-42dc-934d-4cb248f7695b-kube-api-access-hzjdv\") pod \"payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8\" (UID: \"973098f5-b7c1-42dc-934d-4cb248f7695b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" Oct 30 05:34:56.604291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.583643 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" Oct 30 05:34:56.672533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.665603 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" Oct 30 05:34:56.765533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.744661 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-openshift-ca\") pod \"5e40fbdc-a209-4661-8315-6bee6a146ffb\" (UID: \"5e40fbdc-a209-4661-8315-6bee6a146ffb\") " Oct 30 05:34:56.765533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.744813 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-xks-ca\") pod \"5e40fbdc-a209-4661-8315-6bee6a146ffb\" (UID: \"5e40fbdc-a209-4661-8315-6bee6a146ffb\") " Oct 30 05:34:56.765533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.745102 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e40fbdc-a209-4661-8315-6bee6a146ffb-kube-api-access-kc8bd\" (UniqueName: \"kubernetes.io/projected/5e40fbdc-a209-4661-8315-6bee6a146ffb-kube-api-access-kc8bd\") pod \"5e40fbdc-a209-4661-8315-6bee6a146ffb\" (UID: \"5e40fbdc-a209-4661-8315-6bee6a146ffb\") " Oct 30 05:34:56.849101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.848991 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 05:34:56.849101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.849065 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" podUID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 05:34:56.898819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:56.898729 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-openshift-ca" pod "5e40fbdc-a209-4661-8315-6bee6a146ffb" (UID: "5e40fbdc-a209-4661-8315-6bee6a146ffb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:57.036392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.030613 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-xks-ca" pod "5e40fbdc-a209-4661-8315-6bee6a146ffb" (UID: "5e40fbdc-a209-4661-8315-6bee6a146ffb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:34:57.043406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.036767 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e40fbdc-a209-4661-8315-6bee6a146ffb-kube-api-access-kc8bd" pod "5e40fbdc-a209-4661-8315-6bee6a146ffb" (UID: "5e40fbdc-a209-4661-8315-6bee6a146ffb"). InnerVolumeSpecName "kube-api-access-kc8bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:57.060446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.045046 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:57.060446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.045083 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc8bd\" (UniqueName: \"kubernetes.io/projected/5e40fbdc-a209-4661-8315-6bee6a146ffb-kube-api-access-kc8bd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:57.060446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.045098 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e40fbdc-a209-4661-8315-6bee6a146ffb-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:57.060446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.050316 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" event={"ID":"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55","Type":"ContainerDied","Data":"1a9e73c6170a2f78e9ac23643f7f54edbadd7d9e8e7fcb1c949a36a4a86a14dd"} Oct 30 05:34:57.060446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.050389 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a9e73c6170a2f78e9ac23643f7f54edbadd7d9e8e7fcb1c949a36a4a86a14dd" Oct 30 05:34:57.064007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.063972 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" Oct 30 05:34:57.066694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.066560 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" event={"ID":"5e40fbdc-a209-4661-8315-6bee6a146ffb","Type":"ContainerDied","Data":"ab1b4c2ada1861478949960292af501308573182b64fde825efeecc43a4480b9"} Oct 30 05:34:57.066694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.066622 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab1b4c2ada1861478949960292af501308573182b64fde825efeecc43a4480b9" Oct 30 05:34:57.066694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.066695 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f" Oct 30 05:34:57.154808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.154727 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-openshift-ca\") pod \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\" (UID: \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\") " Oct 30 05:34:57.155369 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.155335 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-xks-ca\") pod \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\" (UID: \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\") " Oct 30 05:34:57.155598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.155582 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-kube-api-access-w7lbh\" (UniqueName: \"kubernetes.io/projected/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-kube-api-access-w7lbh\") pod \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\" (UID: \"7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55\") " Oct 30 05:34:57.156596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.156569 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-openshift-ca" pod "7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55" (UID: "7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:34:57.187001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.186917 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-kube-api-access-w7lbh" pod "7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55" (UID: "7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55"). InnerVolumeSpecName "kube-api-access-w7lbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:57.216417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.195764 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-xks-ca" pod "7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55" (UID: "7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:34:57.322395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.321847 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:57.322395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.321892 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:57.322395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.321907 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7lbh\" (UniqueName: \"kubernetes.io/projected/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55-kube-api-access-w7lbh\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:57.456562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.441216 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44"] Oct 30 05:34:57.524379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.524154 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-262c10-4mf44"] Oct 30 05:34:57.778204 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.770964 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Oct 30 05:34:57.778204 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.771015 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" podUID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Oct 30 05:34:57.778204 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:34:57.774975 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e40fbdc_a209_4661_8315_6bee6a146ffb.slice\": RecentStats: unable to find data in memory cache]" Oct 30 05:34:57.789144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.784065 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9092860c-8c05-411c-b377-a5482e6b6d95" path="/var/lib/kubelet/pods/9092860c-8c05-411c-b377-a5482e6b6d95/volumes" Oct 30 05:34:57.926107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.925838 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8"] Oct 30 05:34:57.948653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.948617 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" containerID="cri-o://c780a96e3c9a8b8d6563d624e7a5c673861f4004f4330ffbdd4010d6b20d522b" gracePeriod=45 Oct 30 05:34:57.954006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.953975 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": read tcp 10.134.0.2:38560->10.134.0.82:8000: read: connection reset by peer" Oct 30 05:34:57.981914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:57.981885 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" containerID="cri-o://45de0ce49422efb26c90e99737ef5e3804f30612950e14e7c4ecfaf7ce645f21" gracePeriod=45 Oct 30 05:34:58.100869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.100828 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" event={"ID":"973098f5-b7c1-42dc-934d-4cb248f7695b","Type":"ContainerStarted","Data":"d412c0dc712ae8bfbdd7ac3bb4ed82c4700f9dace26278a5dd6aad58fa7b4128"} Oct 30 05:34:58.107728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.107683 6587 generic.go:356] "Generic (PLEG): container finished" podID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerID="c780a96e3c9a8b8d6563d624e7a5c673861f4004f4330ffbdd4010d6b20d522b" exitCode=0 Oct 30 05:34:58.107859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.107785 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt" Oct 30 05:34:58.108231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.108201 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" event={"ID":"068d2c70-454e-4ebd-ad63-4c17ac9fc1da","Type":"ContainerDied","Data":"c780a96e3c9a8b8d6563d624e7a5c673861f4004f4330ffbdd4010d6b20d522b"} Oct 30 05:34:58.171922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.171867 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9"] Oct 30 05:34:58.179790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.178885 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-889ca9-xcdw9"] Oct 30 05:34:58.339142 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.338796 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:34:58.487018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.486982 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-kube-api-access-x9kk2\" (UniqueName: \"kubernetes.io/projected/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-kube-api-access-x9kk2\") pod \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " Oct 30 05:34:58.487169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.487103 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-model-cache\") pod \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " Oct 30 05:34:58.487249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.487230 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-home\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-home\") pod \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " Oct 30 05:34:58.487555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.487409 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tmp-dir\") pod \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " Oct 30 05:34:58.487698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.487610 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tls-certs\" (UniqueName: \"kubernetes.io/secret/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tls-certs\") pod \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " Oct 30 05:34:58.487768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.487719 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-dshm\") pod \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\" (UID: \"068d2c70-454e-4ebd-ad63-4c17ac9fc1da\") " Oct 30 05:34:58.489878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.489850 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-home" pod "068d2c70-454e-4ebd-ad63-4c17ac9fc1da" (UID: "068d2c70-454e-4ebd-ad63-4c17ac9fc1da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:58.491514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.491485 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-dshm" pod "068d2c70-454e-4ebd-ad63-4c17ac9fc1da" (UID: "068d2c70-454e-4ebd-ad63-4c17ac9fc1da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:58.491870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.491850 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tmp-dir" pod "068d2c70-454e-4ebd-ad63-4c17ac9fc1da" (UID: "068d2c70-454e-4ebd-ad63-4c17ac9fc1da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:58.492793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.492764 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-kube-api-access-x9kk2" pod "068d2c70-454e-4ebd-ad63-4c17ac9fc1da" (UID: "068d2c70-454e-4ebd-ad63-4c17ac9fc1da"). InnerVolumeSpecName "kube-api-access-x9kk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:58.493088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.493064 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-model-cache" pod "068d2c70-454e-4ebd-ad63-4c17ac9fc1da" (UID: "068d2c70-454e-4ebd-ad63-4c17ac9fc1da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:58.496849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.496828 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tls-certs" pod "068d2c70-454e-4ebd-ad63-4c17ac9fc1da" (UID: "068d2c70-454e-4ebd-ad63-4c17ac9fc1da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:34:58.563146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.563116 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:34:58.590711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.590679 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:58.590711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.590711 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:58.590923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.590723 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:58.590923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.590735 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9kk2\" (UniqueName: \"kubernetes.io/projected/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-kube-api-access-x9kk2\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:58.590923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.590749 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:58.590923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.590761 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/068d2c70-454e-4ebd-ad63-4c17ac9fc1da-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:58.692533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.692486 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-model-cache\") pod \"607c7ab6-765b-494e-b2b5-e40f0a16e956\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " Oct 30 05:34:58.692719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.692650 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607c7ab6-765b-494e-b2b5-e40f0a16e956-kube-api-access-kh82t\" (UniqueName: \"kubernetes.io/projected/607c7ab6-765b-494e-b2b5-e40f0a16e956-kube-api-access-kh82t\") pod \"607c7ab6-765b-494e-b2b5-e40f0a16e956\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " Oct 30 05:34:58.692719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.692696 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/607c7ab6-765b-494e-b2b5-e40f0a16e956-tls-certs\" (UniqueName: \"kubernetes.io/secret/607c7ab6-765b-494e-b2b5-e40f0a16e956-tls-certs\") pod \"607c7ab6-765b-494e-b2b5-e40f0a16e956\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " Oct 30 05:34:58.692890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.692766 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-tmp-dir\") pod \"607c7ab6-765b-494e-b2b5-e40f0a16e956\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " Oct 30 05:34:58.692890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.692865 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-home\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-home\") pod \"607c7ab6-765b-494e-b2b5-e40f0a16e956\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " Oct 30 05:34:58.693012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.692935 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-dshm\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-dshm\") pod \"607c7ab6-765b-494e-b2b5-e40f0a16e956\" (UID: \"607c7ab6-765b-494e-b2b5-e40f0a16e956\") " Oct 30 05:34:58.695031 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.694991 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-home" pod "607c7ab6-765b-494e-b2b5-e40f0a16e956" (UID: "607c7ab6-765b-494e-b2b5-e40f0a16e956"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:58.695233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.695210 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-model-cache" pod "607c7ab6-765b-494e-b2b5-e40f0a16e956" (UID: "607c7ab6-765b-494e-b2b5-e40f0a16e956"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:58.695673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.695653 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-tmp-dir" pod "607c7ab6-765b-494e-b2b5-e40f0a16e956" (UID: "607c7ab6-765b-494e-b2b5-e40f0a16e956"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:58.708940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.708751 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607c7ab6-765b-494e-b2b5-e40f0a16e956-kube-api-access-kh82t" pod "607c7ab6-765b-494e-b2b5-e40f0a16e956" (UID: "607c7ab6-765b-494e-b2b5-e40f0a16e956"). InnerVolumeSpecName "kube-api-access-kh82t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:34:58.732484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.732453 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/607c7ab6-765b-494e-b2b5-e40f0a16e956-tls-certs" pod "607c7ab6-765b-494e-b2b5-e40f0a16e956" (UID: "607c7ab6-765b-494e-b2b5-e40f0a16e956"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:34:58.732770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.732752 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-dshm" pod "607c7ab6-765b-494e-b2b5-e40f0a16e956" (UID: "607c7ab6-765b-494e-b2b5-e40f0a16e956"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:34:58.795913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.795873 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:58.795913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.795908 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh82t\" (UniqueName: \"kubernetes.io/projected/607c7ab6-765b-494e-b2b5-e40f0a16e956-kube-api-access-kh82t\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:58.795913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.795918 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607c7ab6-765b-494e-b2b5-e40f0a16e956-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:58.796167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.795928 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:58.796167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.795936 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:58.796167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:58.795943 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607c7ab6-765b-494e-b2b5-e40f0a16e956-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:34:59.107888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.107856 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:59.109756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.109493 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:59.112932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.112909 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:34:59.125707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.125671 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" event={"ID":"068d2c70-454e-4ebd-ad63-4c17ac9fc1da","Type":"ContainerDied","Data":"0a8f0cef2184904e8acee8382ddc6511511d2df68ceacb965b85340176c2b213"} Oct 30 05:34:59.125816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.125686 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw" Oct 30 05:34:59.125889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.125843 6587 scope.go:122] "RemoveContainer" containerID="c780a96e3c9a8b8d6563d624e7a5c673861f4004f4330ffbdd4010d6b20d522b" Oct 30 05:34:59.127703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.127678 6587 generic.go:356] "Generic (PLEG): container finished" podID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerID="45de0ce49422efb26c90e99737ef5e3804f30612950e14e7c4ecfaf7ce645f21" exitCode=0 Oct 30 05:34:59.127981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.127960 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" event={"ID":"607c7ab6-765b-494e-b2b5-e40f0a16e956","Type":"ContainerDied","Data":"45de0ce49422efb26c90e99737ef5e3804f30612950e14e7c4ecfaf7ce645f21"} Oct 30 05:34:59.128053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.127995 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" event={"ID":"607c7ab6-765b-494e-b2b5-e40f0a16e956","Type":"ContainerDied","Data":"13ec8cb8ce962deb5b2bef6a64a9e17f474ea7bfc7237663927f1cc3644ec0d7"} Oct 30 05:34:59.128109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.128057 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw" Oct 30 05:34:59.142565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.142546 6587 scope.go:122] "RemoveContainer" containerID="45de0ce49422efb26c90e99737ef5e3804f30612950e14e7c4ecfaf7ce645f21" Oct 30 05:34:59.146213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.146014 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" event={"ID":"973098f5-b7c1-42dc-934d-4cb248f7695b","Type":"ContainerStarted","Data":"29595456f134e1ef150ac8116a71b4513d1ffd7d15ecfa218d5e65d77aadc849"} Oct 30 05:34:59.146213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.146209 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" Oct 30 05:34:59.160102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.160084 6587 scope.go:122] "RemoveContainer" containerID="45de0ce49422efb26c90e99737ef5e3804f30612950e14e7c4ecfaf7ce645f21" Oct 30 05:34:59.160447 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:34:59.160419 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45de0ce49422efb26c90e99737ef5e3804f30612950e14e7c4ecfaf7ce645f21\": container with ID starting with 45de0ce49422efb26c90e99737ef5e3804f30612950e14e7c4ecfaf7ce645f21 not found: ID does not exist" containerID="45de0ce49422efb26c90e99737ef5e3804f30612950e14e7c4ecfaf7ce645f21" Oct 30 05:34:59.160535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.160457 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45de0ce49422efb26c90e99737ef5e3804f30612950e14e7c4ecfaf7ce645f21"} err="failed to get container status \"45de0ce49422efb26c90e99737ef5e3804f30612950e14e7c4ecfaf7ce645f21\": rpc error: code = NotFound desc = could not find container \"45de0ce49422efb26c90e99737ef5e3804f30612950e14e7c4ecfaf7ce645f21\": container with ID starting with 45de0ce49422efb26c90e99737ef5e3804f30612950e14e7c4ecfaf7ce645f21 not found: ID does not exist" Oct 30 05:34:59.305190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.305148 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw"] Oct 30 05:34:59.309474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.309446 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/distinct-w6-83e83e-kserve-5b7f687bff-2vscw"] Oct 30 05:34:59.348639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.348579 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" podStartSLOduration=4.348564753 podStartE2EDuration="4.348564753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:34:59.345188775 +0000 UTC m=+3023.872955561" watchObservedRunningTime="2026-10-30 05:34:59.348564753 +0000 UTC m=+3023.876331539" Oct 30 05:34:59.394545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.394467 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw"] Oct 30 05:34:59.402350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.402319 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/premium-w6-83e83e-kserve-5c8b9f9944-pmprw"] Oct 30 05:34:59.587624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.587575 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" path="/var/lib/kubelet/pods/068d2c70-454e-4ebd-ad63-4c17ac9fc1da/volumes" Oct 30 05:34:59.588920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.588888 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" path="/var/lib/kubelet/pods/607c7ab6-765b-494e-b2b5-e40f0a16e956/volumes" Oct 30 05:34:59.589649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:34:59.589622 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa5728c-ac1b-47b6-a04c-e9423037ab25" path="/var/lib/kubelet/pods/eaa5728c-ac1b-47b6-a04c-e9423037ab25/volumes" Oct 30 05:35:01.846433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.832437 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw"] Oct 30 05:35:01.846433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.834230 6587 cpu_manager.go:414] "containerMap: removing container" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" Oct 30 05:35:01.846433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.834266 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" Oct 30 05:35:01.846433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.834286 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5e40fbdc-a209-4661-8315-6bee6a146ffb" containerName="revoke-keys" Oct 30 05:35:01.846433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.834295 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e40fbdc-a209-4661-8315-6bee6a146ffb" containerName="revoke-keys" Oct 30 05:35:01.846433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.834338 6587 cpu_manager.go:414] "containerMap: removing container" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" Oct 30 05:35:01.846433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.834348 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" Oct 30 05:35:01.846433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.834433 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55" containerName="revoke-keys" Oct 30 05:35:01.846433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.834442 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55" containerName="revoke-keys" Oct 30 05:35:01.846433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.834678 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="068d2c70-454e-4ebd-ad63-4c17ac9fc1da" containerName="main" Oct 30 05:35:01.846433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.834700 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55" containerName="revoke-keys" Oct 30 05:35:01.846433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.834719 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="607c7ab6-765b-494e-b2b5-e40f0a16e956" containerName="main" Oct 30 05:35:01.846433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.834729 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e40fbdc-a209-4661-8315-6bee6a146ffb" containerName="revoke-keys" Oct 30 05:35:01.870822 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.870791 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" Oct 30 05:35:01.907259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.907225 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:35:01.929071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.928061 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:35:01.959140 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.958793 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw"] Oct 30 05:35:01.993457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:01.974280 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgrkj\" (UniqueName: \"kubernetes.io/projected/e8c76eca-369d-4b82-8f99-a056a7441456-kube-api-access-wgrkj\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw\" (UID: \"e8c76eca-369d-4b82-8f99-a056a7441456\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" Oct 30 05:35:02.014205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:02.014167 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw\" (UID: \"e8c76eca-369d-4b82-8f99-a056a7441456\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" Oct 30 05:35:02.018605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:02.018563 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw\" (UID: \"e8c76eca-369d-4b82-8f99-a056a7441456\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" Oct 30 05:35:02.165486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:02.164556 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw\" (UID: \"e8c76eca-369d-4b82-8f99-a056a7441456\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" Oct 30 05:35:02.165486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:02.164662 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw\" (UID: \"e8c76eca-369d-4b82-8f99-a056a7441456\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" Oct 30 05:35:02.165486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:02.165041 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgrkj\" (UniqueName: \"kubernetes.io/projected/e8c76eca-369d-4b82-8f99-a056a7441456-kube-api-access-wgrkj\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw\" (UID: \"e8c76eca-369d-4b82-8f99-a056a7441456\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" Oct 30 05:35:02.190383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:02.166290 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw\" (UID: \"e8c76eca-369d-4b82-8f99-a056a7441456\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" Oct 30 05:35:02.190383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:02.177980 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw\" (UID: \"e8c76eca-369d-4b82-8f99-a056a7441456\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" Oct 30 05:35:02.244078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:02.244045 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgrkj\" (UniqueName: \"kubernetes.io/projected/e8c76eca-369d-4b82-8f99-a056a7441456-kube-api-access-wgrkj\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw\" (UID: \"e8c76eca-369d-4b82-8f99-a056a7441456\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" Oct 30 05:35:02.345809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:02.345773 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" Oct 30 05:35:02.803271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:02.803237 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw"] Oct 30 05:35:02.803963 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:35:02.803934 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c76eca_369d_4b82_8f99_a056a7441456.slice/crio-ef2f9da88c20f623b12ec5fb05a271b1a888193a0f4f14026a66e6caceb98641 WatchSource:0}: Error finding container ef2f9da88c20f623b12ec5fb05a271b1a888193a0f4f14026a66e6caceb98641: Status 404 returned error can't find the container with id ef2f9da88c20f623b12ec5fb05a271b1a888193a0f4f14026a66e6caceb98641 Oct 30 05:35:03.260801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:03.260751 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" event={"ID":"e8c76eca-369d-4b82-8f99-a056a7441456","Type":"ContainerStarted","Data":"ef2f9da88c20f623b12ec5fb05a271b1a888193a0f4f14026a66e6caceb98641"} Oct 30 05:35:03.795527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:03.795479 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" podUID="82c7bf20-a8c4-426e-b692-41e564879c82" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:35:03.795719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:03.795503 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" podUID="b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:35:04.272287 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:04.272245 6587 generic.go:356] "Generic (PLEG): container finished" podID="e8c76eca-369d-4b82-8f99-a056a7441456" containerID="6ad95de9f665e51f6c54d0681b7e75f6756a1684ec0d4c64c119648f6c8ed63e" exitCode=0 Oct 30 05:35:04.272730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:04.272332 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" event={"ID":"e8c76eca-369d-4b82-8f99-a056a7441456","Type":"ContainerDied","Data":"6ad95de9f665e51f6c54d0681b7e75f6756a1684ec0d4c64c119648f6c8ed63e"} Oct 30 05:35:04.471636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:04.471593 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" podUID="e360f2a8-ce2f-47ae-8e56-a265de776937" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:35:04.472029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:04.472000 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" podUID="00485499-909c-4e26-8206-edb1483727ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:35:05.737180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.737101 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" Oct 30 05:35:05.783535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.780771 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-xks-ca\") pod \"e8c76eca-369d-4b82-8f99-a056a7441456\" (UID: \"e8c76eca-369d-4b82-8f99-a056a7441456\") " Oct 30 05:35:05.783535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.780910 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c76eca-369d-4b82-8f99-a056a7441456-kube-api-access-wgrkj\" (UniqueName: \"kubernetes.io/projected/e8c76eca-369d-4b82-8f99-a056a7441456-kube-api-access-wgrkj\") pod \"e8c76eca-369d-4b82-8f99-a056a7441456\" (UID: \"e8c76eca-369d-4b82-8f99-a056a7441456\") " Oct 30 05:35:05.783535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.781254 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-openshift-ca\") pod \"e8c76eca-369d-4b82-8f99-a056a7441456\" (UID: \"e8c76eca-369d-4b82-8f99-a056a7441456\") " Oct 30 05:35:05.784016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.783970 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-openshift-ca" pod "e8c76eca-369d-4b82-8f99-a056a7441456" (UID: "e8c76eca-369d-4b82-8f99-a056a7441456"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:05.796648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.796595 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-xks-ca" pod "e8c76eca-369d-4b82-8f99-a056a7441456" (UID: "e8c76eca-369d-4b82-8f99-a056a7441456"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:35:05.832380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.830734 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c76eca-369d-4b82-8f99-a056a7441456-kube-api-access-wgrkj" pod "e8c76eca-369d-4b82-8f99-a056a7441456" (UID: "e8c76eca-369d-4b82-8f99-a056a7441456"). InnerVolumeSpecName "kube-api-access-wgrkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:05.874377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.872689 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll"] Oct 30 05:35:05.874377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.872949 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" containerID="cri-o://14d8d9568466e6c3a15430b748ea3a4672d3d7c52ec41810dcbab4b811341bc9" gracePeriod=30 Oct 30 05:35:05.881469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.880820 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v"] Oct 30 05:35:05.881469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.881138 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" podUID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" containerName="istio-proxy" containerID="cri-o://7addf003c8a19630177de95436891b985663ec6ad2874a62d5d48ad7ffa97e0c" gracePeriod=30 Oct 30 05:35:05.885940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.885691 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:05.885940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.885726 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c76eca-369d-4b82-8f99-a056a7441456-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:05.885940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:05.885742 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgrkj\" (UniqueName: \"kubernetes.io/projected/e8c76eca-369d-4b82-8f99-a056a7441456-kube-api-access-wgrkj\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:06.370381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.369531 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml"] Oct 30 05:35:06.370381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.369842 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" podUID="e011432d-3991-4d45-9926-cf7ca5e50194" containerName="istio-proxy" containerID="cri-o://97a9040b91f5f71d100b7fdac094b9780f69b83988b8f33d385979dd28b36894" gracePeriod=30 Oct 30 05:35:06.469467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.468333 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8"] Oct 30 05:35:06.489381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.479860 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e8c76eca-369d-4b82-8f99-a056a7441456" containerName="revoke-keys" Oct 30 05:35:06.489381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.479895 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8c76eca-369d-4b82-8f99-a056a7441456" containerName="revoke-keys" Oct 30 05:35:06.489381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.480189 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8c76eca-369d-4b82-8f99-a056a7441456" containerName="revoke-keys" Oct 30 05:35:06.564056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.553244 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" Oct 30 05:35:06.672096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.663806 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8"] Oct 30 05:35:06.674330 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.674306 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" event={"ID":"e8c76eca-369d-4b82-8f99-a056a7441456","Type":"ContainerDied","Data":"ef2f9da88c20f623b12ec5fb05a271b1a888193a0f4f14026a66e6caceb98641"} Oct 30 05:35:06.675035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.674450 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef2f9da88c20f623b12ec5fb05a271b1a888193a0f4f14026a66e6caceb98641" Oct 30 05:35:06.675035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.674644 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw" Oct 30 05:35:06.681330 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.681171 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8\" (UID: \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" Oct 30 05:35:06.683139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.682242 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bcml\" (UniqueName: \"kubernetes.io/projected/e7dfc5c3-dfbc-407e-889a-5f8800106be2-kube-api-access-9bcml\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8\" (UID: \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" Oct 30 05:35:06.683139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.682385 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8\" (UID: \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" Oct 30 05:35:06.784160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.784116 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bcml\" (UniqueName: \"kubernetes.io/projected/e7dfc5c3-dfbc-407e-889a-5f8800106be2-kube-api-access-9bcml\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8\" (UID: \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" Oct 30 05:35:06.784612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.784203 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8\" (UID: \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" Oct 30 05:35:06.784612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.784266 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8\" (UID: \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" Oct 30 05:35:06.785017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.784990 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8\" (UID: \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" Oct 30 05:35:06.786703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.786682 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8\" (UID: \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" Oct 30 05:35:06.806437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.806407 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bcml\" (UniqueName: \"kubernetes.io/projected/e7dfc5c3-dfbc-407e-889a-5f8800106be2-kube-api-access-9bcml\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8\" (UID: \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" Oct 30 05:35:06.993837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:06.993758 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" Oct 30 05:35:07.318703 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:35:07.318676 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7dfc5c3_dfbc_407e_889a_5f8800106be2.slice/crio-05a18f9df4b5a41b0d8e76404c2491d4129b4237e08901be5e24fa38bb46188a WatchSource:0}: Error finding container 05a18f9df4b5a41b0d8e76404c2491d4129b4237e08901be5e24fa38bb46188a: Status 404 returned error can't find the container with id 05a18f9df4b5a41b0d8e76404c2491d4129b4237e08901be5e24fa38bb46188a Oct 30 05:35:07.322202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:07.322171 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8"] Oct 30 05:35:07.684298 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:07.684211 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" event={"ID":"e7dfc5c3-dfbc-407e-889a-5f8800106be2","Type":"ContainerStarted","Data":"05a18f9df4b5a41b0d8e76404c2491d4129b4237e08901be5e24fa38bb46188a"} Oct 30 05:35:08.697202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:08.697114 6587 generic.go:356] "Generic (PLEG): container finished" podID="e7dfc5c3-dfbc-407e-889a-5f8800106be2" containerID="4b60bf0dd542950ce8b20fd7f10f0955189404b1bdbcfafb866f8a8c26b5ba26" exitCode=0 Oct 30 05:35:08.697628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:08.697237 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" event={"ID":"e7dfc5c3-dfbc-407e-889a-5f8800106be2","Type":"ContainerDied","Data":"4b60bf0dd542950ce8b20fd7f10f0955189404b1bdbcfafb866f8a8c26b5ba26"} Oct 30 05:35:09.602351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:09.602317 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:35:09.602559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:09.602407 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:35:09.898247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:09.898218 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" Oct 30 05:35:10.044800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:10.044758 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-openshift-ca\") pod \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\" (UID: \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\") " Oct 30 05:35:10.045004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:10.044873 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7dfc5c3-dfbc-407e-889a-5f8800106be2-kube-api-access-9bcml\" (UniqueName: \"kubernetes.io/projected/e7dfc5c3-dfbc-407e-889a-5f8800106be2-kube-api-access-9bcml\") pod \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\" (UID: \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\") " Oct 30 05:35:10.045070 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:10.045010 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-xks-ca\") pod \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\" (UID: \"e7dfc5c3-dfbc-407e-889a-5f8800106be2\") " Oct 30 05:35:10.045224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:10.045195 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-openshift-ca" pod "e7dfc5c3-dfbc-407e-889a-5f8800106be2" (UID: "e7dfc5c3-dfbc-407e-889a-5f8800106be2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:10.045831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:10.045810 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:10.047128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:10.047096 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-xks-ca" pod "e7dfc5c3-dfbc-407e-889a-5f8800106be2" (UID: "e7dfc5c3-dfbc-407e-889a-5f8800106be2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:35:10.047571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:10.047550 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dfc5c3-dfbc-407e-889a-5f8800106be2-kube-api-access-9bcml" pod "e7dfc5c3-dfbc-407e-889a-5f8800106be2" (UID: "e7dfc5c3-dfbc-407e-889a-5f8800106be2"). InnerVolumeSpecName "kube-api-access-9bcml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:10.147591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:10.147499 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bcml\" (UniqueName: \"kubernetes.io/projected/e7dfc5c3-dfbc-407e-889a-5f8800106be2-kube-api-access-9bcml\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:10.147591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:10.147538 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dfc5c3-dfbc-407e-889a-5f8800106be2-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:10.167382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:10.167333 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" Oct 30 05:35:10.717125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:10.717097 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" Oct 30 05:35:10.717377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:10.717117 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8" event={"ID":"e7dfc5c3-dfbc-407e-889a-5f8800106be2","Type":"ContainerDied","Data":"05a18f9df4b5a41b0d8e76404c2491d4129b4237e08901be5e24fa38bb46188a"} Oct 30 05:35:10.717377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:10.717171 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05a18f9df4b5a41b0d8e76404c2491d4129b4237e08901be5e24fa38bb46188a" Oct 30 05:35:11.231167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.230541 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:35:11.234092 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.234067 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:35:11.375087 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.375052 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-token\" (UniqueName: \"kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-token\") pod \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " Oct 30 05:35:11.375288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.375152 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-token\" (UniqueName: \"kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-token\") pod \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " Oct 30 05:35:11.375288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.375234 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-credential-socket\") pod \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " Oct 30 05:35:11.375467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.375426 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-kube-api-access-sjzvw\" (UniqueName: \"kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-kube-api-access-sjzvw\") pod \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " Oct 30 05:35:11.375528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.375474 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-certs\") pod \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " Oct 30 05:35:11.375583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.375542 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-credential-socket" pod "0879dd79-1117-46f7-b6e4-8b6f94e9cc67" (UID: "0879dd79-1117-46f7-b6e4-8b6f94e9cc67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.375637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.375550 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-kube-api-access-cqrs9\" (UniqueName: \"kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-kube-api-access-cqrs9\") pod \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " Oct 30 05:35:11.375932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.375904 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-certs" pod "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" (UID: "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.376035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.375979 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-envoy\") pod \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " Oct 30 05:35:11.376103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.376083 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istiod-ca-cert\") pod \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " Oct 30 05:35:11.376162 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.376129 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-data\") pod \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " Oct 30 05:35:11.376220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.376209 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-socket\") pod \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " Oct 30 05:35:11.376279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.376268 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istiod-ca-cert\") pod \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " Oct 30 05:35:11.376334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.376305 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-envoy\") pod \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " Oct 30 05:35:11.376447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.376337 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-credential-socket\") pod \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " Oct 30 05:35:11.376447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.376433 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-data\") pod \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " Oct 30 05:35:11.377214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.376543 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-data" pod "0879dd79-1117-46f7-b6e4-8b6f94e9cc67" (UID: "0879dd79-1117-46f7-b6e4-8b6f94e9cc67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.377214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.376612 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-socket" pod "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" (UID: "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.377214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.376955 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-credential-socket" pod "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" (UID: "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.377214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.377113 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-data" pod "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" (UID: "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.377214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.377135 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istiod-ca-cert" pod "0879dd79-1117-46f7-b6e4-8b6f94e9cc67" (UID: "0879dd79-1117-46f7-b6e4-8b6f94e9cc67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:11.377214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.377187 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istiod-ca-cert" pod "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" (UID: "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:11.377571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.377306 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-certs\") pod \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " Oct 30 05:35:11.377571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.377368 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-podinfo\") pod \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " Oct 30 05:35:11.377571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.377475 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-podinfo\") pod \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\" (UID: \"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c\") " Oct 30 05:35:11.377779 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.377627 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-socket\") pod \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\" (UID: \"0879dd79-1117-46f7-b6e4-8b6f94e9cc67\") " Oct 30 05:35:11.378169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378142 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-token" pod "0879dd79-1117-46f7-b6e4-8b6f94e9cc67" (UID: "0879dd79-1117-46f7-b6e4-8b6f94e9cc67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:11.378266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378181 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-kube-api-access-cqrs9" pod "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" (UID: "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c"). InnerVolumeSpecName "kube-api-access-cqrs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378419 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-socket" pod "0879dd79-1117-46f7-b6e4-8b6f94e9cc67" (UID: "0879dd79-1117-46f7-b6e4-8b6f94e9cc67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378448 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-certs" pod "0879dd79-1117-46f7-b6e4-8b6f94e9cc67" (UID: "0879dd79-1117-46f7-b6e4-8b6f94e9cc67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378609 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378630 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378645 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqrs9\" (UniqueName: \"kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-kube-api-access-cqrs9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378660 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378674 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378686 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378699 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378712 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378724 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378736 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378747 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.378848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378760 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.380844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.378853 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-token" pod "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" (UID: "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:11.380844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.379698 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-envoy" pod "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" (UID: "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.380844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.379737 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-podinfo" pod "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" (UID: "0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:35:11.380844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.380011 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-kube-api-access-sjzvw" pod "0879dd79-1117-46f7-b6e4-8b6f94e9cc67" (UID: "0879dd79-1117-46f7-b6e4-8b6f94e9cc67"). InnerVolumeSpecName "kube-api-access-sjzvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:11.380844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.380790 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-podinfo" pod "0879dd79-1117-46f7-b6e4-8b6f94e9cc67" (UID: "0879dd79-1117-46f7-b6e4-8b6f94e9cc67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:35:11.381130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.381112 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-envoy" pod "0879dd79-1117-46f7-b6e4-8b6f94e9cc67" (UID: "0879dd79-1117-46f7-b6e4-8b6f94e9cc67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.481234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.481172 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.481234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.481209 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjzvw\" (UniqueName: \"kubernetes.io/projected/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-kube-api-access-sjzvw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.481234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.481224 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.481456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.481237 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.481456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.481250 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0879dd79-1117-46f7-b6e4-8b6f94e9cc67-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.481456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.481272 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.654536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.654505 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 05:35:11.736090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.735996 6587 generic.go:356] "Generic (PLEG): container finished" podID="e011432d-3991-4d45-9926-cf7ca5e50194" containerID="97a9040b91f5f71d100b7fdac094b9780f69b83988b8f33d385979dd28b36894" exitCode=0 Oct 30 05:35:11.736394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.736069 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" event={"ID":"e011432d-3991-4d45-9926-cf7ca5e50194","Type":"ContainerDied","Data":"97a9040b91f5f71d100b7fdac094b9780f69b83988b8f33d385979dd28b36894"} Oct 30 05:35:11.736636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.736107 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" Oct 30 05:35:11.736962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.736813 6587 scope.go:122] "RemoveContainer" containerID="97a9040b91f5f71d100b7fdac094b9780f69b83988b8f33d385979dd28b36894" Oct 30 05:35:11.738485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.738456 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml" event={"ID":"e011432d-3991-4d45-9926-cf7ca5e50194","Type":"ContainerDied","Data":"bebf2841b533508688bdce7ed09547a947bf1673c9f17bff844407d9ae0a45dd"} Oct 30 05:35:11.742988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.742954 6587 generic.go:356] "Generic (PLEG): container finished" podID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" containerID="7addf003c8a19630177de95436891b985663ec6ad2874a62d5d48ad7ffa97e0c" exitCode=0 Oct 30 05:35:11.743117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.743096 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" event={"ID":"0879dd79-1117-46f7-b6e4-8b6f94e9cc67","Type":"ContainerDied","Data":"7addf003c8a19630177de95436891b985663ec6ad2874a62d5d48ad7ffa97e0c"} Oct 30 05:35:11.743185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.743147 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" event={"ID":"0879dd79-1117-46f7-b6e4-8b6f94e9cc67","Type":"ContainerDied","Data":"4ce253ce1131dabf85c75c0fea2790f09ab511179415b35cc12d7800ed9b170c"} Oct 30 05:35:11.743307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.743289 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v" Oct 30 05:35:11.746702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.746676 6587 generic.go:356] "Generic (PLEG): container finished" podID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerID="14d8d9568466e6c3a15430b748ea3a4672d3d7c52ec41810dcbab4b811341bc9" exitCode=0 Oct 30 05:35:11.746870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.746853 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" event={"ID":"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c","Type":"ContainerDied","Data":"14d8d9568466e6c3a15430b748ea3a4672d3d7c52ec41810dcbab4b811341bc9"} Oct 30 05:35:11.746948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.746897 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" event={"ID":"0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c","Type":"ContainerDied","Data":"0e06d44bc9d3bca65da2ac3fe20dc456e9fede54121d510a1638ad7ee565bc5a"} Oct 30 05:35:11.747012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.746991 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll" Oct 30 05:35:11.760529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.760507 6587 scope.go:122] "RemoveContainer" containerID="97a9040b91f5f71d100b7fdac094b9780f69b83988b8f33d385979dd28b36894" Oct 30 05:35:11.761217 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:35:11.761185 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a9040b91f5f71d100b7fdac094b9780f69b83988b8f33d385979dd28b36894\": container with ID starting with 97a9040b91f5f71d100b7fdac094b9780f69b83988b8f33d385979dd28b36894 not found: ID does not exist" containerID="97a9040b91f5f71d100b7fdac094b9780f69b83988b8f33d385979dd28b36894" Oct 30 05:35:11.761931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.761227 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a9040b91f5f71d100b7fdac094b9780f69b83988b8f33d385979dd28b36894"} err="failed to get container status \"97a9040b91f5f71d100b7fdac094b9780f69b83988b8f33d385979dd28b36894\": rpc error: code = NotFound desc = could not find container \"97a9040b91f5f71d100b7fdac094b9780f69b83988b8f33d385979dd28b36894\": container with ID starting with 97a9040b91f5f71d100b7fdac094b9780f69b83988b8f33d385979dd28b36894 not found: ID does not exist" Oct 30 05:35:11.761931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.761251 6587 scope.go:122] "RemoveContainer" containerID="7addf003c8a19630177de95436891b985663ec6ad2874a62d5d48ad7ffa97e0c" Oct 30 05:35:11.774699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.774679 6587 scope.go:122] "RemoveContainer" containerID="7addf003c8a19630177de95436891b985663ec6ad2874a62d5d48ad7ffa97e0c" Oct 30 05:35:11.775009 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:35:11.774990 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7addf003c8a19630177de95436891b985663ec6ad2874a62d5d48ad7ffa97e0c\": container with ID starting with 7addf003c8a19630177de95436891b985663ec6ad2874a62d5d48ad7ffa97e0c not found: ID does not exist" containerID="7addf003c8a19630177de95436891b985663ec6ad2874a62d5d48ad7ffa97e0c" Oct 30 05:35:11.775090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.775020 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7addf003c8a19630177de95436891b985663ec6ad2874a62d5d48ad7ffa97e0c"} err="failed to get container status \"7addf003c8a19630177de95436891b985663ec6ad2874a62d5d48ad7ffa97e0c\": rpc error: code = NotFound desc = could not find container \"7addf003c8a19630177de95436891b985663ec6ad2874a62d5d48ad7ffa97e0c\": container with ID starting with 7addf003c8a19630177de95436891b985663ec6ad2874a62d5d48ad7ffa97e0c not found: ID does not exist" Oct 30 05:35:11.775090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.775041 6587 scope.go:122] "RemoveContainer" containerID="14d8d9568466e6c3a15430b748ea3a4672d3d7c52ec41810dcbab4b811341bc9" Oct 30 05:35:11.786160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.786132 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-credential-socket\") pod \"e011432d-3991-4d45-9926-cf7ca5e50194\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " Oct 30 05:35:11.786269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.786243 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-socket\") pod \"e011432d-3991-4d45-9926-cf7ca5e50194\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " Oct 30 05:35:11.786395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.786376 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-istio-token\" (UniqueName: \"kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-istio-token\") pod \"e011432d-3991-4d45-9926-cf7ca5e50194\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " Oct 30 05:35:11.786449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.786423 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e011432d-3991-4d45-9926-cf7ca5e50194-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e011432d-3991-4d45-9926-cf7ca5e50194-istio-podinfo\") pod \"e011432d-3991-4d45-9926-cf7ca5e50194\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " Oct 30 05:35:11.786511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.786496 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-certs\") pod \"e011432d-3991-4d45-9926-cf7ca5e50194\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " Oct 30 05:35:11.786560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.786516 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-credential-socket" pod "e011432d-3991-4d45-9926-cf7ca5e50194" (UID: "e011432d-3991-4d45-9926-cf7ca5e50194"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.786596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.786580 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-kube-api-access-tsv9c\" (UniqueName: \"kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-kube-api-access-tsv9c\") pod \"e011432d-3991-4d45-9926-cf7ca5e50194\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " Oct 30 05:35:11.786667 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.786635 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e011432d-3991-4d45-9926-cf7ca5e50194-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e011432d-3991-4d45-9926-cf7ca5e50194-istiod-ca-cert\") pod \"e011432d-3991-4d45-9926-cf7ca5e50194\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " Oct 30 05:35:11.786768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.786746 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-envoy\") pod \"e011432d-3991-4d45-9926-cf7ca5e50194\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " Oct 30 05:35:11.786812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.786797 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-data\") pod \"e011432d-3991-4d45-9926-cf7ca5e50194\" (UID: \"e011432d-3991-4d45-9926-cf7ca5e50194\") " Oct 30 05:35:11.787014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.786994 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-socket" pod "e011432d-3991-4d45-9926-cf7ca5e50194" (UID: "e011432d-3991-4d45-9926-cf7ca5e50194"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.788006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.787987 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.788086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.788009 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.788275 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.788254 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-data" pod "e011432d-3991-4d45-9926-cf7ca5e50194" (UID: "e011432d-3991-4d45-9926-cf7ca5e50194"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.788604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.788585 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e011432d-3991-4d45-9926-cf7ca5e50194-istiod-ca-cert" pod "e011432d-3991-4d45-9926-cf7ca5e50194" (UID: "e011432d-3991-4d45-9926-cf7ca5e50194"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:11.789066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.789050 6587 scope.go:122] "RemoveContainer" containerID="e48d684c1e26930843b6b216d57b811792f6e1958c5fb2919f8a0756adcc34e0" Oct 30 05:35:11.789204 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.789178 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-istio-token" pod "e011432d-3991-4d45-9926-cf7ca5e50194" (UID: "e011432d-3991-4d45-9926-cf7ca5e50194"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:11.789550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.789520 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-certs" pod "e011432d-3991-4d45-9926-cf7ca5e50194" (UID: "e011432d-3991-4d45-9926-cf7ca5e50194"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.790160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.790124 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-kube-api-access-tsv9c" pod "e011432d-3991-4d45-9926-cf7ca5e50194" (UID: "e011432d-3991-4d45-9926-cf7ca5e50194"). InnerVolumeSpecName "kube-api-access-tsv9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:11.790764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.790714 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-envoy" pod "e011432d-3991-4d45-9926-cf7ca5e50194" (UID: "e011432d-3991-4d45-9926-cf7ca5e50194"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:11.791084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.791059 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e011432d-3991-4d45-9926-cf7ca5e50194-istio-podinfo" pod "e011432d-3991-4d45-9926-cf7ca5e50194" (UID: "e011432d-3991-4d45-9926-cf7ca5e50194"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:35:11.999984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.945208 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.999984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.945243 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e011432d-3991-4d45-9926-cf7ca5e50194-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.999984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.945268 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.999984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.945285 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsv9c\" (UniqueName: \"kubernetes.io/projected/e011432d-3991-4d45-9926-cf7ca5e50194-kube-api-access-tsv9c\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.999984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.945299 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e011432d-3991-4d45-9926-cf7ca5e50194-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.999984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.945312 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:11.999984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:11.945325 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e011432d-3991-4d45-9926-cf7ca5e50194-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:12.048979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:12.041447 6587 scope.go:122] "RemoveContainer" containerID="14d8d9568466e6c3a15430b748ea3a4672d3d7c52ec41810dcbab4b811341bc9" Oct 30 05:35:12.048979 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:35:12.043789 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d8d9568466e6c3a15430b748ea3a4672d3d7c52ec41810dcbab4b811341bc9\": container with ID starting with 14d8d9568466e6c3a15430b748ea3a4672d3d7c52ec41810dcbab4b811341bc9 not found: ID does not exist" containerID="14d8d9568466e6c3a15430b748ea3a4672d3d7c52ec41810dcbab4b811341bc9" Oct 30 05:35:12.048979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:12.043829 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d8d9568466e6c3a15430b748ea3a4672d3d7c52ec41810dcbab4b811341bc9"} err="failed to get container status \"14d8d9568466e6c3a15430b748ea3a4672d3d7c52ec41810dcbab4b811341bc9\": rpc error: code = NotFound desc = could not find container \"14d8d9568466e6c3a15430b748ea3a4672d3d7c52ec41810dcbab4b811341bc9\": container with ID starting with 14d8d9568466e6c3a15430b748ea3a4672d3d7c52ec41810dcbab4b811341bc9 not found: ID does not exist" Oct 30 05:35:12.048979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:12.043856 6587 scope.go:122] "RemoveContainer" containerID="e48d684c1e26930843b6b216d57b811792f6e1958c5fb2919f8a0756adcc34e0" Oct 30 05:35:12.048979 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:35:12.044383 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e48d684c1e26930843b6b216d57b811792f6e1958c5fb2919f8a0756adcc34e0\": container with ID starting with e48d684c1e26930843b6b216d57b811792f6e1958c5fb2919f8a0756adcc34e0 not found: ID does not exist" containerID="e48d684c1e26930843b6b216d57b811792f6e1958c5fb2919f8a0756adcc34e0" Oct 30 05:35:12.048979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:12.044413 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e48d684c1e26930843b6b216d57b811792f6e1958c5fb2919f8a0756adcc34e0"} err="failed to get container status \"e48d684c1e26930843b6b216d57b811792f6e1958c5fb2919f8a0756adcc34e0\": rpc error: code = NotFound desc = could not find container \"e48d684c1e26930843b6b216d57b811792f6e1958c5fb2919f8a0756adcc34e0\": container with ID starting with e48d684c1e26930843b6b216d57b811792f6e1958c5fb2919f8a0756adcc34e0 not found: ID does not exist" Oct 30 05:35:12.069380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:12.064909 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v"] Oct 30 05:35:12.090415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:12.089431 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v"] Oct 30 05:35:12.097160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:12.097131 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll"] Oct 30 05:35:12.103027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:12.102999 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll"] Oct 30 05:35:12.133599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:12.133570 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml"] Oct 30 05:35:12.143556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:12.143529 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml"] Oct 30 05:35:13.584999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:13.584968 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" path="/var/lib/kubelet/pods/0879dd79-1117-46f7-b6e4-8b6f94e9cc67/volumes" Oct 30 05:35:13.585497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:13.585480 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" path="/var/lib/kubelet/pods/0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c/volumes" Oct 30 05:35:13.585994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:13.585979 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e011432d-3991-4d45-9926-cf7ca5e50194" path="/var/lib/kubelet/pods/e011432d-3991-4d45-9926-cf7ca5e50194/volumes" Oct 30 05:35:13.776936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:13.776904 6587 generic.go:356] "Generic (PLEG): container finished" podID="b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc" containerID="f3bf091bde286d02bf2ea48d77d6baf32ded0e349c09825f5935b16269d7a09b" exitCode=0 Oct 30 05:35:13.777117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:13.776986 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" event={"ID":"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc","Type":"ContainerDied","Data":"f3bf091bde286d02bf2ea48d77d6baf32ded0e349c09825f5935b16269d7a09b"} Oct 30 05:35:13.796803 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:35:13.796779 6587 logging.go:55] [core] [Channel #15094 SubChannel #15097]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.55:50052", ServerName: "10.134.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.55:50052: connect: connection refused" Oct 30 05:35:13.796958 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:35:13.796779 6587 logging.go:55] [core] [Channel #15095 SubChannel #15096]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.54:50052", ServerName: "10.134.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.54:50052: connect: connection refused" Oct 30 05:35:13.852168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:13.852067 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4"] Oct 30 05:35:13.884646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:13.882492 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4"] Oct 30 05:35:14.027017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.026988 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" Oct 30 05:35:14.093864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.093663 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-plugins-config-volume\") pod \"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc\" (UID: \"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc\") " Oct 30 05:35:14.094804 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.094657 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-plugins-config-volume" pod "b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc" (UID: "b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:14.094946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.094931 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-kube-api-access-cz5xs\" (UniqueName: \"kubernetes.io/projected/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-kube-api-access-cz5xs\") pod \"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc\" (UID: \"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc\") " Oct 30 05:35:14.097483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.096485 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:14.100334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.100310 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-kube-api-access-cz5xs" pod "b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc" (UID: "b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc"). InnerVolumeSpecName "kube-api-access-cz5xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:14.108556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.108507 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" Oct 30 05:35:14.197316 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.197276 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82c7bf20-a8c4-426e-b692-41e564879c82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c7bf20-a8c4-426e-b692-41e564879c82-plugins-config-volume\") pod \"82c7bf20-a8c4-426e-b692-41e564879c82\" (UID: \"82c7bf20-a8c4-426e-b692-41e564879c82\") " Oct 30 05:35:14.197604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.197579 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82c7bf20-a8c4-426e-b692-41e564879c82-kube-api-access-rcb75\" (UniqueName: \"kubernetes.io/projected/82c7bf20-a8c4-426e-b692-41e564879c82-kube-api-access-rcb75\") pod \"82c7bf20-a8c4-426e-b692-41e564879c82\" (UID: \"82c7bf20-a8c4-426e-b692-41e564879c82\") " Oct 30 05:35:14.198448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.198420 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c7bf20-a8c4-426e-b692-41e564879c82-plugins-config-volume" pod "82c7bf20-a8c4-426e-b692-41e564879c82" (UID: "82c7bf20-a8c4-426e-b692-41e564879c82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:14.198672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.198438 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz5xs\" (UniqueName: \"kubernetes.io/projected/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc-kube-api-access-cz5xs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:14.201444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.201418 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c7bf20-a8c4-426e-b692-41e564879c82-kube-api-access-rcb75" pod "82c7bf20-a8c4-426e-b692-41e564879c82" (UID: "82c7bf20-a8c4-426e-b692-41e564879c82"). InnerVolumeSpecName "kube-api-access-rcb75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:14.300007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.299967 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c7bf20-a8c4-426e-b692-41e564879c82-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:14.300007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.299999 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcb75\" (UniqueName: \"kubernetes.io/projected/82c7bf20-a8c4-426e-b692-41e564879c82-kube-api-access-rcb75\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:14.471496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.471401 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" podUID="00485499-909c-4e26-8206-edb1483727ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:35:14.471671 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.471547 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" podUID="e360f2a8-ce2f-47ae-8e56-a265de776937" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:35:14.807411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.797719 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" podUID="82c7bf20-a8c4-426e-b692-41e564879c82" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.55:50052\" within 1s: context deadline exceeded" Oct 30 05:35:14.807411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.797826 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" Oct 30 05:35:14.807411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.799324 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" podUID="b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.54:50052\" within 1s: context deadline exceeded" Oct 30 05:35:14.807411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.799419 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" Oct 30 05:35:14.899089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.899036 6587 generic.go:356] "Generic (PLEG): container finished" podID="82c7bf20-a8c4-426e-b692-41e564879c82" containerID="1a195695ce00996f05c6d3bd2cd7a0843043f2f61972e773352e231857e244ce" exitCode=0 Oct 30 05:35:14.901670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.901638 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" Oct 30 05:35:14.911417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.902632 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" event={"ID":"82c7bf20-a8c4-426e-b692-41e564879c82","Type":"ContainerDied","Data":"1a195695ce00996f05c6d3bd2cd7a0843043f2f61972e773352e231857e244ce"} Oct 30 05:35:14.911417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.907628 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z" event={"ID":"82c7bf20-a8c4-426e-b692-41e564879c82","Type":"ContainerDied","Data":"f7e119773228bc8d12b494766fc923b0fbaa22729345d2e59f552ec3a4c13a64"} Oct 30 05:35:14.911417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.907676 6587 scope.go:122] "RemoveContainer" containerID="1a195695ce00996f05c6d3bd2cd7a0843043f2f61972e773352e231857e244ce" Oct 30 05:35:14.944678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.926991 6587 generic.go:356] "Generic (PLEG): container finished" podID="00485499-909c-4e26-8206-edb1483727ce" containerID="fa141672bc6b8a6c589b34da7fd732178444548712641360a092a93b7e2b58a2" exitCode=0 Oct 30 05:35:14.944678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.927099 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" event={"ID":"00485499-909c-4e26-8206-edb1483727ce","Type":"ContainerDied","Data":"fa141672bc6b8a6c589b34da7fd732178444548712641360a092a93b7e2b58a2"} Oct 30 05:35:14.947050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.946974 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" event={"ID":"b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc","Type":"ContainerDied","Data":"4df78f7eb7eaf327e24777ea3967422d96134ae94ce8b68ea58dc908ec645793"} Oct 30 05:35:14.947156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.947055 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5" Oct 30 05:35:14.996188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.991685 6587 scope.go:122] "RemoveContainer" containerID="1a195695ce00996f05c6d3bd2cd7a0843043f2f61972e773352e231857e244ce" Oct 30 05:35:14.996188 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:35:14.992037 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a195695ce00996f05c6d3bd2cd7a0843043f2f61972e773352e231857e244ce\": container with ID starting with 1a195695ce00996f05c6d3bd2cd7a0843043f2f61972e773352e231857e244ce not found: ID does not exist" containerID="1a195695ce00996f05c6d3bd2cd7a0843043f2f61972e773352e231857e244ce" Oct 30 05:35:14.996188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.992064 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a195695ce00996f05c6d3bd2cd7a0843043f2f61972e773352e231857e244ce"} err="failed to get container status \"1a195695ce00996f05c6d3bd2cd7a0843043f2f61972e773352e231857e244ce\": rpc error: code = NotFound desc = could not find container \"1a195695ce00996f05c6d3bd2cd7a0843043f2f61972e773352e231857e244ce\": container with ID starting with 1a195695ce00996f05c6d3bd2cd7a0843043f2f61972e773352e231857e244ce not found: ID does not exist" Oct 30 05:35:14.996188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:14.992082 6587 scope.go:122] "RemoveContainer" containerID="f3bf091bde286d02bf2ea48d77d6baf32ded0e349c09825f5935b16269d7a09b" Oct 30 05:35:15.042899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.042862 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z"] Oct 30 05:35:15.094432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.092089 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z"] Oct 30 05:35:15.102218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.101320 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5"] Oct 30 05:35:15.107800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.107775 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5"] Oct 30 05:35:15.163403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.158565 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" Oct 30 05:35:15.226988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.226955 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" Oct 30 05:35:15.259478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.259391 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00485499-909c-4e26-8206-edb1483727ce-kube-api-access-k5684\" (UniqueName: \"kubernetes.io/projected/00485499-909c-4e26-8206-edb1483727ce-kube-api-access-k5684\") pod \"00485499-909c-4e26-8206-edb1483727ce\" (UID: \"00485499-909c-4e26-8206-edb1483727ce\") " Oct 30 05:35:15.259660 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.259595 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00485499-909c-4e26-8206-edb1483727ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00485499-909c-4e26-8206-edb1483727ce-plugins-config-volume\") pod \"00485499-909c-4e26-8206-edb1483727ce\" (UID: \"00485499-909c-4e26-8206-edb1483727ce\") " Oct 30 05:35:15.260302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.260274 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00485499-909c-4e26-8206-edb1483727ce-plugins-config-volume" pod "00485499-909c-4e26-8206-edb1483727ce" (UID: "00485499-909c-4e26-8206-edb1483727ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:15.261835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.261804 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00485499-909c-4e26-8206-edb1483727ce-kube-api-access-k5684" pod "00485499-909c-4e26-8206-edb1483727ce" (UID: "00485499-909c-4e26-8206-edb1483727ce"). InnerVolumeSpecName "kube-api-access-k5684". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:15.361198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.361162 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e360f2a8-ce2f-47ae-8e56-a265de776937-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e360f2a8-ce2f-47ae-8e56-a265de776937-plugins-config-volume\") pod \"e360f2a8-ce2f-47ae-8e56-a265de776937\" (UID: \"e360f2a8-ce2f-47ae-8e56-a265de776937\") " Oct 30 05:35:15.361376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.361241 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e360f2a8-ce2f-47ae-8e56-a265de776937-kube-api-access-hsp6z\" (UniqueName: \"kubernetes.io/projected/e360f2a8-ce2f-47ae-8e56-a265de776937-kube-api-access-hsp6z\") pod \"e360f2a8-ce2f-47ae-8e56-a265de776937\" (UID: \"e360f2a8-ce2f-47ae-8e56-a265de776937\") " Oct 30 05:35:15.361601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.361577 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e360f2a8-ce2f-47ae-8e56-a265de776937-plugins-config-volume" pod "e360f2a8-ce2f-47ae-8e56-a265de776937" (UID: "e360f2a8-ce2f-47ae-8e56-a265de776937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:15.361949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.361931 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5684\" (UniqueName: \"kubernetes.io/projected/00485499-909c-4e26-8206-edb1483727ce-kube-api-access-k5684\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:15.362001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.361952 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e360f2a8-ce2f-47ae-8e56-a265de776937-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:15.362001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.361962 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00485499-909c-4e26-8206-edb1483727ce-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:15.363485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.363453 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e360f2a8-ce2f-47ae-8e56-a265de776937-kube-api-access-hsp6z" pod "e360f2a8-ce2f-47ae-8e56-a265de776937" (UID: "e360f2a8-ce2f-47ae-8e56-a265de776937"). InnerVolumeSpecName "kube-api-access-hsp6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:15.463808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.463774 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsp6z\" (UniqueName: \"kubernetes.io/projected/e360f2a8-ce2f-47ae-8e56-a265de776937-kube-api-access-hsp6z\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:15.585324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.585291 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82c7bf20-a8c4-426e-b692-41e564879c82" path="/var/lib/kubelet/pods/82c7bf20-a8c4-426e-b692-41e564879c82/volumes" Oct 30 05:35:15.585830 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.585813 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc" path="/var/lib/kubelet/pods/b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc/volumes" Oct 30 05:35:15.963985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.963887 6587 generic.go:356] "Generic (PLEG): container finished" podID="e360f2a8-ce2f-47ae-8e56-a265de776937" containerID="e218bdb67d883c7e2b607f467c91b8c9758236f1f6d2f52eb8a565733c8ef0cb" exitCode=0 Oct 30 05:35:15.963985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.963944 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" Oct 30 05:35:15.964580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.963977 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" event={"ID":"e360f2a8-ce2f-47ae-8e56-a265de776937","Type":"ContainerDied","Data":"e218bdb67d883c7e2b607f467c91b8c9758236f1f6d2f52eb8a565733c8ef0cb"} Oct 30 05:35:15.964580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.964039 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h" event={"ID":"e360f2a8-ce2f-47ae-8e56-a265de776937","Type":"ContainerDied","Data":"52de7f34792e61e7e9f8ee23e0a9453a7d98c127ec0a59431f85e7f64dd7062b"} Oct 30 05:35:15.964580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.964071 6587 scope.go:122] "RemoveContainer" containerID="e218bdb67d883c7e2b607f467c91b8c9758236f1f6d2f52eb8a565733c8ef0cb" Oct 30 05:35:15.965769 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.965728 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" event={"ID":"00485499-909c-4e26-8206-edb1483727ce","Type":"ContainerDied","Data":"1a2144f37e0c6e455f4f3568b874164e7597cf43734a7712436628a15c19ffcb"} Oct 30 05:35:15.965852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.965819 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm" Oct 30 05:35:15.980568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.980548 6587 scope.go:122] "RemoveContainer" containerID="e218bdb67d883c7e2b607f467c91b8c9758236f1f6d2f52eb8a565733c8ef0cb" Oct 30 05:35:15.980887 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:35:15.980857 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e218bdb67d883c7e2b607f467c91b8c9758236f1f6d2f52eb8a565733c8ef0cb\": container with ID starting with e218bdb67d883c7e2b607f467c91b8c9758236f1f6d2f52eb8a565733c8ef0cb not found: ID does not exist" containerID="e218bdb67d883c7e2b607f467c91b8c9758236f1f6d2f52eb8a565733c8ef0cb" Oct 30 05:35:15.980955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.980899 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e218bdb67d883c7e2b607f467c91b8c9758236f1f6d2f52eb8a565733c8ef0cb"} err="failed to get container status \"e218bdb67d883c7e2b607f467c91b8c9758236f1f6d2f52eb8a565733c8ef0cb\": rpc error: code = NotFound desc = could not find container \"e218bdb67d883c7e2b607f467c91b8c9758236f1f6d2f52eb8a565733c8ef0cb\": container with ID starting with e218bdb67d883c7e2b607f467c91b8c9758236f1f6d2f52eb8a565733c8ef0cb not found: ID does not exist" Oct 30 05:35:15.980955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:15.980917 6587 scope.go:122] "RemoveContainer" containerID="fa141672bc6b8a6c589b34da7fd732178444548712641360a092a93b7e2b58a2" Oct 30 05:35:16.024202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:16.024180 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h"] Oct 30 05:35:16.039047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:16.039021 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h"] Oct 30 05:35:16.072792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:16.072764 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm"] Oct 30 05:35:16.082066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:16.082045 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm"] Oct 30 05:35:17.586527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:17.586491 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00485499-909c-4e26-8206-edb1483727ce" path="/var/lib/kubelet/pods/00485499-909c-4e26-8206-edb1483727ce/volumes" Oct 30 05:35:17.587152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:17.587044 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e360f2a8-ce2f-47ae-8e56-a265de776937" path="/var/lib/kubelet/pods/e360f2a8-ce2f-47ae-8e56-a265de776937/volumes" Oct 30 05:35:18.852839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:18.852772 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" podUID="85b62bb7-1e0c-470c-b69e-fbd26617030d" containerName="payload-processing" containerID="cri-o://fcdebe8140172f7308ec79fb99ca67d6bcc702e7e5d498747a7db32174149dfa" gracePeriod=25 Oct 30 05:35:18.883090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:18.883055 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" podUID="182c089e-21c6-470b-9dcd-85c1ced34758" containerName="payload-pre-processing" containerID="cri-o://2fa649156ae366cb6305078e4b009725f22b197d20fd8084447dc15a5a51d71a" gracePeriod=25 Oct 30 05:35:18.967004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:18.966974 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv"] Oct 30 05:35:18.967248 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:18.967226 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" podUID="160a7128-f92e-420b-bd00-c44142af02bc" containerName="istio-proxy" containerID="cri-o://976450c7c8d070691b9e44338a8fd6c203deb82ef7d38565ae0356a1874679b7" gracePeriod=30 Oct 30 05:35:18.979564 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:18.979528 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz"] Oct 30 05:35:18.979768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:18.979742 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" podUID="3ffeb6a7-0efc-4336-89f8-2704a03d1398" containerName="istio-proxy" containerID="cri-o://c2de7d60740f9b53c4e3454bb42b8eee64c9c96fc5b36525432f42c7cd29cdac" gracePeriod=30 Oct 30 05:35:18.986139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:18.986104 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c"] Oct 30 05:35:18.986376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:18.986337 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" podUID="25d5a6a4-a651-41fa-8b85-02700b6e12e9" containerName="istio-proxy" containerID="cri-o://24d3ca7d646667ab144d96fcc3f191e8cd7b9137917ef92c69627cd30ecde2ac" gracePeriod=30 Oct 30 05:35:19.013043 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.013007 6587 generic.go:356] "Generic (PLEG): container finished" podID="182c089e-21c6-470b-9dcd-85c1ced34758" containerID="2fa649156ae366cb6305078e4b009725f22b197d20fd8084447dc15a5a51d71a" exitCode=0 Oct 30 05:35:19.013952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.013280 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" event={"ID":"182c089e-21c6-470b-9dcd-85c1ced34758","Type":"ContainerDied","Data":"2fa649156ae366cb6305078e4b009725f22b197d20fd8084447dc15a5a51d71a"} Oct 30 05:35:19.016707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.016676 6587 generic.go:356] "Generic (PLEG): container finished" podID="85b62bb7-1e0c-470c-b69e-fbd26617030d" containerID="fcdebe8140172f7308ec79fb99ca67d6bcc702e7e5d498747a7db32174149dfa" exitCode=0 Oct 30 05:35:19.017088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.016719 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" event={"ID":"85b62bb7-1e0c-470c-b69e-fbd26617030d","Type":"ContainerDied","Data":"fcdebe8140172f7308ec79fb99ca67d6bcc702e7e5d498747a7db32174149dfa"} Oct 30 05:35:19.248042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.248011 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" Oct 30 05:35:19.322256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.322219 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" Oct 30 05:35:19.353611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.353575 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b62bb7-1e0c-470c-b69e-fbd26617030d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b62bb7-1e0c-470c-b69e-fbd26617030d-plugins-config-volume\") pod \"85b62bb7-1e0c-470c-b69e-fbd26617030d\" (UID: \"85b62bb7-1e0c-470c-b69e-fbd26617030d\") " Oct 30 05:35:19.353838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.353821 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182c089e-21c6-470b-9dcd-85c1ced34758-kube-api-access-57dkz\" (UniqueName: \"kubernetes.io/projected/182c089e-21c6-470b-9dcd-85c1ced34758-kube-api-access-57dkz\") pod \"182c089e-21c6-470b-9dcd-85c1ced34758\" (UID: \"182c089e-21c6-470b-9dcd-85c1ced34758\") " Oct 30 05:35:19.353928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.353908 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182c089e-21c6-470b-9dcd-85c1ced34758-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182c089e-21c6-470b-9dcd-85c1ced34758-plugins-config-volume\") pod \"182c089e-21c6-470b-9dcd-85c1ced34758\" (UID: \"182c089e-21c6-470b-9dcd-85c1ced34758\") " Oct 30 05:35:19.354025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.354010 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b62bb7-1e0c-470c-b69e-fbd26617030d-kube-api-access-gc4ps\" (UniqueName: \"kubernetes.io/projected/85b62bb7-1e0c-470c-b69e-fbd26617030d-kube-api-access-gc4ps\") pod \"85b62bb7-1e0c-470c-b69e-fbd26617030d\" (UID: \"85b62bb7-1e0c-470c-b69e-fbd26617030d\") " Oct 30 05:35:19.354085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.354039 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b62bb7-1e0c-470c-b69e-fbd26617030d-plugins-config-volume" pod "85b62bb7-1e0c-470c-b69e-fbd26617030d" (UID: "85b62bb7-1e0c-470c-b69e-fbd26617030d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:19.354786 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.354764 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b62bb7-1e0c-470c-b69e-fbd26617030d-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:19.355095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.354806 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182c089e-21c6-470b-9dcd-85c1ced34758-plugins-config-volume" pod "182c089e-21c6-470b-9dcd-85c1ced34758" (UID: "182c089e-21c6-470b-9dcd-85c1ced34758"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:19.356981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.356955 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b62bb7-1e0c-470c-b69e-fbd26617030d-kube-api-access-gc4ps" pod "85b62bb7-1e0c-470c-b69e-fbd26617030d" (UID: "85b62bb7-1e0c-470c-b69e-fbd26617030d"). InnerVolumeSpecName "kube-api-access-gc4ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:19.357331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.357302 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182c089e-21c6-470b-9dcd-85c1ced34758-kube-api-access-57dkz" pod "182c089e-21c6-470b-9dcd-85c1ced34758" (UID: "182c089e-21c6-470b-9dcd-85c1ced34758"). InnerVolumeSpecName "kube-api-access-57dkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:19.456614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.456526 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57dkz\" (UniqueName: \"kubernetes.io/projected/182c089e-21c6-470b-9dcd-85c1ced34758-kube-api-access-57dkz\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:19.456614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.456556 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182c089e-21c6-470b-9dcd-85c1ced34758-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:19.456614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:19.456567 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc4ps\" (UniqueName: \"kubernetes.io/projected/85b62bb7-1e0c-470c-b69e-fbd26617030d-kube-api-access-gc4ps\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:20.028300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.028272 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" Oct 30 05:35:20.028788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.028301 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4" event={"ID":"85b62bb7-1e0c-470c-b69e-fbd26617030d","Type":"ContainerDied","Data":"52810d5fa5a0e8b6df1f4c01b57de7bb0e3feb8de3d123c50ecf40f875a110f1"} Oct 30 05:35:20.028788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.028447 6587 scope.go:122] "RemoveContainer" containerID="fcdebe8140172f7308ec79fb99ca67d6bcc702e7e5d498747a7db32174149dfa" Oct 30 05:35:20.030481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.030463 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" Oct 30 05:35:20.030609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.030489 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4" event={"ID":"182c089e-21c6-470b-9dcd-85c1ced34758","Type":"ContainerDied","Data":"14164e1d45d4edb443b0c02a64dc5b1607e8e54e5c8265fcb5b8a4124b8538af"} Oct 30 05:35:20.040881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.040863 6587 scope.go:122] "RemoveContainer" containerID="2fa649156ae366cb6305078e4b009725f22b197d20fd8084447dc15a5a51d71a" Oct 30 05:35:20.090609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.090582 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4"] Oct 30 05:35:20.096198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.096170 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4"] Oct 30 05:35:20.129792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.129714 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4"] Oct 30 05:35:20.138024 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.137995 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4"] Oct 30 05:35:20.989151 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.989110 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm"] Oct 30 05:35:20.990128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990096 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" Oct 30 05:35:20.990128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990117 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" Oct 30 05:35:20.990128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990136 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e360f2a8-ce2f-47ae-8e56-a265de776937" containerName="payload-pre-processing" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990142 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e360f2a8-ce2f-47ae-8e56-a265de776937" containerName="payload-pre-processing" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990166 6587 cpu_manager.go:414] "containerMap: removing container" podUID="82c7bf20-a8c4-426e-b692-41e564879c82" containerName="payload-pre-processing" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990171 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82c7bf20-a8c4-426e-b692-41e564879c82" containerName="payload-pre-processing" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990180 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc" containerName="payload-processing" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990184 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc" containerName="payload-processing" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990195 6587 cpu_manager.go:414] "containerMap: removing container" podUID="182c089e-21c6-470b-9dcd-85c1ced34758" containerName="payload-pre-processing" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990200 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182c089e-21c6-470b-9dcd-85c1ced34758" containerName="payload-pre-processing" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990209 6587 cpu_manager.go:414] "containerMap: removing container" podUID="85b62bb7-1e0c-470c-b69e-fbd26617030d" containerName="payload-processing" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990214 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b62bb7-1e0c-470c-b69e-fbd26617030d" containerName="payload-processing" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990228 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e7dfc5c3-dfbc-407e-889a-5f8800106be2" containerName="revoke-keys" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990234 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7dfc5c3-dfbc-407e-889a-5f8800106be2" containerName="revoke-keys" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990241 6587 cpu_manager.go:414] "containerMap: removing container" podUID="00485499-909c-4e26-8206-edb1483727ce" containerName="payload-processing" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990245 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00485499-909c-4e26-8206-edb1483727ce" containerName="payload-processing" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990252 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e011432d-3991-4d45-9926-cf7ca5e50194" containerName="istio-proxy" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990256 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e011432d-3991-4d45-9926-cf7ca5e50194" containerName="istio-proxy" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990275 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" containerName="istio-proxy" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990279 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" containerName="istio-proxy" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990426 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0879dd79-1117-46f7-b6e4-8b6f94e9cc67" containerName="istio-proxy" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990444 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990454 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e360f2a8-ce2f-47ae-8e56-a265de776937" containerName="payload-pre-processing" Oct 30 05:35:20.990451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990464 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="82c7bf20-a8c4-426e-b692-41e564879c82" containerName="payload-pre-processing" Oct 30 05:35:20.991613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990471 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02adafe-f0d8-406c-b9d0-3dc6b6f51ebc" containerName="payload-processing" Oct 30 05:35:20.991613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990478 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="182c089e-21c6-470b-9dcd-85c1ced34758" containerName="payload-pre-processing" Oct 30 05:35:20.991613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990485 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b62bb7-1e0c-470c-b69e-fbd26617030d" containerName="payload-processing" Oct 30 05:35:20.991613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990494 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7dfc5c3-dfbc-407e-889a-5f8800106be2" containerName="revoke-keys" Oct 30 05:35:20.991613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990499 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" Oct 30 05:35:20.991613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990505 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="00485499-909c-4e26-8206-edb1483727ce" containerName="payload-processing" Oct 30 05:35:20.991613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.990510 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e011432d-3991-4d45-9926-cf7ca5e50194" containerName="istio-proxy" Oct 30 05:35:20.995570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:20.995549 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" Oct 30 05:35:21.089404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.086918 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7a4600\"" Oct 30 05:35:21.089404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.087078 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7a4600-dockercfg-292nj\"" Oct 30 05:35:21.402874 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.374132 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clrl7\" (UniqueName: \"kubernetes.io/projected/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-kube-api-access-clrl7\") pod \"payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm\" (UID: \"5d405eaf-ba36-4e25-a606-d94afb1d7ae9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" Oct 30 05:35:21.402874 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.374333 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm\" (UID: \"5d405eaf-ba36-4e25-a606-d94afb1d7ae9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" Oct 30 05:35:21.438687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.438415 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm"] Oct 30 05:35:21.455561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.453183 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f"] Oct 30 05:35:21.455561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.455122 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" Oct 30 05:35:21.455561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.455144 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7f97f0-1587-4cd2-8d3d-eecb48dadc1c" containerName="istio-proxy" Oct 30 05:35:21.460426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.460403 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" Oct 30 05:35:21.462572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.462551 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f"] Oct 30 05:35:21.523502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.523467 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clrl7\" (UniqueName: \"kubernetes.io/projected/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-kube-api-access-clrl7\") pod \"payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm\" (UID: \"5d405eaf-ba36-4e25-a606-d94afb1d7ae9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" Oct 30 05:35:21.523726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.523558 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm\" (UID: \"5d405eaf-ba36-4e25-a606-d94afb1d7ae9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" Oct 30 05:35:21.524277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.524246 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm\" (UID: \"5d405eaf-ba36-4e25-a606-d94afb1d7ae9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" Oct 30 05:35:21.551340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.551305 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clrl7\" (UniqueName: \"kubernetes.io/projected/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-kube-api-access-clrl7\") pod \"payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm\" (UID: \"5d405eaf-ba36-4e25-a606-d94afb1d7ae9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" Oct 30 05:35:21.594321 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.594281 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182c089e-21c6-470b-9dcd-85c1ced34758" path="/var/lib/kubelet/pods/182c089e-21c6-470b-9dcd-85c1ced34758/volumes" Oct 30 05:35:21.596130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.596103 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b62bb7-1e0c-470c-b69e-fbd26617030d" path="/var/lib/kubelet/pods/85b62bb7-1e0c-470c-b69e-fbd26617030d/volumes" Oct 30 05:35:21.611540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.611510 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" Oct 30 05:35:21.627160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.627121 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8j8p\" (UniqueName: \"kubernetes.io/projected/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-kube-api-access-k8j8p\") pod \"payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f\" (UID: \"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" Oct 30 05:35:21.627679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.627636 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f\" (UID: \"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" Oct 30 05:35:21.733301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.733247 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f\" (UID: \"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" Oct 30 05:35:21.733465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.733349 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8j8p\" (UniqueName: \"kubernetes.io/projected/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-kube-api-access-k8j8p\") pod \"payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f\" (UID: \"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" Oct 30 05:35:21.735600 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.735529 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f\" (UID: \"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" Oct 30 05:35:21.757297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.757252 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8j8p\" (UniqueName: \"kubernetes.io/projected/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-kube-api-access-k8j8p\") pod \"payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f\" (UID: \"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" Oct 30 05:35:21.779687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.779645 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" Oct 30 05:35:21.990606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:21.990496 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm"] Oct 30 05:35:21.999269 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:35:21.999233 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d405eaf_ba36_4e25_a606_d94afb1d7ae9.slice/crio-ba007e360d2f3a829c3a53a5cfa41cdb4f8cbebd1cc143ddce37f0331854c041 WatchSource:0}: Error finding container ba007e360d2f3a829c3a53a5cfa41cdb4f8cbebd1cc143ddce37f0331854c041: Status 404 returned error can't find the container with id ba007e360d2f3a829c3a53a5cfa41cdb4f8cbebd1cc143ddce37f0331854c041 Oct 30 05:35:22.146856 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:35:22.146816 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa5dc1f_4f2b_45b9_b74b_d177d8d9050e.slice/crio-85787079b420ec273eb1e6eed26b591161414bbd478f30b932c890f5fe5d5883 WatchSource:0}: Error finding container 85787079b420ec273eb1e6eed26b591161414bbd478f30b932c890f5fe5d5883: Status 404 returned error can't find the container with id 85787079b420ec273eb1e6eed26b591161414bbd478f30b932c890f5fe5d5883 Oct 30 05:35:22.148268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:22.148235 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f"] Oct 30 05:35:22.347801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:22.347759 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" event={"ID":"5d405eaf-ba36-4e25-a606-d94afb1d7ae9","Type":"ContainerStarted","Data":"e97cf30b8f1caf4c86f90b8d00892b8ac848cd9bb8ff430f91c48a110e0ae125"} Oct 30 05:35:22.348005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:22.347828 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" event={"ID":"5d405eaf-ba36-4e25-a606-d94afb1d7ae9","Type":"ContainerStarted","Data":"ba007e360d2f3a829c3a53a5cfa41cdb4f8cbebd1cc143ddce37f0331854c041"} Oct 30 05:35:22.348005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:22.347864 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" Oct 30 05:35:22.349705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:22.349678 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" event={"ID":"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e","Type":"ContainerStarted","Data":"9e81a7c5810b05e439537be5bb091c6c8b4322955df2832ffb01b3f2a8751281"} Oct 30 05:35:22.349836 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:22.349726 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" Oct 30 05:35:22.349836 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:22.349750 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" event={"ID":"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e","Type":"ContainerStarted","Data":"85787079b420ec273eb1e6eed26b591161414bbd478f30b932c890f5fe5d5883"} Oct 30 05:35:22.416299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:22.416251 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" podStartSLOduration=2.416234811 podStartE2EDuration="2.416234811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:35:22.410857176 +0000 UTC m=+3046.938623963" watchObservedRunningTime="2026-10-30 05:35:22.416234811 +0000 UTC m=+3046.944001596" Oct 30 05:35:22.477122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:22.477067 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" podStartSLOduration=1.477052832 podStartE2EDuration="1.477052832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:35:22.471933854 +0000 UTC m=+3046.999700661" watchObservedRunningTime="2026-10-30 05:35:22.477052832 +0000 UTC m=+3047.004819626" Oct 30 05:35:22.992963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:22.992917 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:35:22.993233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:22.993195 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" podUID="160a7128-f92e-420b-bd00-c44142af02bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:35:23.679618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:23.679520 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:35:23.679618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:23.679595 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" podUID="25d5a6a4-a651-41fa-8b85-02700b6e12e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:35:24.380848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.380809 6587 generic.go:356] "Generic (PLEG): container finished" podID="160a7128-f92e-420b-bd00-c44142af02bc" containerID="976450c7c8d070691b9e44338a8fd6c203deb82ef7d38565ae0356a1874679b7" exitCode=0 Oct 30 05:35:24.381032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.381009 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" event={"ID":"160a7128-f92e-420b-bd00-c44142af02bc","Type":"ContainerDied","Data":"976450c7c8d070691b9e44338a8fd6c203deb82ef7d38565ae0356a1874679b7"} Oct 30 05:35:24.386060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.385860 6587 generic.go:356] "Generic (PLEG): container finished" podID="25d5a6a4-a651-41fa-8b85-02700b6e12e9" containerID="24d3ca7d646667ab144d96fcc3f191e8cd7b9137917ef92c69627cd30ecde2ac" exitCode=0 Oct 30 05:35:24.386214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.386130 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" event={"ID":"25d5a6a4-a651-41fa-8b85-02700b6e12e9","Type":"ContainerDied","Data":"24d3ca7d646667ab144d96fcc3f191e8cd7b9137917ef92c69627cd30ecde2ac"} Oct 30 05:35:24.388208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.388120 6587 generic.go:356] "Generic (PLEG): container finished" podID="3ffeb6a7-0efc-4336-89f8-2704a03d1398" containerID="c2de7d60740f9b53c4e3454bb42b8eee64c9c96fc5b36525432f42c7cd29cdac" exitCode=0 Oct 30 05:35:24.388208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.388162 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" event={"ID":"3ffeb6a7-0efc-4336-89f8-2704a03d1398","Type":"ContainerDied","Data":"c2de7d60740f9b53c4e3454bb42b8eee64c9c96fc5b36525432f42c7cd29cdac"} Oct 30 05:35:24.427455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.422726 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:35:24.464191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.464166 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:35:24.485425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.485395 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istiod-ca-cert\") pod \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " Oct 30 05:35:24.485637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.485448 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-envoy\") pod \"160a7128-f92e-420b-bd00-c44142af02bc\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " Oct 30 05:35:24.485637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.485518 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-data\") pod \"160a7128-f92e-420b-bd00-c44142af02bc\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " Oct 30 05:35:24.485637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.485561 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-kube-api-access-vdwxz\" (UniqueName: \"kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-kube-api-access-vdwxz\") pod \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " Oct 30 05:35:24.485637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.485626 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-podinfo\") pod \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " Oct 30 05:35:24.485870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.485720 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160a7128-f92e-420b-bd00-c44142af02bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160a7128-f92e-420b-bd00-c44142af02bc-istiod-ca-cert\") pod \"160a7128-f92e-420b-bd00-c44142af02bc\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " Oct 30 05:35:24.485870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.485794 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-certs\") pod \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " Oct 30 05:35:24.485870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.485837 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-socket\") pod \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " Oct 30 05:35:24.486039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.485896 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-data\") pod \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " Oct 30 05:35:24.486039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.485936 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-kube-api-access-pbqd6\" (UniqueName: \"kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-kube-api-access-pbqd6\") pod \"160a7128-f92e-420b-bd00-c44142af02bc\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " Oct 30 05:35:24.486219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.486034 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/160a7128-f92e-420b-bd00-c44142af02bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160a7128-f92e-420b-bd00-c44142af02bc-istio-podinfo\") pod \"160a7128-f92e-420b-bd00-c44142af02bc\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " Oct 30 05:35:24.486219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.486066 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-token\" (UniqueName: \"kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-token\") pod \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " Oct 30 05:35:24.486219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.486106 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-credential-socket\") pod \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " Oct 30 05:35:24.486219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.486137 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-istio-token\" (UniqueName: \"kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-istio-token\") pod \"160a7128-f92e-420b-bd00-c44142af02bc\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " Oct 30 05:35:24.486219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.486199 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-socket\") pod \"160a7128-f92e-420b-bd00-c44142af02bc\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " Oct 30 05:35:24.486525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.486261 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-certs\") pod \"160a7128-f92e-420b-bd00-c44142af02bc\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " Oct 30 05:35:24.486525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.486324 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-envoy\") pod \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\" (UID: \"3ffeb6a7-0efc-4336-89f8-2704a03d1398\") " Oct 30 05:35:24.486525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.486386 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-credential-socket\") pod \"160a7128-f92e-420b-bd00-c44142af02bc\" (UID: \"160a7128-f92e-420b-bd00-c44142af02bc\") " Oct 30 05:35:24.486910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.485719 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istiod-ca-cert" pod "3ffeb6a7-0efc-4336-89f8-2704a03d1398" (UID: "3ffeb6a7-0efc-4336-89f8-2704a03d1398"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:24.487331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.487267 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-credential-socket" pod "160a7128-f92e-420b-bd00-c44142af02bc" (UID: "160a7128-f92e-420b-bd00-c44142af02bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.488145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.487711 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-credential-socket" pod "3ffeb6a7-0efc-4336-89f8-2704a03d1398" (UID: "3ffeb6a7-0efc-4336-89f8-2704a03d1398"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.488145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.485899 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-data" pod "160a7128-f92e-420b-bd00-c44142af02bc" (UID: "160a7128-f92e-420b-bd00-c44142af02bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.488145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.486502 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-certs" pod "3ffeb6a7-0efc-4336-89f8-2704a03d1398" (UID: "3ffeb6a7-0efc-4336-89f8-2704a03d1398"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.488145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.486882 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160a7128-f92e-420b-bd00-c44142af02bc-istiod-ca-cert" pod "160a7128-f92e-420b-bd00-c44142af02bc" (UID: "160a7128-f92e-420b-bd00-c44142af02bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:24.488145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.487945 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-certs" pod "160a7128-f92e-420b-bd00-c44142af02bc" (UID: "160a7128-f92e-420b-bd00-c44142af02bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.488145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.487951 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-data" pod "3ffeb6a7-0efc-4336-89f8-2704a03d1398" (UID: "3ffeb6a7-0efc-4336-89f8-2704a03d1398"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.488612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.488236 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-socket" pod "3ffeb6a7-0efc-4336-89f8-2704a03d1398" (UID: "3ffeb6a7-0efc-4336-89f8-2704a03d1398"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.488612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.488245 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-socket" pod "160a7128-f92e-420b-bd00-c44142af02bc" (UID: "160a7128-f92e-420b-bd00-c44142af02bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.489193 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.489167 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160a7128-f92e-420b-bd00-c44142af02bc-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.489309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.489198 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.489309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.489214 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.489309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.489228 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.489309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.489241 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.489309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.489255 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.489309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.489277 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.489309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.489291 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.489309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.489304 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.492544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.489318 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.492544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.490208 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-envoy" pod "3ffeb6a7-0efc-4336-89f8-2704a03d1398" (UID: "3ffeb6a7-0efc-4336-89f8-2704a03d1398"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.492544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.492014 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-istio-token" pod "160a7128-f92e-420b-bd00-c44142af02bc" (UID: "160a7128-f92e-420b-bd00-c44142af02bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:24.492710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.492686 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-kube-api-access-vdwxz" pod "3ffeb6a7-0efc-4336-89f8-2704a03d1398" (UID: "3ffeb6a7-0efc-4336-89f8-2704a03d1398"). InnerVolumeSpecName "kube-api-access-vdwxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:24.492960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.492936 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-kube-api-access-pbqd6" pod "160a7128-f92e-420b-bd00-c44142af02bc" (UID: "160a7128-f92e-420b-bd00-c44142af02bc"). InnerVolumeSpecName "kube-api-access-pbqd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:24.493110 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.493094 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/160a7128-f92e-420b-bd00-c44142af02bc-istio-podinfo" pod "160a7128-f92e-420b-bd00-c44142af02bc" (UID: "160a7128-f92e-420b-bd00-c44142af02bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:35:24.495794 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.495766 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-podinfo" pod "3ffeb6a7-0efc-4336-89f8-2704a03d1398" (UID: "3ffeb6a7-0efc-4336-89f8-2704a03d1398"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:35:24.495908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.495863 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-envoy" pod "160a7128-f92e-420b-bd00-c44142af02bc" (UID: "160a7128-f92e-420b-bd00-c44142af02bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.499167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.498558 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-token" pod "3ffeb6a7-0efc-4336-89f8-2704a03d1398" (UID: "3ffeb6a7-0efc-4336-89f8-2704a03d1398"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:24.568516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.568481 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:35:24.581763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.581734 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8"] Oct 30 05:35:24.590710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.590681 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istiod-ca-cert\") pod \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " Oct 30 05:35:24.590904 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.590790 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-credential-socket\") pod \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " Oct 30 05:35:24.590904 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.590817 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-data\") pod \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " Oct 30 05:35:24.590904 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.590839 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-token\" (UniqueName: \"kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-token\") pod \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " Oct 30 05:35:24.591102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.590908 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-certs\") pod \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " Oct 30 05:35:24.591102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.591003 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-podinfo\") pod \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " Oct 30 05:35:24.591102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.591072 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-socket\") pod \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " Oct 30 05:35:24.591280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.591106 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istiod-ca-cert" pod "25d5a6a4-a651-41fa-8b85-02700b6e12e9" (UID: "25d5a6a4-a651-41fa-8b85-02700b6e12e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:24.591280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.591153 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-kube-api-access-87npp\" (UniqueName: \"kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-kube-api-access-87npp\") pod \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " Oct 30 05:35:24.591280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.591185 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-envoy\") pod \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\" (UID: \"25d5a6a4-a651-41fa-8b85-02700b6e12e9\") " Oct 30 05:35:24.591280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.591229 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-data" pod "25d5a6a4-a651-41fa-8b85-02700b6e12e9" (UID: "25d5a6a4-a651-41fa-8b85-02700b6e12e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.591280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.591249 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-credential-socket" pod "25d5a6a4-a651-41fa-8b85-02700b6e12e9" (UID: "25d5a6a4-a651-41fa-8b85-02700b6e12e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.591613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.591484 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-certs" pod "25d5a6a4-a651-41fa-8b85-02700b6e12e9" (UID: "25d5a6a4-a651-41fa-8b85-02700b6e12e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.591613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.591501 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-socket" pod "25d5a6a4-a651-41fa-8b85-02700b6e12e9" (UID: "25d5a6a4-a651-41fa-8b85-02700b6e12e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.592456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.592215 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.592456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.592241 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.592456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.592256 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160a7128-f92e-420b-bd00-c44142af02bc-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.592456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.592269 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.592456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.592285 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdwxz\" (UniqueName: \"kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-kube-api-access-vdwxz\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.592456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.592300 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.592456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.592314 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.592456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.592328 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbqd6\" (UniqueName: \"kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-kube-api-access-pbqd6\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.592456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.592344 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.592456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.592389 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160a7128-f92e-420b-bd00-c44142af02bc-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.592456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.592405 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffeb6a7-0efc-4336-89f8-2704a03d1398-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.592456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.592418 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.592456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.592433 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160a7128-f92e-420b-bd00-c44142af02bc-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.593640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.593573 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-token" pod "25d5a6a4-a651-41fa-8b85-02700b6e12e9" (UID: "25d5a6a4-a651-41fa-8b85-02700b6e12e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:24.594385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.594251 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-podinfo" pod "25d5a6a4-a651-41fa-8b85-02700b6e12e9" (UID: "25d5a6a4-a651-41fa-8b85-02700b6e12e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:35:24.594516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.594471 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-kube-api-access-87npp" pod "25d5a6a4-a651-41fa-8b85-02700b6e12e9" (UID: "25d5a6a4-a651-41fa-8b85-02700b6e12e9"). InnerVolumeSpecName "kube-api-access-87npp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:24.594745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.594721 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-envoy" pod "25d5a6a4-a651-41fa-8b85-02700b6e12e9" (UID: "25d5a6a4-a651-41fa-8b85-02700b6e12e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:24.650242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.649350 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn"] Oct 30 05:35:24.695997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.695891 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.695997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.695928 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87npp\" (UniqueName: \"kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-kube-api-access-87npp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.695997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.695944 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:24.695997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:24.695959 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d5a6a4-a651-41fa-8b85-02700b6e12e9-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:25.306815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.306436 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:35:25.306815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.306507 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" podUID="3ffeb6a7-0efc-4336-89f8-2704a03d1398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:35:25.501556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.501514 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" event={"ID":"3ffeb6a7-0efc-4336-89f8-2704a03d1398","Type":"ContainerDied","Data":"0314b35dae08d65edd72b8e527d10f2137929231fb0281bb08cad7317b4bad05"} Oct 30 05:35:25.501963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.501940 6587 scope.go:122] "RemoveContainer" containerID="c2de7d60740f9b53c4e3454bb42b8eee64c9c96fc5b36525432f42c7cd29cdac" Oct 30 05:35:25.502260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.501871 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz" Oct 30 05:35:25.548428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.548395 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" event={"ID":"160a7128-f92e-420b-bd00-c44142af02bc","Type":"ContainerDied","Data":"45aed2d972b4dcd12451d7644bdcf532dcfcfd2730277575f657294e37ff7b8e"} Oct 30 05:35:25.548730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.548710 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv" Oct 30 05:35:25.560592 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.560512 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" event={"ID":"25d5a6a4-a651-41fa-8b85-02700b6e12e9","Type":"ContainerDied","Data":"ea21a35e25a1c93ccce10dba1cfcf7cb47d7bcbec0ba79748c9a2a191baa40e0"} Oct 30 05:35:25.560715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.560640 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c" Oct 30 05:35:25.578323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.578292 6587 scope.go:122] "RemoveContainer" containerID="976450c7c8d070691b9e44338a8fd6c203deb82ef7d38565ae0356a1874679b7" Oct 30 05:35:25.591787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.591768 6587 scope.go:122] "RemoveContainer" containerID="24d3ca7d646667ab144d96fcc3f191e8cd7b9137917ef92c69627cd30ecde2ac" Oct 30 05:35:25.652467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.652426 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c"] Oct 30 05:35:25.657599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.657565 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c"] Oct 30 05:35:25.691206 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.691164 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz"] Oct 30 05:35:25.699201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.699169 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz"] Oct 30 05:35:25.742447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.742409 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv"] Oct 30 05:35:25.747111 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:25.747083 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv"] Oct 30 05:35:25.991633 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:35:25.991557 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:35:27.584569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:27.584534 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160a7128-f92e-420b-bd00-c44142af02bc" path="/var/lib/kubelet/pods/160a7128-f92e-420b-bd00-c44142af02bc/volumes" Oct 30 05:35:27.585185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:27.585168 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d5a6a4-a651-41fa-8b85-02700b6e12e9" path="/var/lib/kubelet/pods/25d5a6a4-a651-41fa-8b85-02700b6e12e9/volumes" Oct 30 05:35:27.585716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:27.585700 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ffeb6a7-0efc-4336-89f8-2704a03d1398" path="/var/lib/kubelet/pods/3ffeb6a7-0efc-4336-89f8-2704a03d1398/volumes" Oct 30 05:35:29.582026 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:29.581977 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" podUID="e87bea62-c7d6-4b93-8c28-d2a6a7ec0669" containerName="payload-processing" containerID="cri-o://caa86687cd6557f309cb5b4f613ed9ad64927db02a49daf4d3ff67bab39aeb5d" gracePeriod=25 Oct 30 05:35:29.650692 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:29.650636 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" podUID="1bb9624f-f9d9-4b85-89d6-87e470ff40a1" containerName="payload-pre-processing" containerID="cri-o://3ad58e74672e4c7a11179547eaf170b50cc04f1a045e38d9c911587a651b65a0" gracePeriod=25 Oct 30 05:35:29.990448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:29.990418 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" Oct 30 05:35:30.080153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.080124 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" Oct 30 05:35:30.173872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.173837 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-plugins-config-volume\") pod \"1bb9624f-f9d9-4b85-89d6-87e470ff40a1\" (UID: \"1bb9624f-f9d9-4b85-89d6-87e470ff40a1\") " Oct 30 05:35:30.174119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.173932 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-plugins-config-volume\") pod \"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669\" (UID: \"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669\") " Oct 30 05:35:30.174119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.173962 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-kube-api-access-pgszn\" (UniqueName: \"kubernetes.io/projected/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-kube-api-access-pgszn\") pod \"1bb9624f-f9d9-4b85-89d6-87e470ff40a1\" (UID: \"1bb9624f-f9d9-4b85-89d6-87e470ff40a1\") " Oct 30 05:35:30.174119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.174047 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-kube-api-access-7p58g\" (UniqueName: \"kubernetes.io/projected/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-kube-api-access-7p58g\") pod \"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669\" (UID: \"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669\") " Oct 30 05:35:30.174494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.174463 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-plugins-config-volume" pod "e87bea62-c7d6-4b93-8c28-d2a6a7ec0669" (UID: "e87bea62-c7d6-4b93-8c28-d2a6a7ec0669"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:30.174809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.174783 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:30.175017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.174987 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-plugins-config-volume" pod "1bb9624f-f9d9-4b85-89d6-87e470ff40a1" (UID: "1bb9624f-f9d9-4b85-89d6-87e470ff40a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:30.177169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.177144 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-kube-api-access-7p58g" pod "e87bea62-c7d6-4b93-8c28-d2a6a7ec0669" (UID: "e87bea62-c7d6-4b93-8c28-d2a6a7ec0669"). InnerVolumeSpecName "kube-api-access-7p58g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:30.177512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.177481 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-kube-api-access-pgszn" pod "1bb9624f-f9d9-4b85-89d6-87e470ff40a1" (UID: "1bb9624f-f9d9-4b85-89d6-87e470ff40a1"). InnerVolumeSpecName "kube-api-access-pgszn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:30.276565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.276482 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgszn\" (UniqueName: \"kubernetes.io/projected/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-kube-api-access-pgszn\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:30.276565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.276513 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p58g\" (UniqueName: \"kubernetes.io/projected/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669-kube-api-access-7p58g\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:30.276565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.276523 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb9624f-f9d9-4b85-89d6-87e470ff40a1-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:30.619951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.619910 6587 generic.go:356] "Generic (PLEG): container finished" podID="e87bea62-c7d6-4b93-8c28-d2a6a7ec0669" containerID="caa86687cd6557f309cb5b4f613ed9ad64927db02a49daf4d3ff67bab39aeb5d" exitCode=0 Oct 30 05:35:30.620941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.620468 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" event={"ID":"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669","Type":"ContainerDied","Data":"caa86687cd6557f309cb5b4f613ed9ad64927db02a49daf4d3ff67bab39aeb5d"} Oct 30 05:35:30.620941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.620492 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" Oct 30 05:35:30.620941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.620544 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8" event={"ID":"e87bea62-c7d6-4b93-8c28-d2a6a7ec0669","Type":"ContainerDied","Data":"92ce856f3afd4adca3a62ea72d8157df0ea2a987002dae80efbf47462d62896a"} Oct 30 05:35:30.620941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.620573 6587 scope.go:122] "RemoveContainer" containerID="caa86687cd6557f309cb5b4f613ed9ad64927db02a49daf4d3ff67bab39aeb5d" Oct 30 05:35:30.629545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.629520 6587 generic.go:356] "Generic (PLEG): container finished" podID="1bb9624f-f9d9-4b85-89d6-87e470ff40a1" containerID="3ad58e74672e4c7a11179547eaf170b50cc04f1a045e38d9c911587a651b65a0" exitCode=0 Oct 30 05:35:30.629659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.629559 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" event={"ID":"1bb9624f-f9d9-4b85-89d6-87e470ff40a1","Type":"ContainerDied","Data":"3ad58e74672e4c7a11179547eaf170b50cc04f1a045e38d9c911587a651b65a0"} Oct 30 05:35:30.629659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.629602 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" event={"ID":"1bb9624f-f9d9-4b85-89d6-87e470ff40a1","Type":"ContainerDied","Data":"8cbbe3d1a1095490a2347b8b8a858e1b70d9d2c67552d702522521f00c1414b9"} Oct 30 05:35:30.629802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.629675 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn" Oct 30 05:35:30.645698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.645671 6587 scope.go:122] "RemoveContainer" containerID="caa86687cd6557f309cb5b4f613ed9ad64927db02a49daf4d3ff67bab39aeb5d" Oct 30 05:35:30.646049 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:35:30.646022 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa86687cd6557f309cb5b4f613ed9ad64927db02a49daf4d3ff67bab39aeb5d\": container with ID starting with caa86687cd6557f309cb5b4f613ed9ad64927db02a49daf4d3ff67bab39aeb5d not found: ID does not exist" containerID="caa86687cd6557f309cb5b4f613ed9ad64927db02a49daf4d3ff67bab39aeb5d" Oct 30 05:35:30.646167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.646061 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa86687cd6557f309cb5b4f613ed9ad64927db02a49daf4d3ff67bab39aeb5d"} err="failed to get container status \"caa86687cd6557f309cb5b4f613ed9ad64927db02a49daf4d3ff67bab39aeb5d\": rpc error: code = NotFound desc = could not find container \"caa86687cd6557f309cb5b4f613ed9ad64927db02a49daf4d3ff67bab39aeb5d\": container with ID starting with caa86687cd6557f309cb5b4f613ed9ad64927db02a49daf4d3ff67bab39aeb5d not found: ID does not exist" Oct 30 05:35:30.646167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.646088 6587 scope.go:122] "RemoveContainer" containerID="3ad58e74672e4c7a11179547eaf170b50cc04f1a045e38d9c911587a651b65a0" Oct 30 05:35:30.661820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.661799 6587 scope.go:122] "RemoveContainer" containerID="3ad58e74672e4c7a11179547eaf170b50cc04f1a045e38d9c911587a651b65a0" Oct 30 05:35:30.662168 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:35:30.662149 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad58e74672e4c7a11179547eaf170b50cc04f1a045e38d9c911587a651b65a0\": container with ID starting with 3ad58e74672e4c7a11179547eaf170b50cc04f1a045e38d9c911587a651b65a0 not found: ID does not exist" containerID="3ad58e74672e4c7a11179547eaf170b50cc04f1a045e38d9c911587a651b65a0" Oct 30 05:35:30.662261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.662175 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad58e74672e4c7a11179547eaf170b50cc04f1a045e38d9c911587a651b65a0"} err="failed to get container status \"3ad58e74672e4c7a11179547eaf170b50cc04f1a045e38d9c911587a651b65a0\": rpc error: code = NotFound desc = could not find container \"3ad58e74672e4c7a11179547eaf170b50cc04f1a045e38d9c911587a651b65a0\": container with ID starting with 3ad58e74672e4c7a11179547eaf170b50cc04f1a045e38d9c911587a651b65a0 not found: ID does not exist" Oct 30 05:35:30.706629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.706595 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8"] Oct 30 05:35:30.716582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.716549 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8"] Oct 30 05:35:30.753437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.753405 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn"] Oct 30 05:35:30.769072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:30.769030 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn"] Oct 30 05:35:31.593339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:31.593309 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb9624f-f9d9-4b85-89d6-87e470ff40a1" path="/var/lib/kubelet/pods/1bb9624f-f9d9-4b85-89d6-87e470ff40a1/volumes" Oct 30 05:35:31.593824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:31.593807 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87bea62-c7d6-4b93-8c28-d2a6a7ec0669" path="/var/lib/kubelet/pods/e87bea62-c7d6-4b93-8c28-d2a6a7ec0669/volumes" Oct 30 05:35:33.364196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:33.364165 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" Oct 30 05:35:33.364744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:33.364244 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" Oct 30 05:35:33.673801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:33.673727 6587 generic.go:356] "Generic (PLEG): container finished" podID="f360336e-96e3-41c3-b483-aecf620a959f" containerID="fcbc9da06e47375d5ca044c36e00ab5d6c3306744c2444d01175dcd759dea8e9" exitCode=0 Oct 30 05:35:33.673954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:33.673795 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" event={"ID":"f360336e-96e3-41c3-b483-aecf620a959f","Type":"ContainerDied","Data":"fcbc9da06e47375d5ca044c36e00ab5d6c3306744c2444d01175dcd759dea8e9"} Oct 30 05:35:33.674192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:33.674177 6587 scope.go:122] "RemoveContainer" containerID="fcbc9da06e47375d5ca044c36e00ab5d6c3306744c2444d01175dcd759dea8e9" Oct 30 05:35:33.675857 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:33.675833 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:35:33.675935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:33.675896 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:35:33.675935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:33.675927 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:35:34.685955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:34.685919 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" event={"ID":"f360336e-96e3-41c3-b483-aecf620a959f","Type":"ContainerStarted","Data":"647fc8243b3bc774146f26476db37b07471a5420259af7c0066a45e7d503d6e8"} Oct 30 05:35:34.924382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:34.922154 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:35:34.945678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:34.945581 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body= Oct 30 05:35:34.945678 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:34.945644 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" Oct 30 05:35:35.926743 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:35.926707 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body= Oct 30 05:35:35.927337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:35.927302 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" Oct 30 05:35:36.928384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:36.925775 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body= Oct 30 05:35:36.928384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:36.925836 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" Oct 30 05:35:37.920383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:37.919683 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body= Oct 30 05:35:37.920383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:37.919765 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" Oct 30 05:35:38.921383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:38.919745 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body= Oct 30 05:35:38.921383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:38.919815 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" Oct 30 05:35:39.921233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:39.920148 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body= Oct 30 05:35:39.921233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:39.920211 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" Oct 30 05:35:40.925138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:40.925101 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:35:40.925711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:40.925679 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:35:40.926588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:40.926562 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:35:42.034691 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:42.034655 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s"] Oct 30 05:35:42.044672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:42.044641 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c3906b-kjb2s"] Oct 30 05:35:43.583369 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:43.583333 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c1ab68-48af-4566-a2b2-8556053d0a63" path="/var/lib/kubelet/pods/e4c1ab68-48af-4566-a2b2-8556053d0a63/volumes" Oct 30 05:35:43.593552 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:43.593528 6587 scope.go:122] "RemoveContainer" containerID="966ec984ddb09ae81ad4c142bdf84a13685efd00682a0d9ac20b43d4ec040e2b" Oct 30 05:35:43.605498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:43.605476 6587 scope.go:122] "RemoveContainer" containerID="e345a84edf7d30bac0d8dc113fad51286b4de0c2f71bf54757e525d1f49c9b23" Oct 30 05:35:47.168507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:47.168412 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc"] Oct 30 05:35:47.179141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:47.179113 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-83e83e-dgcgc"] Oct 30 05:35:47.585895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:47.585856 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a" path="/var/lib/kubelet/pods/3ca89ca8-f51a-4f71-ac27-eb7d714aaa1a/volumes" Oct 30 05:35:51.411703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:51.411664 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq"] Oct 30 05:35:51.412185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:51.411867 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" podUID="34622d6f-d009-43d4-affa-f16d4b51e5e6" containerName="payload-processing" containerID="cri-o://238cdf3903459c83cc6f781768b73e73b3bb5b2592f685f3199eabb2bd6e6a4b" gracePeriod=30 Oct 30 05:35:51.468594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:51.468559 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz"] Oct 30 05:35:51.468825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:51.468780 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" podUID="370172e2-ad90-418b-8068-83529060cff3" containerName="payload-pre-processing" containerID="cri-o://3b20d53248fb248a52bfac9de09cb6af938cc4c1a5a0f4753a63c11ac8f36272" gracePeriod=30 Oct 30 05:35:51.702119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:51.702084 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8"] Oct 30 05:35:51.709429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:51.709398 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-674082-c6dg8"] Oct 30 05:35:51.814331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:51.814305 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" Oct 30 05:35:51.915785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:51.914436 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34622d6f-d009-43d4-affa-f16d4b51e5e6-kube-api-access-f4wkt\" (UniqueName: \"kubernetes.io/projected/34622d6f-d009-43d4-affa-f16d4b51e5e6-kube-api-access-f4wkt\") pod \"34622d6f-d009-43d4-affa-f16d4b51e5e6\" (UID: \"34622d6f-d009-43d4-affa-f16d4b51e5e6\") " Oct 30 05:35:51.915785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:51.914514 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34622d6f-d009-43d4-affa-f16d4b51e5e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34622d6f-d009-43d4-affa-f16d4b51e5e6-plugins-config-volume\") pod \"34622d6f-d009-43d4-affa-f16d4b51e5e6\" (UID: \"34622d6f-d009-43d4-affa-f16d4b51e5e6\") " Oct 30 05:35:52.047490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.028689 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34622d6f-d009-43d4-affa-f16d4b51e5e6-plugins-config-volume" pod "34622d6f-d009-43d4-affa-f16d4b51e5e6" (UID: "34622d6f-d009-43d4-affa-f16d4b51e5e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:52.055277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.048789 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34622d6f-d009-43d4-affa-f16d4b51e5e6-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:52.068152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.057582 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34622d6f-d009-43d4-affa-f16d4b51e5e6-kube-api-access-f4wkt" pod "34622d6f-d009-43d4-affa-f16d4b51e5e6" (UID: "34622d6f-d009-43d4-affa-f16d4b51e5e6"). InnerVolumeSpecName "kube-api-access-f4wkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:52.093527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.093466 6587 generic.go:356] "Generic (PLEG): container finished" podID="370172e2-ad90-418b-8068-83529060cff3" containerID="3b20d53248fb248a52bfac9de09cb6af938cc4c1a5a0f4753a63c11ac8f36272" exitCode=0 Oct 30 05:35:52.094402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.094294 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" event={"ID":"370172e2-ad90-418b-8068-83529060cff3","Type":"ContainerDied","Data":"3b20d53248fb248a52bfac9de09cb6af938cc4c1a5a0f4753a63c11ac8f36272"} Oct 30 05:35:52.096760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.096706 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" event={"ID":"370172e2-ad90-418b-8068-83529060cff3","Type":"ContainerDied","Data":"f0dc335d1947f236fefc8badd488bcdf878e418b24ae960784cc75807af7f3de"} Oct 30 05:35:52.097007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.096968 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0dc335d1947f236fefc8badd488bcdf878e418b24ae960784cc75807af7f3de" Oct 30 05:35:52.100608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.100543 6587 generic.go:356] "Generic (PLEG): container finished" podID="34622d6f-d009-43d4-affa-f16d4b51e5e6" containerID="238cdf3903459c83cc6f781768b73e73b3bb5b2592f685f3199eabb2bd6e6a4b" exitCode=0 Oct 30 05:35:52.101885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.100905 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" event={"ID":"34622d6f-d009-43d4-affa-f16d4b51e5e6","Type":"ContainerDied","Data":"238cdf3903459c83cc6f781768b73e73b3bb5b2592f685f3199eabb2bd6e6a4b"} Oct 30 05:35:52.101885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.101680 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" event={"ID":"34622d6f-d009-43d4-affa-f16d4b51e5e6","Type":"ContainerDied","Data":"ae9a16f48cdbcc515e5207224e7d9a3354904efb8cc243d885bd37f46a271403"} Oct 30 05:35:52.101885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.101730 6587 scope.go:122] "RemoveContainer" containerID="238cdf3903459c83cc6f781768b73e73b3bb5b2592f685f3199eabb2bd6e6a4b" Oct 30 05:35:52.101885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.101014 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq" Oct 30 05:35:52.105018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.104994 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" Oct 30 05:35:52.127516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.127340 6587 scope.go:122] "RemoveContainer" containerID="238cdf3903459c83cc6f781768b73e73b3bb5b2592f685f3199eabb2bd6e6a4b" Oct 30 05:35:52.127744 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:35:52.127716 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238cdf3903459c83cc6f781768b73e73b3bb5b2592f685f3199eabb2bd6e6a4b\": container with ID starting with 238cdf3903459c83cc6f781768b73e73b3bb5b2592f685f3199eabb2bd6e6a4b not found: ID does not exist" containerID="238cdf3903459c83cc6f781768b73e73b3bb5b2592f685f3199eabb2bd6e6a4b" Oct 30 05:35:52.127848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.127756 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238cdf3903459c83cc6f781768b73e73b3bb5b2592f685f3199eabb2bd6e6a4b"} err="failed to get container status \"238cdf3903459c83cc6f781768b73e73b3bb5b2592f685f3199eabb2bd6e6a4b\": rpc error: code = NotFound desc = could not find container \"238cdf3903459c83cc6f781768b73e73b3bb5b2592f685f3199eabb2bd6e6a4b\": container with ID starting with 238cdf3903459c83cc6f781768b73e73b3bb5b2592f685f3199eabb2bd6e6a4b not found: ID does not exist" Oct 30 05:35:52.150416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.150384 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4wkt\" (UniqueName: \"kubernetes.io/projected/34622d6f-d009-43d4-affa-f16d4b51e5e6-kube-api-access-f4wkt\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:52.163926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.163901 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq"] Oct 30 05:35:52.168014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.167992 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq"] Oct 30 05:35:52.251134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.251106 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370172e2-ad90-418b-8068-83529060cff3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370172e2-ad90-418b-8068-83529060cff3-plugins-config-volume\") pod \"370172e2-ad90-418b-8068-83529060cff3\" (UID: \"370172e2-ad90-418b-8068-83529060cff3\") " Oct 30 05:35:52.251321 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.251240 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370172e2-ad90-418b-8068-83529060cff3-kube-api-access-q7nbr\" (UniqueName: \"kubernetes.io/projected/370172e2-ad90-418b-8068-83529060cff3-kube-api-access-q7nbr\") pod \"370172e2-ad90-418b-8068-83529060cff3\" (UID: \"370172e2-ad90-418b-8068-83529060cff3\") " Oct 30 05:35:52.251522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.251496 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370172e2-ad90-418b-8068-83529060cff3-plugins-config-volume" pod "370172e2-ad90-418b-8068-83529060cff3" (UID: "370172e2-ad90-418b-8068-83529060cff3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:52.252077 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.252042 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370172e2-ad90-418b-8068-83529060cff3-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:52.253527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.253504 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370172e2-ad90-418b-8068-83529060cff3-kube-api-access-q7nbr" pod "370172e2-ad90-418b-8068-83529060cff3" (UID: "370172e2-ad90-418b-8068-83529060cff3"). InnerVolumeSpecName "kube-api-access-q7nbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:52.353737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.353701 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7nbr\" (UniqueName: \"kubernetes.io/projected/370172e2-ad90-418b-8068-83529060cff3-kube-api-access-q7nbr\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:52.363742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.363704 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw"] Oct 30 05:35:52.368893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:52.368867 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d21ab-bffcw"] Oct 30 05:35:53.114466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:53.114431 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz" Oct 30 05:35:53.177676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:53.177636 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz"] Oct 30 05:35:53.193853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:53.193816 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz"] Oct 30 05:35:53.584279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:53.584241 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34622d6f-d009-43d4-affa-f16d4b51e5e6" path="/var/lib/kubelet/pods/34622d6f-d009-43d4-affa-f16d4b51e5e6/volumes" Oct 30 05:35:53.584836 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:53.584816 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370172e2-ad90-418b-8068-83529060cff3" path="/var/lib/kubelet/pods/370172e2-ad90-418b-8068-83529060cff3/volumes" Oct 30 05:35:53.585339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:53.585322 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7dfc5c3-dfbc-407e-889a-5f8800106be2" path="/var/lib/kubelet/pods/e7dfc5c3-dfbc-407e-889a-5f8800106be2/volumes" Oct 30 05:35:53.586101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:53.586082 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8c76eca-369d-4b82-8f99-a056a7441456" path="/var/lib/kubelet/pods/e8c76eca-369d-4b82-8f99-a056a7441456/volumes" Oct 30 05:35:53.876680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:53.876568 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz"] Oct 30 05:35:53.876882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:53.876846 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" podUID="fe7c6a91-845f-4333-83fa-2c23d0b94128" containerName="istio-proxy" containerID="cri-o://f3cfa2ac17ecb8b7d9489f0412304cf0ce43f8bd14def17c71ea991232b27c3a" gracePeriod=30 Oct 30 05:35:55.990841 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:35:55.990799 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.472717 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5"] Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.473959 6587 cpu_manager.go:414] "containerMap: removing container" podUID="25d5a6a4-a651-41fa-8b85-02700b6e12e9" containerName="istio-proxy" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.473983 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d5a6a4-a651-41fa-8b85-02700b6e12e9" containerName="istio-proxy" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474010 6587 cpu_manager.go:414] "containerMap: removing container" podUID="160a7128-f92e-420b-bd00-c44142af02bc" containerName="istio-proxy" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474017 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160a7128-f92e-420b-bd00-c44142af02bc" containerName="istio-proxy" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474034 6587 cpu_manager.go:414] "containerMap: removing container" podUID="34622d6f-d009-43d4-affa-f16d4b51e5e6" containerName="payload-processing" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474042 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34622d6f-d009-43d4-affa-f16d4b51e5e6" containerName="payload-processing" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474065 6587 cpu_manager.go:414] "containerMap: removing container" podUID="370172e2-ad90-418b-8068-83529060cff3" containerName="payload-pre-processing" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474072 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370172e2-ad90-418b-8068-83529060cff3" containerName="payload-pre-processing" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474102 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e87bea62-c7d6-4b93-8c28-d2a6a7ec0669" containerName="payload-processing" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474111 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87bea62-c7d6-4b93-8c28-d2a6a7ec0669" containerName="payload-processing" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474137 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3ffeb6a7-0efc-4336-89f8-2704a03d1398" containerName="istio-proxy" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474146 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ffeb6a7-0efc-4336-89f8-2704a03d1398" containerName="istio-proxy" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474178 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1bb9624f-f9d9-4b85-89d6-87e470ff40a1" containerName="payload-pre-processing" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474186 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb9624f-f9d9-4b85-89d6-87e470ff40a1" containerName="payload-pre-processing" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474406 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d5a6a4-a651-41fa-8b85-02700b6e12e9" containerName="istio-proxy" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474425 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="160a7128-f92e-420b-bd00-c44142af02bc" containerName="istio-proxy" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474436 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="34622d6f-d009-43d4-affa-f16d4b51e5e6" containerName="payload-processing" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474448 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="370172e2-ad90-418b-8068-83529060cff3" containerName="payload-pre-processing" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474462 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87bea62-c7d6-4b93-8c28-d2a6a7ec0669" containerName="payload-processing" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474472 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ffeb6a7-0efc-4336-89f8-2704a03d1398" containerName="istio-proxy" Oct 30 05:35:56.475380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.474485 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb9624f-f9d9-4b85-89d6-87e470ff40a1" containerName="payload-pre-processing" Oct 30 05:35:56.494801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.494762 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" Oct 30 05:35:56.517073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.517024 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4bc875-dockercfg-hwkj8\"" Oct 30 05:35:56.517452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.517430 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4bc875\"" Oct 30 05:35:56.557599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.557556 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5"] Oct 30 05:35:56.559661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.559632 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5\" (UID: \"84ba2105-85d3-4bd1-bbe6-07460d9ad35c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" Oct 30 05:35:56.559778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.559701 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xzfd\" (UniqueName: \"kubernetes.io/projected/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-kube-api-access-9xzfd\") pod \"payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5\" (UID: \"84ba2105-85d3-4bd1-bbe6-07460d9ad35c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" Oct 30 05:35:56.571299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.571267 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g"] Oct 30 05:35:56.581471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.578473 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g"] Oct 30 05:35:56.581471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.578579 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" Oct 30 05:35:56.661643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.661611 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6151042f-5d54-42f1-9e2c-83278374ee75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g\" (UID: \"6151042f-5d54-42f1-9e2c-83278374ee75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" Oct 30 05:35:56.661838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.661665 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5\" (UID: \"84ba2105-85d3-4bd1-bbe6-07460d9ad35c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" Oct 30 05:35:56.661838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.661778 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xzfd\" (UniqueName: \"kubernetes.io/projected/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-kube-api-access-9xzfd\") pod \"payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5\" (UID: \"84ba2105-85d3-4bd1-bbe6-07460d9ad35c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" Oct 30 05:35:56.662007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.661989 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jxh9\" (UniqueName: \"kubernetes.io/projected/6151042f-5d54-42f1-9e2c-83278374ee75-kube-api-access-5jxh9\") pod \"payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g\" (UID: \"6151042f-5d54-42f1-9e2c-83278374ee75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" Oct 30 05:35:56.662459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.662439 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5\" (UID: \"84ba2105-85d3-4bd1-bbe6-07460d9ad35c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" Oct 30 05:35:56.683726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.683700 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xzfd\" (UniqueName: \"kubernetes.io/projected/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-kube-api-access-9xzfd\") pod \"payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5\" (UID: \"84ba2105-85d3-4bd1-bbe6-07460d9ad35c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" Oct 30 05:35:56.764442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.764349 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6151042f-5d54-42f1-9e2c-83278374ee75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g\" (UID: \"6151042f-5d54-42f1-9e2c-83278374ee75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" Oct 30 05:35:56.764617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.764601 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jxh9\" (UniqueName: \"kubernetes.io/projected/6151042f-5d54-42f1-9e2c-83278374ee75-kube-api-access-5jxh9\") pod \"payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g\" (UID: \"6151042f-5d54-42f1-9e2c-83278374ee75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" Oct 30 05:35:56.765036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.765017 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6151042f-5d54-42f1-9e2c-83278374ee75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g\" (UID: \"6151042f-5d54-42f1-9e2c-83278374ee75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" Oct 30 05:35:56.785553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.785527 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jxh9\" (UniqueName: \"kubernetes.io/projected/6151042f-5d54-42f1-9e2c-83278374ee75-kube-api-access-5jxh9\") pod \"payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g\" (UID: \"6151042f-5d54-42f1-9e2c-83278374ee75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" Oct 30 05:35:56.888200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.888168 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" Oct 30 05:35:56.901906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:56.901879 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" Oct 30 05:35:57.209190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:57.208113 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5"] Oct 30 05:35:57.209190 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:35:57.208459 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ba2105_85d3_4bd1_bbe6_07460d9ad35c.slice/crio-32023d27966a4746aaa73bc62708a3c76cfd6f688860a6f5f2c5c0378b010654 WatchSource:0}: Error finding container 32023d27966a4746aaa73bc62708a3c76cfd6f688860a6f5f2c5c0378b010654: Status 404 returned error can't find the container with id 32023d27966a4746aaa73bc62708a3c76cfd6f688860a6f5f2c5c0378b010654 Oct 30 05:35:57.258963 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:35:57.258930 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6151042f_5d54_42f1_9e2c_83278374ee75.slice/crio-8cdad52c7d4aba715da9dd182e7a45dfe549f3968cfa723a86fe91644ee40589 WatchSource:0}: Error finding container 8cdad52c7d4aba715da9dd182e7a45dfe549f3968cfa723a86fe91644ee40589: Status 404 returned error can't find the container with id 8cdad52c7d4aba715da9dd182e7a45dfe549f3968cfa723a86fe91644ee40589 Oct 30 05:35:57.260162 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:57.259937 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g"] Oct 30 05:35:58.222332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:58.222289 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" event={"ID":"6151042f-5d54-42f1-9e2c-83278374ee75","Type":"ContainerStarted","Data":"978d4830c72bc84eafd062770c2af373aa1c84f0e7abc73b0028edbf3184983a"} Oct 30 05:35:58.222865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:58.222383 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" event={"ID":"6151042f-5d54-42f1-9e2c-83278374ee75","Type":"ContainerStarted","Data":"8cdad52c7d4aba715da9dd182e7a45dfe549f3968cfa723a86fe91644ee40589"} Oct 30 05:35:58.222865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:58.222419 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" Oct 30 05:35:58.224246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:58.224216 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" event={"ID":"84ba2105-85d3-4bd1-bbe6-07460d9ad35c","Type":"ContainerStarted","Data":"15cbc63f6830985416f8482bdaf4edcbd8dab9fa11e4931734e3540b9593d96f"} Oct 30 05:35:58.224431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:58.224260 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" event={"ID":"84ba2105-85d3-4bd1-bbe6-07460d9ad35c","Type":"ContainerStarted","Data":"32023d27966a4746aaa73bc62708a3c76cfd6f688860a6f5f2c5c0378b010654"} Oct 30 05:35:58.224431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:58.224325 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" Oct 30 05:35:58.270025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:58.269960 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" podStartSLOduration=2.269941962 podStartE2EDuration="2.269941962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:35:58.266854538 +0000 UTC m=+3082.794621327" watchObservedRunningTime="2026-10-30 05:35:58.269941962 +0000 UTC m=+3082.797708748" Oct 30 05:35:59.192544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.192521 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 05:35:59.217573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.217527 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe7c6a91-845f-4333-83fa-2c23d0b94128-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe7c6a91-845f-4333-83fa-2c23d0b94128-istiod-ca-cert\") pod \"fe7c6a91-845f-4333-83fa-2c23d0b94128\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " Oct 30 05:35:59.218259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.218152 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe7c6a91-845f-4333-83fa-2c23d0b94128-istiod-ca-cert" pod "fe7c6a91-845f-4333-83fa-2c23d0b94128" (UID: "fe7c6a91-845f-4333-83fa-2c23d0b94128"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:35:59.218435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.218387 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-kube-api-access-9z6sq\" (UniqueName: \"kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-kube-api-access-9z6sq\") pod \"fe7c6a91-845f-4333-83fa-2c23d0b94128\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " Oct 30 05:35:59.219764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.219000 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-data\") pod \"fe7c6a91-845f-4333-83fa-2c23d0b94128\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " Oct 30 05:35:59.219764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.219371 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-data" pod "fe7c6a91-845f-4333-83fa-2c23d0b94128" (UID: "fe7c6a91-845f-4333-83fa-2c23d0b94128"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:59.219764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.219429 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-podinfo\") pod \"fe7c6a91-845f-4333-83fa-2c23d0b94128\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " Oct 30 05:35:59.219764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.219498 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-token\" (UniqueName: \"kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-token\") pod \"fe7c6a91-845f-4333-83fa-2c23d0b94128\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " Oct 30 05:35:59.219764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.219545 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-envoy\") pod \"fe7c6a91-845f-4333-83fa-2c23d0b94128\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " Oct 30 05:35:59.242480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.242005 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-credential-socket\") pod \"fe7c6a91-845f-4333-83fa-2c23d0b94128\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " Oct 30 05:35:59.242480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.242215 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-certs\") pod \"fe7c6a91-845f-4333-83fa-2c23d0b94128\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " Oct 30 05:35:59.242480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.242279 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-socket\") pod \"fe7c6a91-845f-4333-83fa-2c23d0b94128\" (UID: \"fe7c6a91-845f-4333-83fa-2c23d0b94128\") " Oct 30 05:35:59.246720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.243648 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-credential-socket" pod "fe7c6a91-845f-4333-83fa-2c23d0b94128" (UID: "fe7c6a91-845f-4333-83fa-2c23d0b94128"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:59.246720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.243959 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-certs" pod "fe7c6a91-845f-4333-83fa-2c23d0b94128" (UID: "fe7c6a91-845f-4333-83fa-2c23d0b94128"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:59.246720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.245701 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-socket" pod "fe7c6a91-845f-4333-83fa-2c23d0b94128" (UID: "fe7c6a91-845f-4333-83fa-2c23d0b94128"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:59.246720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.246194 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:59.246720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.246222 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:59.246720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.246237 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:59.246720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.246251 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:59.246720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.246263 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe7c6a91-845f-4333-83fa-2c23d0b94128-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:59.280228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.250896 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-kube-api-access-9z6sq" pod "fe7c6a91-845f-4333-83fa-2c23d0b94128" (UID: "fe7c6a91-845f-4333-83fa-2c23d0b94128"). InnerVolumeSpecName "kube-api-access-9z6sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:59.280228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.251174 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-podinfo" pod "fe7c6a91-845f-4333-83fa-2c23d0b94128" (UID: "fe7c6a91-845f-4333-83fa-2c23d0b94128"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:35:59.294884 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.291542 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-token" pod "fe7c6a91-845f-4333-83fa-2c23d0b94128" (UID: "fe7c6a91-845f-4333-83fa-2c23d0b94128"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:35:59.298160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.298110 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-envoy" pod "fe7c6a91-845f-4333-83fa-2c23d0b94128" (UID: "fe7c6a91-845f-4333-83fa-2c23d0b94128"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:35:59.298858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.298827 6587 generic.go:356] "Generic (PLEG): container finished" podID="fe7c6a91-845f-4333-83fa-2c23d0b94128" containerID="f3cfa2ac17ecb8b7d9489f0412304cf0ce43f8bd14def17c71ea991232b27c3a" exitCode=0 Oct 30 05:35:59.298978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.298900 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" Oct 30 05:35:59.298978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.298934 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" event={"ID":"fe7c6a91-845f-4333-83fa-2c23d0b94128","Type":"ContainerDied","Data":"f3cfa2ac17ecb8b7d9489f0412304cf0ce43f8bd14def17c71ea991232b27c3a"} Oct 30 05:35:59.299068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.299011 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz" event={"ID":"fe7c6a91-845f-4333-83fa-2c23d0b94128","Type":"ContainerDied","Data":"09e6fd07d522acda4410332dd4a7036bb3dd0b1e577818e581431f72d2c03af5"} Oct 30 05:35:59.299068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.299043 6587 scope.go:122] "RemoveContainer" containerID="f3cfa2ac17ecb8b7d9489f0412304cf0ce43f8bd14def17c71ea991232b27c3a" Oct 30 05:35:59.314481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.313176 6587 scope.go:122] "RemoveContainer" containerID="f3cfa2ac17ecb8b7d9489f0412304cf0ce43f8bd14def17c71ea991232b27c3a" Oct 30 05:35:59.314481 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:35:59.313657 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cfa2ac17ecb8b7d9489f0412304cf0ce43f8bd14def17c71ea991232b27c3a\": container with ID starting with f3cfa2ac17ecb8b7d9489f0412304cf0ce43f8bd14def17c71ea991232b27c3a not found: ID does not exist" containerID="f3cfa2ac17ecb8b7d9489f0412304cf0ce43f8bd14def17c71ea991232b27c3a" Oct 30 05:35:59.314481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.313704 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cfa2ac17ecb8b7d9489f0412304cf0ce43f8bd14def17c71ea991232b27c3a"} err="failed to get container status \"f3cfa2ac17ecb8b7d9489f0412304cf0ce43f8bd14def17c71ea991232b27c3a\": rpc error: code = NotFound desc = could not find container \"f3cfa2ac17ecb8b7d9489f0412304cf0ce43f8bd14def17c71ea991232b27c3a\": container with ID starting with f3cfa2ac17ecb8b7d9489f0412304cf0ce43f8bd14def17c71ea991232b27c3a not found: ID does not exist" Oct 30 05:35:59.318233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.318158 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" podStartSLOduration=3.318142529 podStartE2EDuration="3.318142529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:35:58.323339582 +0000 UTC m=+3082.851106367" watchObservedRunningTime="2026-10-30 05:35:59.318142529 +0000 UTC m=+3083.845909316" Oct 30 05:35:59.348435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.348352 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z6sq\" (UniqueName: \"kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-kube-api-access-9z6sq\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:59.348435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.348399 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:59.348435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.348413 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:59.348435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.348428 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe7c6a91-845f-4333-83fa-2c23d0b94128-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:35:59.390979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.390952 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz"] Oct 30 05:35:59.418106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.418036 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz"] Oct 30 05:35:59.593192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.593153 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe7c6a91-845f-4333-83fa-2c23d0b94128" path="/var/lib/kubelet/pods/fe7c6a91-845f-4333-83fa-2c23d0b94128/volumes" Oct 30 05:35:59.711086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.710998 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr"] Oct 30 05:35:59.711308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.711280 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" containerID="cri-o://175e1dd6f95e7859066bdaa6ba59e6861cbe393b2efb0bc8534d533ff34d9b44" gracePeriod=30 Oct 30 05:35:59.726714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.726680 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w"] Oct 30 05:35:59.728545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:35:59.726958 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" podUID="58d2942b-f563-4e66-b8b5-f46845890275" containerName="istio-proxy" containerID="cri-o://b36ebdcb6a80f376458cf97bb50df624534720fd1c900b56f45bd352d6153d38" gracePeriod=30 Oct 30 05:36:00.393914 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.393883 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd"] Oct 30 05:36:00.394799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.394777 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fe7c6a91-845f-4333-83fa-2c23d0b94128" containerName="istio-proxy" Oct 30 05:36:00.394799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.394797 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe7c6a91-845f-4333-83fa-2c23d0b94128" containerName="istio-proxy" Oct 30 05:36:00.394998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.394892 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe7c6a91-845f-4333-83fa-2c23d0b94128" containerName="istio-proxy" Oct 30 05:36:00.400194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.400173 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" Oct 30 05:36:00.419682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.419647 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd"] Oct 30 05:36:00.496707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.496666 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd\" (UID: \"4c4d86c6-2546-4520-9550-63c07f6069e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" Oct 30 05:36:00.496707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.496709 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd\" (UID: \"4c4d86c6-2546-4520-9550-63c07f6069e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" Oct 30 05:36:00.496929 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.496912 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wllhj\" (UniqueName: \"kubernetes.io/projected/4c4d86c6-2546-4520-9550-63c07f6069e6-kube-api-access-wllhj\") pod \"maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd\" (UID: \"4c4d86c6-2546-4520-9550-63c07f6069e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" Oct 30 05:36:00.509893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.509868 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:36:00.510218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.510200 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:36:00.598784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.598748 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd\" (UID: \"4c4d86c6-2546-4520-9550-63c07f6069e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" Oct 30 05:36:00.598938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.598789 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd\" (UID: \"4c4d86c6-2546-4520-9550-63c07f6069e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" Oct 30 05:36:00.598938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.598922 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wllhj\" (UniqueName: \"kubernetes.io/projected/4c4d86c6-2546-4520-9550-63c07f6069e6-kube-api-access-wllhj\") pod \"maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd\" (UID: \"4c4d86c6-2546-4520-9550-63c07f6069e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" Oct 30 05:36:00.599452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.599433 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd\" (UID: \"4c4d86c6-2546-4520-9550-63c07f6069e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" Oct 30 05:36:00.601205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.601184 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd\" (UID: \"4c4d86c6-2546-4520-9550-63c07f6069e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" Oct 30 05:36:00.619799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.619771 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wllhj\" (UniqueName: \"kubernetes.io/projected/4c4d86c6-2546-4520-9550-63c07f6069e6-kube-api-access-wllhj\") pod \"maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd\" (UID: \"4c4d86c6-2546-4520-9550-63c07f6069e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" Oct 30 05:36:00.715536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.715439 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" Oct 30 05:36:00.942468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.942416 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd"] Oct 30 05:36:00.942721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.942664 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" containerID="cri-o://9eb9228aa36076f94b3a55f880c0f669fd0653015e8c74a55232fef3f8977eeb" gracePeriod=30 Oct 30 05:36:00.992801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.992712 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2"] Oct 30 05:36:00.994940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:00.993922 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" podUID="5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" containerName="istio-proxy" containerID="cri-o://4e8358ce19edfc71ed77677c149dda992ea9fe91ef2c2430199344c4077761e8" gracePeriod=30 Oct 30 05:36:01.221842 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:36:01.221795 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4d86c6_2546_4520_9550_63c07f6069e6.slice/crio-e636fbe787d39539901fc1011c0c53ee9080f989826420116c4a0e5196fa3d0f WatchSource:0}: Error finding container e636fbe787d39539901fc1011c0c53ee9080f989826420116c4a0e5196fa3d0f: Status 404 returned error can't find the container with id e636fbe787d39539901fc1011c0c53ee9080f989826420116c4a0e5196fa3d0f Oct 30 05:36:01.230571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.230543 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd"] Oct 30 05:36:01.319157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.319117 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" event={"ID":"4c4d86c6-2546-4520-9550-63c07f6069e6","Type":"ContainerStarted","Data":"e636fbe787d39539901fc1011c0c53ee9080f989826420116c4a0e5196fa3d0f"} Oct 30 05:36:01.583057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.583018 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w"] Oct 30 05:36:01.589751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.589728 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" Oct 30 05:36:01.603626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.603594 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w"] Oct 30 05:36:01.773910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.773865 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rhwb\" (UniqueName: \"kubernetes.io/projected/8657c2a6-caa4-4668-969e-1a6433fed6e1-kube-api-access-7rhwb\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w\" (UID: \"8657c2a6-caa4-4668-969e-1a6433fed6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" Oct 30 05:36:01.774121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.773923 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w\" (UID: \"8657c2a6-caa4-4668-969e-1a6433fed6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" Oct 30 05:36:01.774121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.774038 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w\" (UID: \"8657c2a6-caa4-4668-969e-1a6433fed6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" Oct 30 05:36:01.875982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.875880 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rhwb\" (UniqueName: \"kubernetes.io/projected/8657c2a6-caa4-4668-969e-1a6433fed6e1-kube-api-access-7rhwb\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w\" (UID: \"8657c2a6-caa4-4668-969e-1a6433fed6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" Oct 30 05:36:01.875982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.875949 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w\" (UID: \"8657c2a6-caa4-4668-969e-1a6433fed6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" Oct 30 05:36:01.876217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.876023 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w\" (UID: \"8657c2a6-caa4-4668-969e-1a6433fed6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" Oct 30 05:36:01.876871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.876844 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w\" (UID: \"8657c2a6-caa4-4668-969e-1a6433fed6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" Oct 30 05:36:01.882441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.879755 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w\" (UID: \"8657c2a6-caa4-4668-969e-1a6433fed6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" Oct 30 05:36:01.898199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.898174 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rhwb\" (UniqueName: \"kubernetes.io/projected/8657c2a6-caa4-4668-969e-1a6433fed6e1-kube-api-access-7rhwb\") pod \"maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w\" (UID: \"8657c2a6-caa4-4668-969e-1a6433fed6e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" Oct 30 05:36:01.911505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:01.911474 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" Oct 30 05:36:02.234406 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:36:02.233794 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8657c2a6_caa4_4668_969e_1a6433fed6e1.slice/crio-0504906fa323f760e1e5c8547765d5920011fe549e04d3a604d86665be35c8b4 WatchSource:0}: Error finding container 0504906fa323f760e1e5c8547765d5920011fe549e04d3a604d86665be35c8b4: Status 404 returned error can't find the container with id 0504906fa323f760e1e5c8547765d5920011fe549e04d3a604d86665be35c8b4 Oct 30 05:36:02.234406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:02.234235 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w"] Oct 30 05:36:02.326327 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:02.326290 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:36:02.326538 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:02.326402 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:36:02.333144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:02.333114 6587 generic.go:356] "Generic (PLEG): container finished" podID="4c4d86c6-2546-4520-9550-63c07f6069e6" containerID="b69cdc867d68eb979af67687f366ef701e5ed6d667e35fb44836173862e8acfc" exitCode=0 Oct 30 05:36:02.333270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:02.333191 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" event={"ID":"4c4d86c6-2546-4520-9550-63c07f6069e6","Type":"ContainerDied","Data":"b69cdc867d68eb979af67687f366ef701e5ed6d667e35fb44836173862e8acfc"} Oct 30 05:36:02.334841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:02.334812 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" event={"ID":"8657c2a6-caa4-4668-969e-1a6433fed6e1","Type":"ContainerStarted","Data":"0504906fa323f760e1e5c8547765d5920011fe549e04d3a604d86665be35c8b4"} Oct 30 05:36:03.348984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:03.348949 6587 generic.go:356] "Generic (PLEG): container finished" podID="8657c2a6-caa4-4668-969e-1a6433fed6e1" containerID="b876d3262a6ed5c3448c399adfaf5d59c49068025d0dc8453f01182e804030f8" exitCode=0 Oct 30 05:36:03.349500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:03.349043 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" event={"ID":"8657c2a6-caa4-4668-969e-1a6433fed6e1","Type":"ContainerDied","Data":"b876d3262a6ed5c3448c399adfaf5d59c49068025d0dc8453f01182e804030f8"} Oct 30 05:36:03.519686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:03.519613 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" Oct 30 05:36:03.599541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:03.599499 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-xks-ca\") pod \"4c4d86c6-2546-4520-9550-63c07f6069e6\" (UID: \"4c4d86c6-2546-4520-9550-63c07f6069e6\") " Oct 30 05:36:03.599714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:03.599556 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-openshift-ca\") pod \"4c4d86c6-2546-4520-9550-63c07f6069e6\" (UID: \"4c4d86c6-2546-4520-9550-63c07f6069e6\") " Oct 30 05:36:03.599809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:03.599750 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4d86c6-2546-4520-9550-63c07f6069e6-kube-api-access-wllhj\" (UniqueName: \"kubernetes.io/projected/4c4d86c6-2546-4520-9550-63c07f6069e6-kube-api-access-wllhj\") pod \"4c4d86c6-2546-4520-9550-63c07f6069e6\" (UID: \"4c4d86c6-2546-4520-9550-63c07f6069e6\") " Oct 30 05:36:03.601869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:03.601835 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-openshift-ca" pod "4c4d86c6-2546-4520-9550-63c07f6069e6" (UID: "4c4d86c6-2546-4520-9550-63c07f6069e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:03.602466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:03.602434 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-xks-ca" pod "4c4d86c6-2546-4520-9550-63c07f6069e6" (UID: "4c4d86c6-2546-4520-9550-63c07f6069e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:03.602587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:03.602517 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4d86c6-2546-4520-9550-63c07f6069e6-kube-api-access-wllhj" pod "4c4d86c6-2546-4520-9550-63c07f6069e6" (UID: "4c4d86c6-2546-4520-9550-63c07f6069e6"). InnerVolumeSpecName "kube-api-access-wllhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:03.702796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:03.702756 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:03.702796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:03.702789 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4d86c6-2546-4520-9550-63c07f6069e6-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:03.702796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:03.702803 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wllhj\" (UniqueName: \"kubernetes.io/projected/4c4d86c6-2546-4520-9550-63c07f6069e6-kube-api-access-wllhj\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:04.142775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.142742 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t"] Oct 30 05:36:04.143588 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.143571 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4c4d86c6-2546-4520-9550-63c07f6069e6" containerName="revoke-keys" Oct 30 05:36:04.143648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.143589 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4d86c6-2546-4520-9550-63c07f6069e6" containerName="revoke-keys" Oct 30 05:36:04.143729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.143720 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4d86c6-2546-4520-9550-63c07f6069e6" containerName="revoke-keys" Oct 30 05:36:04.147766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.147737 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.169711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.169684 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7a4600-serving-cert\"" Oct 30 05:36:04.176964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.176933 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t"] Oct 30 05:36:04.209732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.209697 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6jwr\" (UniqueName: \"kubernetes.io/projected/34b792b9-3587-447d-8e6a-a3d21bc4131b-kube-api-access-f6jwr\") pod \"maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.209929 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.209914 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b792b9-3587-447d-8e6a-a3d21bc4131b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.209985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.209971 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.210037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.210022 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-metrics-certs\") pod \"maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.311733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.311701 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b792b9-3587-447d-8e6a-a3d21bc4131b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.311919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.311758 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.311919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.311786 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-metrics-certs\") pod \"maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.311919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.311898 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6jwr\" (UniqueName: \"kubernetes.io/projected/34b792b9-3587-447d-8e6a-a3d21bc4131b-kube-api-access-f6jwr\") pod \"maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.312492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.312465 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b792b9-3587-447d-8e6a-a3d21bc4131b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.314537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.314511 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.314762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.314743 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-metrics-certs\") pod \"maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.334109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.334078 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6jwr\" (UniqueName: \"kubernetes.io/projected/34b792b9-3587-447d-8e6a-a3d21bc4131b-kube-api-access-f6jwr\") pod \"maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.362835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.362799 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" Oct 30 05:36:04.363240 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.362833 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd" event={"ID":"4c4d86c6-2546-4520-9550-63c07f6069e6","Type":"ContainerDied","Data":"e636fbe787d39539901fc1011c0c53ee9080f989826420116c4a0e5196fa3d0f"} Oct 30 05:36:04.363240 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.362893 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e636fbe787d39539901fc1011c0c53ee9080f989826420116c4a0e5196fa3d0f" Oct 30 05:36:04.475726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.475621 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:04.622526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.622489 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" Oct 30 05:36:04.732930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.732901 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-xks-ca\") pod \"8657c2a6-caa4-4668-969e-1a6433fed6e1\" (UID: \"8657c2a6-caa4-4668-969e-1a6433fed6e1\") " Oct 30 05:36:04.733176 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.733158 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8657c2a6-caa4-4668-969e-1a6433fed6e1-kube-api-access-7rhwb\" (UniqueName: \"kubernetes.io/projected/8657c2a6-caa4-4668-969e-1a6433fed6e1-kube-api-access-7rhwb\") pod \"8657c2a6-caa4-4668-969e-1a6433fed6e1\" (UID: \"8657c2a6-caa4-4668-969e-1a6433fed6e1\") " Oct 30 05:36:04.733308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.733287 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-openshift-ca\") pod \"8657c2a6-caa4-4668-969e-1a6433fed6e1\" (UID: \"8657c2a6-caa4-4668-969e-1a6433fed6e1\") " Oct 30 05:36:04.733740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.733711 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-openshift-ca" pod "8657c2a6-caa4-4668-969e-1a6433fed6e1" (UID: "8657c2a6-caa4-4668-969e-1a6433fed6e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:04.734515 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.734490 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:04.736210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.736187 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-xks-ca" pod "8657c2a6-caa4-4668-969e-1a6433fed6e1" (UID: "8657c2a6-caa4-4668-969e-1a6433fed6e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:04.737543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.737517 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8657c2a6-caa4-4668-969e-1a6433fed6e1-kube-api-access-7rhwb" pod "8657c2a6-caa4-4668-969e-1a6433fed6e1" (UID: "8657c2a6-caa4-4668-969e-1a6433fed6e1"). InnerVolumeSpecName "kube-api-access-7rhwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:04.835948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.835919 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8657c2a6-caa4-4668-969e-1a6433fed6e1-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:04.835948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.835950 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rhwb\" (UniqueName: \"kubernetes.io/projected/8657c2a6-caa4-4668-969e-1a6433fed6e1-kube-api-access-7rhwb\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:04.840898 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:04.840866 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t"] Oct 30 05:36:04.840898 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:36:04.840887 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b792b9_3587_447d_8e6a_a3d21bc4131b.slice/crio-0e8ecd218fa787d4579a3c56ed67b0c500467890a7c93dd0a7243710b4baf47d WatchSource:0}: Error finding container 0e8ecd218fa787d4579a3c56ed67b0c500467890a7c93dd0a7243710b4baf47d: Status 404 returned error can't find the container with id 0e8ecd218fa787d4579a3c56ed67b0c500467890a7c93dd0a7243710b4baf47d Oct 30 05:36:05.022006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.021972 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:36:05.062636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.062602 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:36:05.147651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.147411 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-certs\") pod \"58d2942b-f563-4e66-b8b5-f46845890275\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " Oct 30 05:36:05.147651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.147476 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-token\" (UniqueName: \"kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-token\") pod \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " Oct 30 05:36:05.147651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.147541 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-data\") pod \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " Oct 30 05:36:05.149203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.148881 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-kube-api-access-xsdhx\" (UniqueName: \"kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-kube-api-access-xsdhx\") pod \"58d2942b-f563-4e66-b8b5-f46845890275\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " Oct 30 05:36:05.149203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.148980 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-certs\") pod \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " Oct 30 05:36:05.149203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.149027 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-socket\") pod \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " Oct 30 05:36:05.149203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.149060 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istiod-ca-cert\") pod \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " Oct 30 05:36:05.149203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.149095 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-istio-token\" (UniqueName: \"kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-istio-token\") pod \"58d2942b-f563-4e66-b8b5-f46845890275\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " Oct 30 05:36:05.149203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.149149 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-kube-api-access-sgnxh\" (UniqueName: \"kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-kube-api-access-sgnxh\") pod \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " Oct 30 05:36:05.154562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.154446 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-kube-api-access-sgnxh" pod "cfc65f69-cd06-4575-9e52-29f042c2bdc7" (UID: "cfc65f69-cd06-4575-9e52-29f042c2bdc7"). InnerVolumeSpecName "kube-api-access-sgnxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:05.157886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.154762 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-envoy\") pod \"58d2942b-f563-4e66-b8b5-f46845890275\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " Oct 30 05:36:05.157886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.154815 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58d2942b-f563-4e66-b8b5-f46845890275-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2942b-f563-4e66-b8b5-f46845890275-istio-podinfo\") pod \"58d2942b-f563-4e66-b8b5-f46845890275\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " Oct 30 05:36:05.157886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.155229 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-certs" pod "58d2942b-f563-4e66-b8b5-f46845890275" (UID: "58d2942b-f563-4e66-b8b5-f46845890275"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.158342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.158282 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-credential-socket\") pod \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " Oct 30 05:36:05.158615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.158564 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-podinfo\") pod \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " Oct 30 05:36:05.159023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.158916 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-credential-socket\") pod \"58d2942b-f563-4e66-b8b5-f46845890275\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " Oct 30 05:36:05.159023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.158985 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-socket\") pod \"58d2942b-f563-4e66-b8b5-f46845890275\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " Oct 30 05:36:05.159258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.159223 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d2942b-f563-4e66-b8b5-f46845890275-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2942b-f563-4e66-b8b5-f46845890275-istiod-ca-cert\") pod \"58d2942b-f563-4e66-b8b5-f46845890275\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " Oct 30 05:36:05.170865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.159512 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-envoy\") pod \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\" (UID: \"cfc65f69-cd06-4575-9e52-29f042c2bdc7\") " Oct 30 05:36:05.170865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.159555 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-data\") pod \"58d2942b-f563-4e66-b8b5-f46845890275\" (UID: \"58d2942b-f563-4e66-b8b5-f46845890275\") " Oct 30 05:36:05.172332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.172029 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.172332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.172068 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgnxh\" (UniqueName: \"kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-kube-api-access-sgnxh\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.172691 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.172657 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-data" pod "58d2942b-f563-4e66-b8b5-f46845890275" (UID: "58d2942b-f563-4e66-b8b5-f46845890275"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.177425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.173133 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istiod-ca-cert" pod "cfc65f69-cd06-4575-9e52-29f042c2bdc7" (UID: "cfc65f69-cd06-4575-9e52-29f042c2bdc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:05.177425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.173431 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-certs" pod "cfc65f69-cd06-4575-9e52-29f042c2bdc7" (UID: "cfc65f69-cd06-4575-9e52-29f042c2bdc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.177425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.174819 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-socket" pod "cfc65f69-cd06-4575-9e52-29f042c2bdc7" (UID: "cfc65f69-cd06-4575-9e52-29f042c2bdc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.177425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.175891 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-data" pod "cfc65f69-cd06-4575-9e52-29f042c2bdc7" (UID: "cfc65f69-cd06-4575-9e52-29f042c2bdc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.177425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.175908 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-envoy" pod "58d2942b-f563-4e66-b8b5-f46845890275" (UID: "58d2942b-f563-4e66-b8b5-f46845890275"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.178437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.178305 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-istio-token" pod "58d2942b-f563-4e66-b8b5-f46845890275" (UID: "58d2942b-f563-4e66-b8b5-f46845890275"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:05.178597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.178575 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-credential-socket" pod "58d2942b-f563-4e66-b8b5-f46845890275" (UID: "58d2942b-f563-4e66-b8b5-f46845890275"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.179101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.178770 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-credential-socket" pod "cfc65f69-cd06-4575-9e52-29f042c2bdc7" (UID: "cfc65f69-cd06-4575-9e52-29f042c2bdc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.180372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.180330 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-socket" pod "58d2942b-f563-4e66-b8b5-f46845890275" (UID: "58d2942b-f563-4e66-b8b5-f46845890275"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.181935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.181534 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-kube-api-access-xsdhx" pod "58d2942b-f563-4e66-b8b5-f46845890275" (UID: "58d2942b-f563-4e66-b8b5-f46845890275"). InnerVolumeSpecName "kube-api-access-xsdhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:05.182587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.182561 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-token" pod "cfc65f69-cd06-4575-9e52-29f042c2bdc7" (UID: "cfc65f69-cd06-4575-9e52-29f042c2bdc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:05.182762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.182744 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-podinfo" pod "cfc65f69-cd06-4575-9e52-29f042c2bdc7" (UID: "cfc65f69-cd06-4575-9e52-29f042c2bdc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:05.182762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.182750 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-envoy" pod "cfc65f69-cd06-4575-9e52-29f042c2bdc7" (UID: "cfc65f69-cd06-4575-9e52-29f042c2bdc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:05.183146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.183130 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58d2942b-f563-4e66-b8b5-f46845890275-istio-podinfo" pod "58d2942b-f563-4e66-b8b5-f46845890275" (UID: "58d2942b-f563-4e66-b8b5-f46845890275"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:05.183634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.183588 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d2942b-f563-4e66-b8b5-f46845890275-istiod-ca-cert" pod "58d2942b-f563-4e66-b8b5-f46845890275" (UID: "58d2942b-f563-4e66-b8b5-f46845890275"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:05.212807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.212769 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz"] Oct 30 05:36:05.273798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273768 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.273906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273803 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2942b-f563-4e66-b8b5-f46845890275-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.273906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273818 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.273906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273831 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.273906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273846 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.273906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273861 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.273906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273874 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2942b-f563-4e66-b8b5-f46845890275-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.273906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273887 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.273906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273899 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2942b-f563-4e66-b8b5-f46845890275-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.274180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273911 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.274180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273922 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.274180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273935 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsdhx\" (UniqueName: \"kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-kube-api-access-xsdhx\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.274180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273948 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.274180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273962 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc65f69-cd06-4575-9e52-29f042c2bdc7-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.274180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273976 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc65f69-cd06-4575-9e52-29f042c2bdc7-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.274180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.273988 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2942b-f563-4e66-b8b5-f46845890275-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:05.374783 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.374744 6587 generic.go:356] "Generic (PLEG): container finished" podID="58d2942b-f563-4e66-b8b5-f46845890275" containerID="b36ebdcb6a80f376458cf97bb50df624534720fd1c900b56f45bd352d6153d38" exitCode=0 Oct 30 05:36:05.375238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.375008 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" event={"ID":"58d2942b-f563-4e66-b8b5-f46845890275","Type":"ContainerDied","Data":"b36ebdcb6a80f376458cf97bb50df624534720fd1c900b56f45bd352d6153d38"} Oct 30 05:36:05.375238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.375069 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" event={"ID":"58d2942b-f563-4e66-b8b5-f46845890275","Type":"ContainerDied","Data":"467601ced7aa8fa9424732069291c05e4ad90f44a13278cd224fc0175715afca"} Oct 30 05:36:05.375238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.375097 6587 scope.go:122] "RemoveContainer" containerID="b36ebdcb6a80f376458cf97bb50df624534720fd1c900b56f45bd352d6153d38" Oct 30 05:36:05.375402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.375263 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w" Oct 30 05:36:05.377448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.377285 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" Oct 30 05:36:05.377448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.377328 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w" event={"ID":"8657c2a6-caa4-4668-969e-1a6433fed6e1","Type":"ContainerDied","Data":"0504906fa323f760e1e5c8547765d5920011fe549e04d3a604d86665be35c8b4"} Oct 30 05:36:05.377448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.377418 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0504906fa323f760e1e5c8547765d5920011fe549e04d3a604d86665be35c8b4" Oct 30 05:36:05.379425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.379400 6587 generic.go:356] "Generic (PLEG): container finished" podID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerID="175e1dd6f95e7859066bdaa6ba59e6861cbe393b2efb0bc8534d533ff34d9b44" exitCode=0 Oct 30 05:36:05.379551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.379457 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" Oct 30 05:36:05.379551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.379483 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" event={"ID":"cfc65f69-cd06-4575-9e52-29f042c2bdc7","Type":"ContainerDied","Data":"175e1dd6f95e7859066bdaa6ba59e6861cbe393b2efb0bc8534d533ff34d9b44"} Oct 30 05:36:05.379551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.379546 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr" event={"ID":"cfc65f69-cd06-4575-9e52-29f042c2bdc7","Type":"ContainerDied","Data":"e856e9426e0f6b5fc2405d4df27eca5a4fe59c71417d9f20a3fbe30e0ae11a5c"} Oct 30 05:36:05.381680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.381654 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" event={"ID":"34b792b9-3587-447d-8e6a-a3d21bc4131b","Type":"ContainerStarted","Data":"b57e8bcaee26c7336f41e695739e97af1f202a0809ba42a0c9a1a7a2979fee2e"} Oct 30 05:36:05.381798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.381699 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" event={"ID":"34b792b9-3587-447d-8e6a-a3d21bc4131b","Type":"ContainerStarted","Data":"0e8ecd218fa787d4579a3c56ed67b0c500467890a7c93dd0a7243710b4baf47d"} Oct 30 05:36:05.382106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.382085 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:05.397153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.397058 6587 scope.go:122] "RemoveContainer" containerID="b36ebdcb6a80f376458cf97bb50df624534720fd1c900b56f45bd352d6153d38" Oct 30 05:36:05.398756 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:05.397864 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36ebdcb6a80f376458cf97bb50df624534720fd1c900b56f45bd352d6153d38\": container with ID starting with b36ebdcb6a80f376458cf97bb50df624534720fd1c900b56f45bd352d6153d38 not found: ID does not exist" containerID="b36ebdcb6a80f376458cf97bb50df624534720fd1c900b56f45bd352d6153d38" Oct 30 05:36:05.398756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.397900 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36ebdcb6a80f376458cf97bb50df624534720fd1c900b56f45bd352d6153d38"} err="failed to get container status \"b36ebdcb6a80f376458cf97bb50df624534720fd1c900b56f45bd352d6153d38\": rpc error: code = NotFound desc = could not find container \"b36ebdcb6a80f376458cf97bb50df624534720fd1c900b56f45bd352d6153d38\": container with ID starting with b36ebdcb6a80f376458cf97bb50df624534720fd1c900b56f45bd352d6153d38 not found: ID does not exist" Oct 30 05:36:05.398756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.397921 6587 scope.go:122] "RemoveContainer" containerID="175e1dd6f95e7859066bdaa6ba59e6861cbe393b2efb0bc8534d533ff34d9b44" Oct 30 05:36:05.411042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.411023 6587 scope.go:122] "RemoveContainer" containerID="052301a58b96cf1fd15e0abddb5a44657e019d60019881734c819567f07d7bd6" Oct 30 05:36:05.425273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.425240 6587 scope.go:122] "RemoveContainer" containerID="175e1dd6f95e7859066bdaa6ba59e6861cbe393b2efb0bc8534d533ff34d9b44" Oct 30 05:36:05.425647 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:05.425615 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175e1dd6f95e7859066bdaa6ba59e6861cbe393b2efb0bc8534d533ff34d9b44\": container with ID starting with 175e1dd6f95e7859066bdaa6ba59e6861cbe393b2efb0bc8534d533ff34d9b44 not found: ID does not exist" containerID="175e1dd6f95e7859066bdaa6ba59e6861cbe393b2efb0bc8534d533ff34d9b44" Oct 30 05:36:05.425741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.425658 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175e1dd6f95e7859066bdaa6ba59e6861cbe393b2efb0bc8534d533ff34d9b44"} err="failed to get container status \"175e1dd6f95e7859066bdaa6ba59e6861cbe393b2efb0bc8534d533ff34d9b44\": rpc error: code = NotFound desc = could not find container \"175e1dd6f95e7859066bdaa6ba59e6861cbe393b2efb0bc8534d533ff34d9b44\": container with ID starting with 175e1dd6f95e7859066bdaa6ba59e6861cbe393b2efb0bc8534d533ff34d9b44 not found: ID does not exist" Oct 30 05:36:05.425741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.425681 6587 scope.go:122] "RemoveContainer" containerID="052301a58b96cf1fd15e0abddb5a44657e019d60019881734c819567f07d7bd6" Oct 30 05:36:05.425984 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:05.425962 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052301a58b96cf1fd15e0abddb5a44657e019d60019881734c819567f07d7bd6\": container with ID starting with 052301a58b96cf1fd15e0abddb5a44657e019d60019881734c819567f07d7bd6 not found: ID does not exist" containerID="052301a58b96cf1fd15e0abddb5a44657e019d60019881734c819567f07d7bd6" Oct 30 05:36:05.426064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.425994 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052301a58b96cf1fd15e0abddb5a44657e019d60019881734c819567f07d7bd6"} err="failed to get container status \"052301a58b96cf1fd15e0abddb5a44657e019d60019881734c819567f07d7bd6\": rpc error: code = NotFound desc = could not find container \"052301a58b96cf1fd15e0abddb5a44657e019d60019881734c819567f07d7bd6\": container with ID starting with 052301a58b96cf1fd15e0abddb5a44657e019d60019881734c819567f07d7bd6 not found: ID does not exist" Oct 30 05:36:05.447625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.447595 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w"] Oct 30 05:36:05.460196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.460163 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w"] Oct 30 05:36:05.513963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.513855 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" podStartSLOduration=1.118343018 podStartE2EDuration="1.513840456s" totalImagesPullingTime="395.497438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:05.509514247 +0000 UTC m=+3090.037281037" watchObservedRunningTime="2026-10-30 05:36:05.513840456 +0000 UTC m=+3090.041607243" Oct 30 05:36:05.587026 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.586992 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d2942b-f563-4e66-b8b5-f46845890275" path="/var/lib/kubelet/pods/58d2942b-f563-4e66-b8b5-f46845890275/volumes" Oct 30 05:36:05.591004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.590974 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr"] Oct 30 05:36:05.595653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:05.595630 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr"] Oct 30 05:36:06.311757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.311729 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:36:06.384037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.384007 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:36:06.395743 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.395715 6587 generic.go:356] "Generic (PLEG): container finished" podID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerID="9eb9228aa36076f94b3a55f880c0f669fd0653015e8c74a55232fef3f8977eeb" exitCode=0 Oct 30 05:36:06.395981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.395960 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" event={"ID":"a9342248-e93d-4bb7-8f9e-1a20117d05be","Type":"ContainerDied","Data":"9eb9228aa36076f94b3a55f880c0f669fd0653015e8c74a55232fef3f8977eeb"} Oct 30 05:36:06.396068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.396015 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" event={"ID":"a9342248-e93d-4bb7-8f9e-1a20117d05be","Type":"ContainerDied","Data":"fa790f818ed13046496bb1a81416bce69f370828eee240b4c9943133181ff3da"} Oct 30 05:36:06.396068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.396048 6587 scope.go:122] "RemoveContainer" containerID="9eb9228aa36076f94b3a55f880c0f669fd0653015e8c74a55232fef3f8977eeb" Oct 30 05:36:06.396205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.396163 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd" Oct 30 05:36:06.400117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.400094 6587 generic.go:356] "Generic (PLEG): container finished" podID="5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" containerID="4e8358ce19edfc71ed77677c149dda992ea9fe91ef2c2430199344c4077761e8" exitCode=0 Oct 30 05:36:06.401079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.401053 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" Oct 30 05:36:06.401213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.401194 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" event={"ID":"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee","Type":"ContainerDied","Data":"4e8358ce19edfc71ed77677c149dda992ea9fe91ef2c2430199344c4077761e8"} Oct 30 05:36:06.401300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.401237 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2" event={"ID":"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee","Type":"ContainerDied","Data":"c42efb55e6e7251d26e88229459ccd32ad84bd1e5fcada47f0dbd730506627fa"} Oct 30 05:36:06.401679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.401659 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-data\") pod \"a9342248-e93d-4bb7-8f9e-1a20117d05be\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " Oct 30 05:36:06.401785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.401752 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-socket\") pod \"a9342248-e93d-4bb7-8f9e-1a20117d05be\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " Oct 30 05:36:06.401852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.401834 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-token\" (UniqueName: \"kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-token\") pod \"a9342248-e93d-4bb7-8f9e-1a20117d05be\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " Oct 30 05:36:06.401905 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.401859 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-kube-api-access-txxss\" (UniqueName: \"kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-kube-api-access-txxss\") pod \"a9342248-e93d-4bb7-8f9e-1a20117d05be\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " Oct 30 05:36:06.401942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.401903 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-credential-socket\") pod \"a9342248-e93d-4bb7-8f9e-1a20117d05be\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " Oct 30 05:36:06.401942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.401932 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-certs\") pod \"a9342248-e93d-4bb7-8f9e-1a20117d05be\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " Oct 30 05:36:06.402041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.401970 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-envoy\") pod \"a9342248-e93d-4bb7-8f9e-1a20117d05be\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " Oct 30 05:36:06.402041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.402000 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9342248-e93d-4bb7-8f9e-1a20117d05be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9342248-e93d-4bb7-8f9e-1a20117d05be-istiod-ca-cert\") pod \"a9342248-e93d-4bb7-8f9e-1a20117d05be\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " Oct 30 05:36:06.402149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.402102 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-podinfo\") pod \"a9342248-e93d-4bb7-8f9e-1a20117d05be\" (UID: \"a9342248-e93d-4bb7-8f9e-1a20117d05be\") " Oct 30 05:36:06.405226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.405201 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-podinfo" pod "a9342248-e93d-4bb7-8f9e-1a20117d05be" (UID: "a9342248-e93d-4bb7-8f9e-1a20117d05be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:06.405455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.405438 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-credential-socket" pod "a9342248-e93d-4bb7-8f9e-1a20117d05be" (UID: "a9342248-e93d-4bb7-8f9e-1a20117d05be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.405597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.405582 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-certs" pod "a9342248-e93d-4bb7-8f9e-1a20117d05be" (UID: "a9342248-e93d-4bb7-8f9e-1a20117d05be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.405741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.405726 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-kube-api-access-txxss" pod "a9342248-e93d-4bb7-8f9e-1a20117d05be" (UID: "a9342248-e93d-4bb7-8f9e-1a20117d05be"). InnerVolumeSpecName "kube-api-access-txxss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:06.406312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.406290 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-data" pod "a9342248-e93d-4bb7-8f9e-1a20117d05be" (UID: "a9342248-e93d-4bb7-8f9e-1a20117d05be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.406519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.406496 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-socket" pod "a9342248-e93d-4bb7-8f9e-1a20117d05be" (UID: "a9342248-e93d-4bb7-8f9e-1a20117d05be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.408495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.408472 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-token" pod "a9342248-e93d-4bb7-8f9e-1a20117d05be" (UID: "a9342248-e93d-4bb7-8f9e-1a20117d05be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:06.408909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.408880 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-envoy" pod "a9342248-e93d-4bb7-8f9e-1a20117d05be" (UID: "a9342248-e93d-4bb7-8f9e-1a20117d05be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.409012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.408920 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9342248-e93d-4bb7-8f9e-1a20117d05be-istiod-ca-cert" pod "a9342248-e93d-4bb7-8f9e-1a20117d05be" (UID: "a9342248-e93d-4bb7-8f9e-1a20117d05be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:06.417313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.417296 6587 scope.go:122] "RemoveContainer" containerID="22dea87412cbf5bbc4a0e512fbb02cf95405e767b261bdb3c9c07677b580e6e2" Oct 30 05:36:06.455457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.455157 6587 scope.go:122] "RemoveContainer" containerID="9eb9228aa36076f94b3a55f880c0f669fd0653015e8c74a55232fef3f8977eeb" Oct 30 05:36:06.455948 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:06.455670 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb9228aa36076f94b3a55f880c0f669fd0653015e8c74a55232fef3f8977eeb\": container with ID starting with 9eb9228aa36076f94b3a55f880c0f669fd0653015e8c74a55232fef3f8977eeb not found: ID does not exist" containerID="9eb9228aa36076f94b3a55f880c0f669fd0653015e8c74a55232fef3f8977eeb" Oct 30 05:36:06.455948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.455708 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb9228aa36076f94b3a55f880c0f669fd0653015e8c74a55232fef3f8977eeb"} err="failed to get container status \"9eb9228aa36076f94b3a55f880c0f669fd0653015e8c74a55232fef3f8977eeb\": rpc error: code = NotFound desc = could not find container \"9eb9228aa36076f94b3a55f880c0f669fd0653015e8c74a55232fef3f8977eeb\": container with ID starting with 9eb9228aa36076f94b3a55f880c0f669fd0653015e8c74a55232fef3f8977eeb not found: ID does not exist" Oct 30 05:36:06.455948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.455732 6587 scope.go:122] "RemoveContainer" containerID="22dea87412cbf5bbc4a0e512fbb02cf95405e767b261bdb3c9c07677b580e6e2" Oct 30 05:36:06.456930 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:06.456234 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22dea87412cbf5bbc4a0e512fbb02cf95405e767b261bdb3c9c07677b580e6e2\": container with ID starting with 22dea87412cbf5bbc4a0e512fbb02cf95405e767b261bdb3c9c07677b580e6e2 not found: ID does not exist" containerID="22dea87412cbf5bbc4a0e512fbb02cf95405e767b261bdb3c9c07677b580e6e2" Oct 30 05:36:06.456930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.456262 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22dea87412cbf5bbc4a0e512fbb02cf95405e767b261bdb3c9c07677b580e6e2"} err="failed to get container status \"22dea87412cbf5bbc4a0e512fbb02cf95405e767b261bdb3c9c07677b580e6e2\": rpc error: code = NotFound desc = could not find container \"22dea87412cbf5bbc4a0e512fbb02cf95405e767b261bdb3c9c07677b580e6e2\": container with ID starting with 22dea87412cbf5bbc4a0e512fbb02cf95405e767b261bdb3c9c07677b580e6e2 not found: ID does not exist" Oct 30 05:36:06.456930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.456280 6587 scope.go:122] "RemoveContainer" containerID="4e8358ce19edfc71ed77677c149dda992ea9fe91ef2c2430199344c4077761e8" Oct 30 05:36:06.483068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.483045 6587 scope.go:122] "RemoveContainer" containerID="4e8358ce19edfc71ed77677c149dda992ea9fe91ef2c2430199344c4077761e8" Oct 30 05:36:06.483428 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:06.483404 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8358ce19edfc71ed77677c149dda992ea9fe91ef2c2430199344c4077761e8\": container with ID starting with 4e8358ce19edfc71ed77677c149dda992ea9fe91ef2c2430199344c4077761e8 not found: ID does not exist" containerID="4e8358ce19edfc71ed77677c149dda992ea9fe91ef2c2430199344c4077761e8" Oct 30 05:36:06.483517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.483435 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8358ce19edfc71ed77677c149dda992ea9fe91ef2c2430199344c4077761e8"} err="failed to get container status \"4e8358ce19edfc71ed77677c149dda992ea9fe91ef2c2430199344c4077761e8\": rpc error: code = NotFound desc = could not find container \"4e8358ce19edfc71ed77677c149dda992ea9fe91ef2c2430199344c4077761e8\": container with ID starting with 4e8358ce19edfc71ed77677c149dda992ea9fe91ef2c2430199344c4077761e8 not found: ID does not exist" Oct 30 05:36:06.503925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.503901 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-token\" (UniqueName: \"kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-token\") pod \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " Oct 30 05:36:06.504126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.504001 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-socket\") pod \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " Oct 30 05:36:06.504126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.504031 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-envoy\") pod \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " Oct 30 05:36:06.504126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.504092 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-certs\") pod \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " Oct 30 05:36:06.504126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.504122 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istiod-ca-cert\") pod \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " Oct 30 05:36:06.504400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.504176 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-podinfo\") pod \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " Oct 30 05:36:06.504400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.504280 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-kube-api-access-cmfxm\" (UniqueName: \"kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-kube-api-access-cmfxm\") pod \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " Oct 30 05:36:06.504400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.504325 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-credential-socket\") pod \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " Oct 30 05:36:06.504597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.504438 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-socket" pod "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" (UID: "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.504597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.504501 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-data\") pod \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\" (UID: \"5e2dba4c-5813-4ad4-9a12-2c86d5e817ee\") " Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.504768 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-credential-socket" pod "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" (UID: "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.504832 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istiod-ca-cert" pod "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" (UID: "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.504994 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-certs" pod "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" (UID: "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505642 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505663 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505678 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505693 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505709 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txxss\" (UniqueName: \"kubernetes.io/projected/a9342248-e93d-4bb7-8f9e-1a20117d05be-kube-api-access-txxss\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505724 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505737 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505750 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505748 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-data" pod "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" (UID: "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505763 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505776 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9342248-e93d-4bb7-8f9e-1a20117d05be-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.505777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505789 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.506731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505804 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.506731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.505818 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9342248-e93d-4bb7-8f9e-1a20117d05be-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.509808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.509782 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-token" pod "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" (UID: "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:06.509937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.509891 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-kube-api-access-cmfxm" pod "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" (UID: "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee"). InnerVolumeSpecName "kube-api-access-cmfxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:06.510053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.510028 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-podinfo" pod "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" (UID: "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:06.510270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.510252 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-envoy" pod "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" (UID: "5e2dba4c-5813-4ad4-9a12-2c86d5e817ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:06.611967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.611852 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmfxm\" (UniqueName: \"kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-kube-api-access-cmfxm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.611967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.611896 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.611967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.611912 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.611967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.611928 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.611967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.611940 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:06.753409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.753376 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd"] Oct 30 05:36:06.767002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.766973 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd"] Oct 30 05:36:06.801398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.801328 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2"] Oct 30 05:36:06.805341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:06.805320 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2"] Oct 30 05:36:07.036722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:07.036666 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf"] Oct 30 05:36:07.040742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:07.040717 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68396a5e-759vf"] Oct 30 05:36:07.582777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:07.582743 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" path="/var/lib/kubelet/pods/5e2dba4c-5813-4ad4-9a12-2c86d5e817ee/volumes" Oct 30 05:36:07.583854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:07.583835 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" path="/var/lib/kubelet/pods/a9342248-e93d-4bb7-8f9e-1a20117d05be/volumes" Oct 30 05:36:07.584316 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:07.584302 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5b72fa-94e2-47ea-805b-13192f14c1a3" path="/var/lib/kubelet/pods/ca5b72fa-94e2-47ea-805b-13192f14c1a3/volumes" Oct 30 05:36:07.585310 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:07.585288 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" path="/var/lib/kubelet/pods/cfc65f69-cd06-4575-9e52-29f042c2bdc7/volumes" Oct 30 05:36:08.633444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.633405 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9"] Oct 30 05:36:08.634655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.634626 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" Oct 30 05:36:08.634655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.634656 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" Oct 30 05:36:08.634875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.634681 6587 cpu_manager.go:414] "containerMap: removing container" podUID="8657c2a6-caa4-4668-969e-1a6433fed6e1" containerName="revoke-keys" Oct 30 05:36:08.634875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.634690 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8657c2a6-caa4-4668-969e-1a6433fed6e1" containerName="revoke-keys" Oct 30 05:36:08.634875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.634701 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" Oct 30 05:36:08.634875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.634709 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" Oct 30 05:36:08.634875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.634742 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" containerName="istio-proxy" Oct 30 05:36:08.639945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.634750 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" containerName="istio-proxy" Oct 30 05:36:08.639945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.639598 6587 cpu_manager.go:414] "containerMap: removing container" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" Oct 30 05:36:08.639945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.639616 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" Oct 30 05:36:08.639945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.639661 6587 cpu_manager.go:414] "containerMap: removing container" podUID="58d2942b-f563-4e66-b8b5-f46845890275" containerName="istio-proxy" Oct 30 05:36:08.639945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.639673 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d2942b-f563-4e66-b8b5-f46845890275" containerName="istio-proxy" Oct 30 05:36:08.640218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.639960 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" Oct 30 05:36:08.640218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.639990 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9342248-e93d-4bb7-8f9e-1a20117d05be" containerName="istio-proxy" Oct 30 05:36:08.640218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.640006 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="8657c2a6-caa4-4668-969e-1a6433fed6e1" containerName="revoke-keys" Oct 30 05:36:08.640218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.640026 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e2dba4c-5813-4ad4-9a12-2c86d5e817ee" containerName="istio-proxy" Oct 30 05:36:08.640218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.640051 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" Oct 30 05:36:08.640218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.640069 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d2942b-f563-4e66-b8b5-f46845890275" containerName="istio-proxy" Oct 30 05:36:08.646507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.646482 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9"] Oct 30 05:36:08.646637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.646616 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" Oct 30 05:36:08.666755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.666730 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7226a8-dockercfg-z2c8z\"" Oct 30 05:36:08.680241 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.680214 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7226a8\"" Oct 30 05:36:08.816498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.816463 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f145154-169d-4566-a3a9-8dec41416ceb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9\" (UID: \"4f145154-169d-4566-a3a9-8dec41416ceb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" Oct 30 05:36:08.816690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.816522 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47zvh\" (UniqueName: \"kubernetes.io/projected/4f145154-169d-4566-a3a9-8dec41416ceb-kube-api-access-47zvh\") pod \"payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9\" (UID: \"4f145154-169d-4566-a3a9-8dec41416ceb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" Oct 30 05:36:08.918165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.918078 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f145154-169d-4566-a3a9-8dec41416ceb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9\" (UID: \"4f145154-169d-4566-a3a9-8dec41416ceb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" Oct 30 05:36:08.918165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.918155 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47zvh\" (UniqueName: \"kubernetes.io/projected/4f145154-169d-4566-a3a9-8dec41416ceb-kube-api-access-47zvh\") pod \"payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9\" (UID: \"4f145154-169d-4566-a3a9-8dec41416ceb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" Oct 30 05:36:08.918789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.918768 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f145154-169d-4566-a3a9-8dec41416ceb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9\" (UID: \"4f145154-169d-4566-a3a9-8dec41416ceb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" Oct 30 05:36:08.940611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.940582 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47zvh\" (UniqueName: \"kubernetes.io/projected/4f145154-169d-4566-a3a9-8dec41416ceb-kube-api-access-47zvh\") pod \"payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9\" (UID: \"4f145154-169d-4566-a3a9-8dec41416ceb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" Oct 30 05:36:08.967312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:08.967288 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" Oct 30 05:36:09.293889 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:36:09.293849 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f145154_169d_4566_a3a9_8dec41416ceb.slice/crio-55ae0e0ddcb94a1b42113cef0a8095c875a11d75b5b3516b59bd2574eb889256 WatchSource:0}: Error finding container 55ae0e0ddcb94a1b42113cef0a8095c875a11d75b5b3516b59bd2574eb889256: Status 404 returned error can't find the container with id 55ae0e0ddcb94a1b42113cef0a8095c875a11d75b5b3516b59bd2574eb889256 Oct 30 05:36:09.297255 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:09.297228 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9"] Oct 30 05:36:09.303251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:09.303227 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" Oct 30 05:36:09.303419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:09.303301 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" Oct 30 05:36:09.524637 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:09.524596 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" event={"ID":"4f145154-169d-4566-a3a9-8dec41416ceb","Type":"ContainerStarted","Data":"733356db3d653e9a1a383fbf6522313ffd9062f0c7126ea45705aed186c567cf"} Oct 30 05:36:09.524863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:09.524668 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" event={"ID":"4f145154-169d-4566-a3a9-8dec41416ceb","Type":"ContainerStarted","Data":"55ae0e0ddcb94a1b42113cef0a8095c875a11d75b5b3516b59bd2574eb889256"} Oct 30 05:36:09.524863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:09.524694 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" Oct 30 05:36:09.573824 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:09.573766 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" podStartSLOduration=1.5737503959999999 podStartE2EDuration="1.573750396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:09.569844022 +0000 UTC m=+3094.097610808" watchObservedRunningTime="2026-10-30 05:36:09.573750396 +0000 UTC m=+3094.101517182" Oct 30 05:36:10.907064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:10.907021 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2"] Oct 30 05:36:10.941898 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:10.941862 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v"] Oct 30 05:36:11.409146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:11.409116 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:36:12.017785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.017748 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq"] Oct 30 05:36:12.018624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.018604 6587 cpu_manager.go:414] "containerMap: removing container" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" Oct 30 05:36:12.018679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.018625 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" Oct 30 05:36:12.018801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.018791 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc65f69-cd06-4575-9e52-29f042c2bdc7" containerName="istio-proxy" Oct 30 05:36:12.022697 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.022677 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" Oct 30 05:36:12.042221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.042196 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:36:12.043054 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.043026 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq"] Oct 30 05:36:12.050496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.050472 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:36:12.082978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.082943 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq\" (UID: \"c0835306-e772-4743-8dae-4e67254a4ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" Oct 30 05:36:12.083171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.083104 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq\" (UID: \"c0835306-e772-4743-8dae-4e67254a4ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" Oct 30 05:36:12.083253 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.083219 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl94p\" (UniqueName: \"kubernetes.io/projected/c0835306-e772-4743-8dae-4e67254a4ff4-kube-api-access-jl94p\") pod \"maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq\" (UID: \"c0835306-e772-4743-8dae-4e67254a4ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" Oct 30 05:36:12.184827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.184793 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq\" (UID: \"c0835306-e772-4743-8dae-4e67254a4ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" Oct 30 05:36:12.185022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.184901 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq\" (UID: \"c0835306-e772-4743-8dae-4e67254a4ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" Oct 30 05:36:12.185022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.184995 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl94p\" (UniqueName: \"kubernetes.io/projected/c0835306-e772-4743-8dae-4e67254a4ff4-kube-api-access-jl94p\") pod \"maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq\" (UID: \"c0835306-e772-4743-8dae-4e67254a4ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" Oct 30 05:36:12.185662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.185636 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq\" (UID: \"c0835306-e772-4743-8dae-4e67254a4ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" Oct 30 05:36:12.187427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.187404 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq\" (UID: \"c0835306-e772-4743-8dae-4e67254a4ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" Oct 30 05:36:12.206662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.206630 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl94p\" (UniqueName: \"kubernetes.io/projected/c0835306-e772-4743-8dae-4e67254a4ff4-kube-api-access-jl94p\") pod \"maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq\" (UID: \"c0835306-e772-4743-8dae-4e67254a4ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" Oct 30 05:36:12.338642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.338606 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" Oct 30 05:36:12.667754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:12.667722 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq"] Oct 30 05:36:12.668678 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:36:12.668653 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0835306_e772_4743_8dae_4e67254a4ff4.slice/crio-70c0c93f7a81a80417ef0e2b26b8b544a71b698c26b4d5cd663334b3a7e139a8 WatchSource:0}: Error finding container 70c0c93f7a81a80417ef0e2b26b8b544a71b698c26b4d5cd663334b3a7e139a8: Status 404 returned error can't find the container with id 70c0c93f7a81a80417ef0e2b26b8b544a71b698c26b4d5cd663334b3a7e139a8 Oct 30 05:36:13.572250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:13.572200 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" event={"ID":"c0835306-e772-4743-8dae-4e67254a4ff4","Type":"ContainerStarted","Data":"c1c9328a2f9bbfb6f768cca1923822974bb05d8520f771e752041123508d1974"} Oct 30 05:36:13.572716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:13.572268 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" event={"ID":"c0835306-e772-4743-8dae-4e67254a4ff4","Type":"ContainerStarted","Data":"70c0c93f7a81a80417ef0e2b26b8b544a71b698c26b4d5cd663334b3a7e139a8"} Oct 30 05:36:13.633433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:13.633369 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" podStartSLOduration=2.633336461 podStartE2EDuration="2.633336461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:13.630151422 +0000 UTC m=+3098.157918209" watchObservedRunningTime="2026-10-30 05:36:13.633336461 +0000 UTC m=+3098.161103248" Oct 30 05:36:14.582652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:14.582621 6587 generic.go:356] "Generic (PLEG): container finished" podID="c0835306-e772-4743-8dae-4e67254a4ff4" containerID="c1c9328a2f9bbfb6f768cca1923822974bb05d8520f771e752041123508d1974" exitCode=0 Oct 30 05:36:14.583051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:14.582704 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" event={"ID":"c0835306-e772-4743-8dae-4e67254a4ff4","Type":"ContainerDied","Data":"c1c9328a2f9bbfb6f768cca1923822974bb05d8520f771e752041123508d1974"} Oct 30 05:36:15.788434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:15.788401 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" Oct 30 05:36:15.847416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:15.847319 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0835306-e772-4743-8dae-4e67254a4ff4-kube-api-access-jl94p\" (UniqueName: \"kubernetes.io/projected/c0835306-e772-4743-8dae-4e67254a4ff4-kube-api-access-jl94p\") pod \"c0835306-e772-4743-8dae-4e67254a4ff4\" (UID: \"c0835306-e772-4743-8dae-4e67254a4ff4\") " Oct 30 05:36:15.847416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:15.847388 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-xks-ca\") pod \"c0835306-e772-4743-8dae-4e67254a4ff4\" (UID: \"c0835306-e772-4743-8dae-4e67254a4ff4\") " Oct 30 05:36:15.847653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:15.847459 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-openshift-ca\") pod \"c0835306-e772-4743-8dae-4e67254a4ff4\" (UID: \"c0835306-e772-4743-8dae-4e67254a4ff4\") " Oct 30 05:36:15.847901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:15.847864 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-openshift-ca" pod "c0835306-e772-4743-8dae-4e67254a4ff4" (UID: "c0835306-e772-4743-8dae-4e67254a4ff4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:15.848236 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:15.848213 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:15.849918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:15.849887 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0835306-e772-4743-8dae-4e67254a4ff4-kube-api-access-jl94p" pod "c0835306-e772-4743-8dae-4e67254a4ff4" (UID: "c0835306-e772-4743-8dae-4e67254a4ff4"). InnerVolumeSpecName "kube-api-access-jl94p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:15.850429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:15.850402 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-xks-ca" pod "c0835306-e772-4743-8dae-4e67254a4ff4" (UID: "c0835306-e772-4743-8dae-4e67254a4ff4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:15.908384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:15.908336 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" podUID="f557a246-4b9a-4475-b2f1-5b7df159e4ac" containerName="payload-processing" containerID="cri-o://df893f2c6caf6dcbafc1e2100e09b6625787783476c4a33f29f077cf43972896" gracePeriod=25 Oct 30 05:36:15.942313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:15.942267 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" podUID="a00441e4-210e-48f0-bd77-057db2fe849d" containerName="payload-pre-processing" containerID="cri-o://65a69e5de952121f728ec6ec17b89b30295e41552e317b849de694e9574bc2f3" gracePeriod=25 Oct 30 05:36:15.949829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:15.949798 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl94p\" (UniqueName: \"kubernetes.io/projected/c0835306-e772-4743-8dae-4e67254a4ff4-kube-api-access-jl94p\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:15.949952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:15.949831 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0835306-e772-4743-8dae-4e67254a4ff4-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:16.600721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:16.600695 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" Oct 30 05:36:16.600878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:16.600693 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq" event={"ID":"c0835306-e772-4743-8dae-4e67254a4ff4","Type":"ContainerDied","Data":"70c0c93f7a81a80417ef0e2b26b8b544a71b698c26b4d5cd663334b3a7e139a8"} Oct 30 05:36:16.600878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:16.600817 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70c0c93f7a81a80417ef0e2b26b8b544a71b698c26b4d5cd663334b3a7e139a8" Oct 30 05:36:18.042332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:18.042296 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp"] Oct 30 05:36:18.050789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:18.050761 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26545910-vvpxp"] Oct 30 05:36:18.643727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:18.640905 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g"] Oct 30 05:36:18.643727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:18.641196 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" podUID="aee01eed-8cc5-4994-a5fc-7d860679fe86" containerName="maas-api" containerID="cri-o://6920a24e0f4032f2ad768dce636d6386b8a59fc06e2580a8c989e3a17cfb1b6c" gracePeriod=30 Oct 30 05:36:18.988599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:18.988567 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:36:19.117533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.117500 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-metrics-certs\") pod \"aee01eed-8cc5-4994-a5fc-7d860679fe86\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " Oct 30 05:36:19.118012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.117563 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee01eed-8cc5-4994-a5fc-7d860679fe86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee01eed-8cc5-4994-a5fc-7d860679fe86-openshift-service-ca\") pod \"aee01eed-8cc5-4994-a5fc-7d860679fe86\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " Oct 30 05:36:19.118012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.117747 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee01eed-8cc5-4994-a5fc-7d860679fe86-kube-api-access-spxr4\" (UniqueName: \"kubernetes.io/projected/aee01eed-8cc5-4994-a5fc-7d860679fe86-kube-api-access-spxr4\") pod \"aee01eed-8cc5-4994-a5fc-7d860679fe86\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " Oct 30 05:36:19.118012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.117854 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-maas-api-tls\") pod \"aee01eed-8cc5-4994-a5fc-7d860679fe86\" (UID: \"aee01eed-8cc5-4994-a5fc-7d860679fe86\") " Oct 30 05:36:19.118012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.117912 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee01eed-8cc5-4994-a5fc-7d860679fe86-openshift-service-ca" pod "aee01eed-8cc5-4994-a5fc-7d860679fe86" (UID: "aee01eed-8cc5-4994-a5fc-7d860679fe86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:19.118657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.118635 6587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee01eed-8cc5-4994-a5fc-7d860679fe86-openshift-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:19.119767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.119735 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-metrics-certs" pod "aee01eed-8cc5-4994-a5fc-7d860679fe86" (UID: "aee01eed-8cc5-4994-a5fc-7d860679fe86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:19.119876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.119816 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-maas-api-tls" pod "aee01eed-8cc5-4994-a5fc-7d860679fe86" (UID: "aee01eed-8cc5-4994-a5fc-7d860679fe86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:19.119876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.119818 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee01eed-8cc5-4994-a5fc-7d860679fe86-kube-api-access-spxr4" pod "aee01eed-8cc5-4994-a5fc-7d860679fe86" (UID: "aee01eed-8cc5-4994-a5fc-7d860679fe86"). InnerVolumeSpecName "kube-api-access-spxr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:19.228443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.227878 6587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-metrics-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:19.228443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.227918 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spxr4\" (UniqueName: \"kubernetes.io/projected/aee01eed-8cc5-4994-a5fc-7d860679fe86-kube-api-access-spxr4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:19.228443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.227933 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee01eed-8cc5-4994-a5fc-7d860679fe86-maas-api-tls\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:19.585511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.585476 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d829ce66-6d19-46de-8029-2ebf1f3fbe09" path="/var/lib/kubelet/pods/d829ce66-6d19-46de-8029-2ebf1f3fbe09/volumes" Oct 30 05:36:19.633603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.633565 6587 generic.go:356] "Generic (PLEG): container finished" podID="aee01eed-8cc5-4994-a5fc-7d860679fe86" containerID="6920a24e0f4032f2ad768dce636d6386b8a59fc06e2580a8c989e3a17cfb1b6c" exitCode=0 Oct 30 05:36:19.633803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.633621 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" event={"ID":"aee01eed-8cc5-4994-a5fc-7d860679fe86","Type":"ContainerDied","Data":"6920a24e0f4032f2ad768dce636d6386b8a59fc06e2580a8c989e3a17cfb1b6c"} Oct 30 05:36:19.633803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.633636 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" Oct 30 05:36:19.633803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.633667 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g" event={"ID":"aee01eed-8cc5-4994-a5fc-7d860679fe86","Type":"ContainerDied","Data":"2e374326d1316d97d5233493024856a72587e366f21455952ceb3a244bc2c4cd"} Oct 30 05:36:19.633803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.633689 6587 scope.go:122] "RemoveContainer" containerID="6920a24e0f4032f2ad768dce636d6386b8a59fc06e2580a8c989e3a17cfb1b6c" Oct 30 05:36:19.645693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.645677 6587 scope.go:122] "RemoveContainer" containerID="6920a24e0f4032f2ad768dce636d6386b8a59fc06e2580a8c989e3a17cfb1b6c" Oct 30 05:36:19.646005 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:19.645986 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6920a24e0f4032f2ad768dce636d6386b8a59fc06e2580a8c989e3a17cfb1b6c\": container with ID starting with 6920a24e0f4032f2ad768dce636d6386b8a59fc06e2580a8c989e3a17cfb1b6c not found: ID does not exist" containerID="6920a24e0f4032f2ad768dce636d6386b8a59fc06e2580a8c989e3a17cfb1b6c" Oct 30 05:36:19.646071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.646015 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6920a24e0f4032f2ad768dce636d6386b8a59fc06e2580a8c989e3a17cfb1b6c"} err="failed to get container status \"6920a24e0f4032f2ad768dce636d6386b8a59fc06e2580a8c989e3a17cfb1b6c\": rpc error: code = NotFound desc = could not find container \"6920a24e0f4032f2ad768dce636d6386b8a59fc06e2580a8c989e3a17cfb1b6c\": container with ID starting with 6920a24e0f4032f2ad768dce636d6386b8a59fc06e2580a8c989e3a17cfb1b6c not found: ID does not exist" Oct 30 05:36:19.651017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.650993 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w"] Oct 30 05:36:19.682448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.677712 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b5b27e-9lj7w"] Oct 30 05:36:19.710047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.710012 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g"] Oct 30 05:36:19.716867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:19.716837 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b5b27e-74f45fdc9f-tvf2g"] Oct 30 05:36:20.237495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.237455 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" containerID="cri-o://61a646507e98ca985d624b02ba40160aec40498453f64db0037ca3e39825c0d8" gracePeriod=45 Oct 30 05:36:20.537619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.537543 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" Oct 30 05:36:20.571894 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.571868 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:36:20.649114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.649076 6587 generic.go:356] "Generic (PLEG): container finished" podID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerID="61a646507e98ca985d624b02ba40160aec40498453f64db0037ca3e39825c0d8" exitCode=0 Oct 30 05:36:20.649443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.649418 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" event={"ID":"5e79efb5-e47f-4837-8193-0cd9f6343bc4","Type":"ContainerDied","Data":"61a646507e98ca985d624b02ba40160aec40498453f64db0037ca3e39825c0d8"} Oct 30 05:36:20.649554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.649466 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" event={"ID":"5e79efb5-e47f-4837-8193-0cd9f6343bc4","Type":"ContainerDied","Data":"7533e7e74596d15ebdca1d308787b15c2a8805ebe661dd4dc1604e9705831647"} Oct 30 05:36:20.649554 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.649489 6587 scope.go:122] "RemoveContainer" containerID="61a646507e98ca985d624b02ba40160aec40498453f64db0037ca3e39825c0d8" Oct 30 05:36:20.649693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.649645 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" Oct 30 05:36:20.652406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.652382 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-model-cache\") pod \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " Oct 30 05:36:20.652530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.652476 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tls-certs\") pod \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " Oct 30 05:36:20.652530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.652509 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-dshm\") pod \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " Oct 30 05:36:20.652654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.652623 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-home\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-home\") pod \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " Oct 30 05:36:20.652715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.652704 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tmp-dir\") pod \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " Oct 30 05:36:20.652821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.652797 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e79efb5-e47f-4837-8193-0cd9f6343bc4-kube-api-access-ltnzj\" (UniqueName: \"kubernetes.io/projected/5e79efb5-e47f-4837-8193-0cd9f6343bc4-kube-api-access-ltnzj\") pod \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\" (UID: \"5e79efb5-e47f-4837-8193-0cd9f6343bc4\") " Oct 30 05:36:20.655194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.654485 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tmp-dir" pod "5e79efb5-e47f-4837-8193-0cd9f6343bc4" (UID: "5e79efb5-e47f-4837-8193-0cd9f6343bc4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:20.655531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.655510 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-home" pod "5e79efb5-e47f-4837-8193-0cd9f6343bc4" (UID: "5e79efb5-e47f-4837-8193-0cd9f6343bc4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:20.656400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.656113 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-model-cache" pod "5e79efb5-e47f-4837-8193-0cd9f6343bc4" (UID: "5e79efb5-e47f-4837-8193-0cd9f6343bc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:20.667475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.667442 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-dshm" pod "5e79efb5-e47f-4837-8193-0cd9f6343bc4" (UID: "5e79efb5-e47f-4837-8193-0cd9f6343bc4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:20.667626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.667511 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tls-certs" pod "5e79efb5-e47f-4837-8193-0cd9f6343bc4" (UID: "5e79efb5-e47f-4837-8193-0cd9f6343bc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:20.672048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.672014 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e79efb5-e47f-4837-8193-0cd9f6343bc4-kube-api-access-ltnzj" pod "5e79efb5-e47f-4837-8193-0cd9f6343bc4" (UID: "5e79efb5-e47f-4837-8193-0cd9f6343bc4"). InnerVolumeSpecName "kube-api-access-ltnzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:20.715981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.715229 6587 scope.go:122] "RemoveContainer" containerID="61a646507e98ca985d624b02ba40160aec40498453f64db0037ca3e39825c0d8" Oct 30 05:36:20.715981 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:20.715608 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a646507e98ca985d624b02ba40160aec40498453f64db0037ca3e39825c0d8\": container with ID starting with 61a646507e98ca985d624b02ba40160aec40498453f64db0037ca3e39825c0d8 not found: ID does not exist" containerID="61a646507e98ca985d624b02ba40160aec40498453f64db0037ca3e39825c0d8" Oct 30 05:36:20.715981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.715646 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a646507e98ca985d624b02ba40160aec40498453f64db0037ca3e39825c0d8"} err="failed to get container status \"61a646507e98ca985d624b02ba40160aec40498453f64db0037ca3e39825c0d8\": rpc error: code = NotFound desc = could not find container \"61a646507e98ca985d624b02ba40160aec40498453f64db0037ca3e39825c0d8\": container with ID starting with 61a646507e98ca985d624b02ba40160aec40498453f64db0037ca3e39825c0d8 not found: ID does not exist" Oct 30 05:36:20.756162 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.756123 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.756162 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.756160 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.756452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.756173 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.756452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.756185 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.756452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.756198 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltnzj\" (UniqueName: \"kubernetes.io/projected/5e79efb5-e47f-4837-8193-0cd9f6343bc4-kube-api-access-ltnzj\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.756452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.756210 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e79efb5-e47f-4837-8193-0cd9f6343bc4-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:20.761056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.761022 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" podUID="a00441e4-210e-48f0-bd77-057db2fe849d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:36:20.767045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:20.767008 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" podUID="f557a246-4b9a-4475-b2f1-5b7df159e4ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:36:21.009190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:21.009155 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz"] Oct 30 05:36:21.016802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:21.016771 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz"] Oct 30 05:36:21.448301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:21.448255 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/distinct-w4-d55037-kserve-7dc5744b9c-dltpz" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded" Oct 30 05:36:21.582853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:21.582820 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" path="/var/lib/kubelet/pods/5e79efb5-e47f-4837-8193-0cd9f6343bc4/volumes" Oct 30 05:36:21.583407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:21.583390 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8657c2a6-caa4-4668-969e-1a6433fed6e1" path="/var/lib/kubelet/pods/8657c2a6-caa4-4668-969e-1a6433fed6e1/volumes" Oct 30 05:36:21.583886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:21.583871 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee01eed-8cc5-4994-a5fc-7d860679fe86" path="/var/lib/kubelet/pods/aee01eed-8cc5-4994-a5fc-7d860679fe86/volumes" Oct 30 05:36:23.910908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.910872 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6"] Oct 30 05:36:23.911744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.911726 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c0835306-e772-4743-8dae-4e67254a4ff4" containerName="revoke-keys" Oct 30 05:36:23.911816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.911745 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0835306-e772-4743-8dae-4e67254a4ff4" containerName="revoke-keys" Oct 30 05:36:23.911816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.911777 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" Oct 30 05:36:23.911816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.911782 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" Oct 30 05:36:23.911816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.911799 6587 cpu_manager.go:414] "containerMap: removing container" podUID="aee01eed-8cc5-4994-a5fc-7d860679fe86" containerName="maas-api" Oct 30 05:36:23.911816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.911804 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee01eed-8cc5-4994-a5fc-7d860679fe86" containerName="maas-api" Oct 30 05:36:23.912527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.911946 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee01eed-8cc5-4994-a5fc-7d860679fe86" containerName="maas-api" Oct 30 05:36:23.912527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.911963 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0835306-e772-4743-8dae-4e67254a4ff4" containerName="revoke-keys" Oct 30 05:36:23.912527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.911973 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e79efb5-e47f-4837-8193-0cd9f6343bc4" containerName="main" Oct 30 05:36:23.917197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.917173 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" Oct 30 05:36:23.928643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.928617 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0439879d-zmhv6\" (UID: \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" Oct 30 05:36:23.928772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.928653 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0439879d-zmhv6\" (UID: \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" Oct 30 05:36:23.928878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.928856 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmn74\" (UniqueName: \"kubernetes.io/projected/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-kube-api-access-mmn74\") pod \"maas-api-revoke-sub-0439879d-zmhv6\" (UID: \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" Oct 30 05:36:23.940117 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.940089 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6"] Oct 30 05:36:23.949481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.949451 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:36:23.949945 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.949921 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" Oct 30 05:36:23.950121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.950102 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:36:23.961055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.961029 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6"] Oct 30 05:36:23.982161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:23.982135 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6"] Oct 30 05:36:24.029998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.029958 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0439879d-zmhv6\" (UID: \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" Oct 30 05:36:24.030200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.030010 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0439879d-zmhv6\" (UID: \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" Oct 30 05:36:24.030200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.030103 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79946a68-kdhb6\" (UID: \"f1e12583-5094-4e4d-9593-c71070951f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" Oct 30 05:36:24.030200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.030134 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x67j\" (UniqueName: \"kubernetes.io/projected/f1e12583-5094-4e4d-9593-c71070951f96-kube-api-access-2x67j\") pod \"maas-api-revoke-sub-79946a68-kdhb6\" (UID: \"f1e12583-5094-4e4d-9593-c71070951f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" Oct 30 05:36:24.030392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.030269 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79946a68-kdhb6\" (UID: \"f1e12583-5094-4e4d-9593-c71070951f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" Oct 30 05:36:24.030485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.030463 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmn74\" (UniqueName: \"kubernetes.io/projected/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-kube-api-access-mmn74\") pod \"maas-api-revoke-sub-0439879d-zmhv6\" (UID: \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" Oct 30 05:36:24.030685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.030644 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0439879d-zmhv6\" (UID: \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" Oct 30 05:36:24.032643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.032618 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0439879d-zmhv6\" (UID: \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" Oct 30 05:36:24.052077 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.052042 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmn74\" (UniqueName: \"kubernetes.io/projected/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-kube-api-access-mmn74\") pod \"maas-api-revoke-sub-0439879d-zmhv6\" (UID: \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" Oct 30 05:36:24.132099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.132059 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79946a68-kdhb6\" (UID: \"f1e12583-5094-4e4d-9593-c71070951f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" Oct 30 05:36:24.132319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.132119 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x67j\" (UniqueName: \"kubernetes.io/projected/f1e12583-5094-4e4d-9593-c71070951f96-kube-api-access-2x67j\") pod \"maas-api-revoke-sub-79946a68-kdhb6\" (UID: \"f1e12583-5094-4e4d-9593-c71070951f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" Oct 30 05:36:24.132319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.132196 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79946a68-kdhb6\" (UID: \"f1e12583-5094-4e4d-9593-c71070951f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" Oct 30 05:36:24.132826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.132795 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79946a68-kdhb6\" (UID: \"f1e12583-5094-4e4d-9593-c71070951f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" Oct 30 05:36:24.134642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.134619 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79946a68-kdhb6\" (UID: \"f1e12583-5094-4e4d-9593-c71070951f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" Oct 30 05:36:24.157014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.156981 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x67j\" (UniqueName: \"kubernetes.io/projected/f1e12583-5094-4e4d-9593-c71070951f96-kube-api-access-2x67j\") pod \"maas-api-revoke-sub-79946a68-kdhb6\" (UID: \"f1e12583-5094-4e4d-9593-c71070951f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" Oct 30 05:36:24.232710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.232626 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" Oct 30 05:36:24.278564 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.278532 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" Oct 30 05:36:24.545452 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:36:24.545421 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa50cecf_fcd8_4ada_98f2_787c4d9942ac.slice/crio-bc20701fb8050c442888df8cd4b08b066d144563f1521d60eaa8d4d39337dc23 WatchSource:0}: Error finding container bc20701fb8050c442888df8cd4b08b066d144563f1521d60eaa8d4d39337dc23: Status 404 returned error can't find the container with id bc20701fb8050c442888df8cd4b08b066d144563f1521d60eaa8d4d39337dc23 Oct 30 05:36:24.546286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.546264 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6"] Oct 30 05:36:24.626570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.626541 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6"] Oct 30 05:36:24.692434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:24.692397 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" event={"ID":"aa50cecf-fcd8-4ada-98f2-787c4d9942ac","Type":"ContainerStarted","Data":"bc20701fb8050c442888df8cd4b08b066d144563f1521d60eaa8d4d39337dc23"} Oct 30 05:36:24.719453 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:36:24.719419 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e12583_5094_4e4d_9593_c71070951f96.slice/crio-43dc81c420212f550cf60ca0669861cb0703c01d5313b26ce6af45a87783c59f WatchSource:0}: Error finding container 43dc81c420212f550cf60ca0669861cb0703c01d5313b26ce6af45a87783c59f: Status 404 returned error can't find the container with id 43dc81c420212f550cf60ca0669861cb0703c01d5313b26ce6af45a87783c59f Oct 30 05:36:25.705409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:25.705346 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" event={"ID":"f1e12583-5094-4e4d-9593-c71070951f96","Type":"ContainerStarted","Data":"7f7dcf7618a96faedd5d471580855de53a475d4e6671cfbc5a88b174a094a626"} Oct 30 05:36:25.705854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:25.705427 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" event={"ID":"f1e12583-5094-4e4d-9593-c71070951f96","Type":"ContainerStarted","Data":"43dc81c420212f550cf60ca0669861cb0703c01d5313b26ce6af45a87783c59f"} Oct 30 05:36:25.707499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:25.707472 6587 generic.go:356] "Generic (PLEG): container finished" podID="aa50cecf-fcd8-4ada-98f2-787c4d9942ac" containerID="1ee13303596c09f598758d7ec4308a5e878ddb2c60d2588d25636adb13e3ec8b" exitCode=0 Oct 30 05:36:25.707616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:25.707553 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" event={"ID":"aa50cecf-fcd8-4ada-98f2-787c4d9942ac","Type":"ContainerDied","Data":"1ee13303596c09f598758d7ec4308a5e878ddb2c60d2588d25636adb13e3ec8b"} Oct 30 05:36:25.763867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:25.763809 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" podStartSLOduration=2.763789381 podStartE2EDuration="2.763789381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:25.758538382 +0000 UTC m=+3110.286305169" watchObservedRunningTime="2026-10-30 05:36:25.763789381 +0000 UTC m=+3110.291556169" Oct 30 05:36:25.987455 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:25.987350 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:36:26.718674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:26.718640 6587 generic.go:356] "Generic (PLEG): container finished" podID="f1e12583-5094-4e4d-9593-c71070951f96" containerID="7f7dcf7618a96faedd5d471580855de53a475d4e6671cfbc5a88b174a094a626" exitCode=0 Oct 30 05:36:26.719170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:26.718727 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" event={"ID":"f1e12583-5094-4e4d-9593-c71070951f96","Type":"ContainerDied","Data":"7f7dcf7618a96faedd5d471580855de53a475d4e6671cfbc5a88b174a094a626"} Oct 30 05:36:26.896950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:26.896885 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" Oct 30 05:36:26.985053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:26.984965 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-openshift-ca\") pod \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\" (UID: \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\") " Oct 30 05:36:26.985053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:26.985047 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-kube-api-access-mmn74\" (UniqueName: \"kubernetes.io/projected/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-kube-api-access-mmn74\") pod \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\" (UID: \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\") " Oct 30 05:36:26.985317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:26.985293 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-xks-ca\") pod \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\" (UID: \"aa50cecf-fcd8-4ada-98f2-787c4d9942ac\") " Oct 30 05:36:26.985438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:26.985413 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-openshift-ca" pod "aa50cecf-fcd8-4ada-98f2-787c4d9942ac" (UID: "aa50cecf-fcd8-4ada-98f2-787c4d9942ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:26.986337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:26.986272 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:26.998455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:26.988663 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-xks-ca" pod "aa50cecf-fcd8-4ada-98f2-787c4d9942ac" (UID: "aa50cecf-fcd8-4ada-98f2-787c4d9942ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:26.998455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:26.988734 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-kube-api-access-mmn74" pod "aa50cecf-fcd8-4ada-98f2-787c4d9942ac" (UID: "aa50cecf-fcd8-4ada-98f2-787c4d9942ac"). InnerVolumeSpecName "kube-api-access-mmn74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:27.087912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:27.087878 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmn74\" (UniqueName: \"kubernetes.io/projected/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-kube-api-access-mmn74\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:27.087912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:27.087911 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa50cecf-fcd8-4ada-98f2-787c4d9942ac-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:27.838850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:27.827955 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" Oct 30 05:36:27.841156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:27.840269 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6" event={"ID":"aa50cecf-fcd8-4ada-98f2-787c4d9942ac","Type":"ContainerDied","Data":"bc20701fb8050c442888df8cd4b08b066d144563f1521d60eaa8d4d39337dc23"} Oct 30 05:36:27.841156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:27.840333 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc20701fb8050c442888df8cd4b08b066d144563f1521d60eaa8d4d39337dc23" Oct 30 05:36:28.081289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.081248 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" Oct 30 05:36:28.209986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.209898 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-openshift-ca\") pod \"f1e12583-5094-4e4d-9593-c71070951f96\" (UID: \"f1e12583-5094-4e4d-9593-c71070951f96\") " Oct 30 05:36:28.210150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.210028 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-xks-ca\") pod \"f1e12583-5094-4e4d-9593-c71070951f96\" (UID: \"f1e12583-5094-4e4d-9593-c71070951f96\") " Oct 30 05:36:28.210150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.210052 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e12583-5094-4e4d-9593-c71070951f96-kube-api-access-2x67j\" (UniqueName: \"kubernetes.io/projected/f1e12583-5094-4e4d-9593-c71070951f96-kube-api-access-2x67j\") pod \"f1e12583-5094-4e4d-9593-c71070951f96\" (UID: \"f1e12583-5094-4e4d-9593-c71070951f96\") " Oct 30 05:36:28.210316 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.210288 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-openshift-ca" pod "f1e12583-5094-4e4d-9593-c71070951f96" (UID: "f1e12583-5094-4e4d-9593-c71070951f96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:28.210888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.210857 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:28.212171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.212144 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-xks-ca" pod "f1e12583-5094-4e4d-9593-c71070951f96" (UID: "f1e12583-5094-4e4d-9593-c71070951f96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:28.212336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.212316 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e12583-5094-4e4d-9593-c71070951f96-kube-api-access-2x67j" pod "f1e12583-5094-4e4d-9593-c71070951f96" (UID: "f1e12583-5094-4e4d-9593-c71070951f96"). InnerVolumeSpecName "kube-api-access-2x67j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:28.312331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.312298 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e12583-5094-4e4d-9593-c71070951f96-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:28.312331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.312328 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x67j\" (UniqueName: \"kubernetes.io/projected/f1e12583-5094-4e4d-9593-c71070951f96-kube-api-access-2x67j\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:28.451223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.451188 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7"] Oct 30 05:36:28.452449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.452414 6587 cpu_manager.go:414] "containerMap: removing container" podUID="aa50cecf-fcd8-4ada-98f2-787c4d9942ac" containerName="revoke-keys" Oct 30 05:36:28.452449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.452442 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa50cecf-fcd8-4ada-98f2-787c4d9942ac" containerName="revoke-keys" Oct 30 05:36:28.452670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.452463 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f1e12583-5094-4e4d-9593-c71070951f96" containerName="revoke-keys" Oct 30 05:36:28.452670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.452471 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e12583-5094-4e4d-9593-c71070951f96" containerName="revoke-keys" Oct 30 05:36:28.452782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.452716 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa50cecf-fcd8-4ada-98f2-787c4d9942ac" containerName="revoke-keys" Oct 30 05:36:28.452782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.452733 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e12583-5094-4e4d-9593-c71070951f96" containerName="revoke-keys" Oct 30 05:36:28.457413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.457389 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" Oct 30 05:36:28.515899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.515866 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7"] Oct 30 05:36:28.522386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.522339 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7\" (UID: \"0bc7fb10-7d71-443c-8251-c714fe13f358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" Oct 30 05:36:28.522725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.522703 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4qt4\" (UniqueName: \"kubernetes.io/projected/0bc7fb10-7d71-443c-8251-c714fe13f358-kube-api-access-q4qt4\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7\" (UID: \"0bc7fb10-7d71-443c-8251-c714fe13f358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" Oct 30 05:36:28.522881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.522867 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7\" (UID: \"0bc7fb10-7d71-443c-8251-c714fe13f358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" Oct 30 05:36:28.625208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.625167 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7\" (UID: \"0bc7fb10-7d71-443c-8251-c714fe13f358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" Oct 30 05:36:28.625427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.625407 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4qt4\" (UniqueName: \"kubernetes.io/projected/0bc7fb10-7d71-443c-8251-c714fe13f358-kube-api-access-q4qt4\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7\" (UID: \"0bc7fb10-7d71-443c-8251-c714fe13f358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" Oct 30 05:36:28.625505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.625488 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7\" (UID: \"0bc7fb10-7d71-443c-8251-c714fe13f358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" Oct 30 05:36:28.626322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.626300 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7\" (UID: \"0bc7fb10-7d71-443c-8251-c714fe13f358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" Oct 30 05:36:28.630865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.630837 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7\" (UID: \"0bc7fb10-7d71-443c-8251-c714fe13f358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" Oct 30 05:36:28.653691 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.653661 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4qt4\" (UniqueName: \"kubernetes.io/projected/0bc7fb10-7d71-443c-8251-c714fe13f358-kube-api-access-q4qt4\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7\" (UID: \"0bc7fb10-7d71-443c-8251-c714fe13f358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" Oct 30 05:36:28.779865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.779786 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" Oct 30 05:36:28.822616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.822580 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc"] Oct 30 05:36:28.829452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.829425 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" Oct 30 05:36:28.860528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.858960 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a62fee\"" Oct 30 05:36:28.877923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.869478 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" event={"ID":"f1e12583-5094-4e4d-9593-c71070951f96","Type":"ContainerDied","Data":"43dc81c420212f550cf60ca0669861cb0703c01d5313b26ce6af45a87783c59f"} Oct 30 05:36:28.877923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.869536 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43dc81c420212f550cf60ca0669861cb0703c01d5313b26ce6af45a87783c59f" Oct 30 05:36:28.877923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.869618 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6" Oct 30 05:36:28.877923 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.872888 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc"] Oct 30 05:36:28.888370 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.888323 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb"] Oct 30 05:36:28.895021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.895000 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" Oct 30 05:36:28.913468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.913442 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb"] Oct 30 05:36:28.913833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.913816 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a62fee-dockercfg-d42sl\"" Oct 30 05:36:28.930158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.930130 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21371b98-c41e-4845-9e9b-596a87089643-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb\" (UID: \"21371b98-c41e-4845-9e9b-596a87089643\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" Oct 30 05:36:28.930552 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.930528 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjkb6\" (UniqueName: \"kubernetes.io/projected/21371b98-c41e-4845-9e9b-596a87089643-kube-api-access-pjkb6\") pod \"payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb\" (UID: \"21371b98-c41e-4845-9e9b-596a87089643\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" Oct 30 05:36:28.930690 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.930674 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390e4b2a-b9a8-473c-8338-65eb60160014-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc\" (UID: \"390e4b2a-b9a8-473c-8338-65eb60160014\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" Oct 30 05:36:28.930968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:28.930955 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75nz2\" (UniqueName: \"kubernetes.io/projected/390e4b2a-b9a8-473c-8338-65eb60160014-kube-api-access-75nz2\") pod \"payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc\" (UID: \"390e4b2a-b9a8-473c-8338-65eb60160014\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" Oct 30 05:36:29.181165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.181110 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7"] Oct 30 05:36:29.214023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.213978 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21371b98-c41e-4845-9e9b-596a87089643-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb\" (UID: \"21371b98-c41e-4845-9e9b-596a87089643\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" Oct 30 05:36:29.214192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.214151 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjkb6\" (UniqueName: \"kubernetes.io/projected/21371b98-c41e-4845-9e9b-596a87089643-kube-api-access-pjkb6\") pod \"payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb\" (UID: \"21371b98-c41e-4845-9e9b-596a87089643\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" Oct 30 05:36:29.214258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.214193 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390e4b2a-b9a8-473c-8338-65eb60160014-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc\" (UID: \"390e4b2a-b9a8-473c-8338-65eb60160014\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" Oct 30 05:36:29.214509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.214407 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75nz2\" (UniqueName: \"kubernetes.io/projected/390e4b2a-b9a8-473c-8338-65eb60160014-kube-api-access-75nz2\") pod \"payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc\" (UID: \"390e4b2a-b9a8-473c-8338-65eb60160014\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" Oct 30 05:36:29.215626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.215602 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21371b98-c41e-4845-9e9b-596a87089643-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb\" (UID: \"21371b98-c41e-4845-9e9b-596a87089643\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" Oct 30 05:36:29.215745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.215726 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390e4b2a-b9a8-473c-8338-65eb60160014-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc\" (UID: \"390e4b2a-b9a8-473c-8338-65eb60160014\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" Oct 30 05:36:29.238321 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.238304 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75nz2\" (UniqueName: \"kubernetes.io/projected/390e4b2a-b9a8-473c-8338-65eb60160014-kube-api-access-75nz2\") pod \"payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc\" (UID: \"390e4b2a-b9a8-473c-8338-65eb60160014\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" Oct 30 05:36:29.238528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.238509 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjkb6\" (UniqueName: \"kubernetes.io/projected/21371b98-c41e-4845-9e9b-596a87089643-kube-api-access-pjkb6\") pod \"payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb\" (UID: \"21371b98-c41e-4845-9e9b-596a87089643\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" Oct 30 05:36:29.489988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.489919 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" Oct 30 05:36:29.530483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.530451 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" Oct 30 05:36:29.827677 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:36:29.827639 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390e4b2a_b9a8_473c_8338_65eb60160014.slice/crio-4c4b40f0ec8894ccc9f3a95eb33d0b72f7cf896d0287e9f79599211b78e51c73 WatchSource:0}: Error finding container 4c4b40f0ec8894ccc9f3a95eb33d0b72f7cf896d0287e9f79599211b78e51c73: Status 404 returned error can't find the container with id 4c4b40f0ec8894ccc9f3a95eb33d0b72f7cf896d0287e9f79599211b78e51c73 Oct 30 05:36:29.827887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.827860 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc"] Oct 30 05:36:29.858399 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:36:29.858087 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21371b98_c41e_4845_9e9b_596a87089643.slice/crio-638f4161398cada24a0daa9d492a174759e3c350621094d28fd89117243e03f2 WatchSource:0}: Error finding container 638f4161398cada24a0daa9d492a174759e3c350621094d28fd89117243e03f2: Status 404 returned error can't find the container with id 638f4161398cada24a0daa9d492a174759e3c350621094d28fd89117243e03f2 Oct 30 05:36:29.859443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.859414 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb"] Oct 30 05:36:29.880833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.880805 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" event={"ID":"21371b98-c41e-4845-9e9b-596a87089643","Type":"ContainerStarted","Data":"638f4161398cada24a0daa9d492a174759e3c350621094d28fd89117243e03f2"} Oct 30 05:36:29.882246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.882220 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" event={"ID":"390e4b2a-b9a8-473c-8338-65eb60160014","Type":"ContainerStarted","Data":"4c4b40f0ec8894ccc9f3a95eb33d0b72f7cf896d0287e9f79599211b78e51c73"} Oct 30 05:36:29.884267 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.884241 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" event={"ID":"0bc7fb10-7d71-443c-8251-c714fe13f358","Type":"ContainerStarted","Data":"064c76ebe104a40ff1f2110648bc69592d450806b91e0fef82449c8e3fba67ed"} Oct 30 05:36:29.884392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.884294 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" event={"ID":"0bc7fb10-7d71-443c-8251-c714fe13f358","Type":"ContainerStarted","Data":"823c70bc8ddf7a261a5ccaef227c1d2e5fdab8da88d09d954ec2bf597bcf2033"} Oct 30 05:36:29.935062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:29.934830 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" podStartSLOduration=1.934808622 podStartE2EDuration="1.934808622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:29.928694109 +0000 UTC m=+3114.456460899" watchObservedRunningTime="2026-10-30 05:36:29.934808622 +0000 UTC m=+3114.462575409" Oct 30 05:36:30.761042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:30.760996 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" podUID="a00441e4-210e-48f0-bd77-057db2fe849d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:36:30.761263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:30.761002 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" podUID="f557a246-4b9a-4475-b2f1-5b7df159e4ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:36:30.899341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:30.899292 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" event={"ID":"21371b98-c41e-4845-9e9b-596a87089643","Type":"ContainerStarted","Data":"1bec9fc7c3ddfb7c733861f387b1e3b57a6ff452be0e59ee154d9fe0b5771452"} Oct 30 05:36:30.900626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:30.900500 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" Oct 30 05:36:30.902278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:30.902245 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" event={"ID":"390e4b2a-b9a8-473c-8338-65eb60160014","Type":"ContainerStarted","Data":"46ae69d8350a8d3ce711ebb6cbdb23ce6d5289ea16c9e75ad9aadbf13f1845d2"} Oct 30 05:36:30.902938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:30.902902 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" Oct 30 05:36:30.904560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:30.904526 6587 generic.go:356] "Generic (PLEG): container finished" podID="0bc7fb10-7d71-443c-8251-c714fe13f358" containerID="064c76ebe104a40ff1f2110648bc69592d450806b91e0fef82449c8e3fba67ed" exitCode=0 Oct 30 05:36:30.904893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:30.904875 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" event={"ID":"0bc7fb10-7d71-443c-8251-c714fe13f358","Type":"ContainerDied","Data":"064c76ebe104a40ff1f2110648bc69592d450806b91e0fef82449c8e3fba67ed"} Oct 30 05:36:30.962158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:30.962100 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" podStartSLOduration=2.962085445 podStartE2EDuration="2.962085445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:30.946692058 +0000 UTC m=+3115.474458843" watchObservedRunningTime="2026-10-30 05:36:30.962085445 +0000 UTC m=+3115.489852231" Oct 30 05:36:31.037917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:31.037824 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59"] Oct 30 05:36:31.042597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:31.042544 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" podStartSLOduration=3.042529441 podStartE2EDuration="3.042529441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:31.038117646 +0000 UTC m=+3115.565884432" watchObservedRunningTime="2026-10-30 05:36:31.042529441 +0000 UTC m=+3115.570296227" Oct 30 05:36:31.053074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:31.053043 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d87fd3-xpw59"] Oct 30 05:36:31.583391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:31.583343 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4786587b-cb5c-472b-81af-8a0a9eeb1d70" path="/var/lib/kubelet/pods/4786587b-cb5c-472b-81af-8a0a9eeb1d70/volumes" Oct 30 05:36:32.105275 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:32.105245 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" Oct 30 05:36:32.171553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:32.171465 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc7fb10-7d71-443c-8251-c714fe13f358-kube-api-access-q4qt4\" (UniqueName: \"kubernetes.io/projected/0bc7fb10-7d71-443c-8251-c714fe13f358-kube-api-access-q4qt4\") pod \"0bc7fb10-7d71-443c-8251-c714fe13f358\" (UID: \"0bc7fb10-7d71-443c-8251-c714fe13f358\") " Oct 30 05:36:32.171696 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:32.171563 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-openshift-ca\") pod \"0bc7fb10-7d71-443c-8251-c714fe13f358\" (UID: \"0bc7fb10-7d71-443c-8251-c714fe13f358\") " Oct 30 05:36:32.171696 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:32.171679 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-xks-ca\") pod \"0bc7fb10-7d71-443c-8251-c714fe13f358\" (UID: \"0bc7fb10-7d71-443c-8251-c714fe13f358\") " Oct 30 05:36:32.171952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:32.171925 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-openshift-ca" pod "0bc7fb10-7d71-443c-8251-c714fe13f358" (UID: "0bc7fb10-7d71-443c-8251-c714fe13f358"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:32.172442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:32.172414 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:32.174238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:32.173842 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc7fb10-7d71-443c-8251-c714fe13f358-kube-api-access-q4qt4" pod "0bc7fb10-7d71-443c-8251-c714fe13f358" (UID: "0bc7fb10-7d71-443c-8251-c714fe13f358"). InnerVolumeSpecName "kube-api-access-q4qt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:32.176491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:32.176467 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-xks-ca" pod "0bc7fb10-7d71-443c-8251-c714fe13f358" (UID: "0bc7fb10-7d71-443c-8251-c714fe13f358"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:32.273867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:32.273825 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4qt4\" (UniqueName: \"kubernetes.io/projected/0bc7fb10-7d71-443c-8251-c714fe13f358-kube-api-access-q4qt4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:32.273867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:32.273860 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bc7fb10-7d71-443c-8251-c714fe13f358-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:32.930109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:32.930080 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" Oct 30 05:36:32.930339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:32.930120 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7" event={"ID":"0bc7fb10-7d71-443c-8251-c714fe13f358","Type":"ContainerDied","Data":"823c70bc8ddf7a261a5ccaef227c1d2e5fdab8da88d09d954ec2bf597bcf2033"} Oct 30 05:36:32.930339 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:32.930173 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="823c70bc8ddf7a261a5ccaef227c1d2e5fdab8da88d09d954ec2bf597bcf2033" Oct 30 05:36:36.275049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.274633 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" Oct 30 05:36:36.346304 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.346273 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" Oct 30 05:36:36.349683 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.349645 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f557a246-4b9a-4475-b2f1-5b7df159e4ac-kube-api-access-rwsrq\" (UniqueName: \"kubernetes.io/projected/f557a246-4b9a-4475-b2f1-5b7df159e4ac-kube-api-access-rwsrq\") pod \"f557a246-4b9a-4475-b2f1-5b7df159e4ac\" (UID: \"f557a246-4b9a-4475-b2f1-5b7df159e4ac\") " Oct 30 05:36:36.349853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.349752 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f557a246-4b9a-4475-b2f1-5b7df159e4ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f557a246-4b9a-4475-b2f1-5b7df159e4ac-plugins-config-volume\") pod \"f557a246-4b9a-4475-b2f1-5b7df159e4ac\" (UID: \"f557a246-4b9a-4475-b2f1-5b7df159e4ac\") " Oct 30 05:36:36.350601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.350500 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f557a246-4b9a-4475-b2f1-5b7df159e4ac-plugins-config-volume" pod "f557a246-4b9a-4475-b2f1-5b7df159e4ac" (UID: "f557a246-4b9a-4475-b2f1-5b7df159e4ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:36.350770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.350669 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f557a246-4b9a-4475-b2f1-5b7df159e4ac-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:36.352773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.352745 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f557a246-4b9a-4475-b2f1-5b7df159e4ac-kube-api-access-rwsrq" pod "f557a246-4b9a-4475-b2f1-5b7df159e4ac" (UID: "f557a246-4b9a-4475-b2f1-5b7df159e4ac"). InnerVolumeSpecName "kube-api-access-rwsrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:36.451976 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.451879 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00441e4-210e-48f0-bd77-057db2fe849d-kube-api-access-2266h\" (UniqueName: \"kubernetes.io/projected/a00441e4-210e-48f0-bd77-057db2fe849d-kube-api-access-2266h\") pod \"a00441e4-210e-48f0-bd77-057db2fe849d\" (UID: \"a00441e4-210e-48f0-bd77-057db2fe849d\") " Oct 30 05:36:36.452180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.452156 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00441e4-210e-48f0-bd77-057db2fe849d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a00441e4-210e-48f0-bd77-057db2fe849d-plugins-config-volume\") pod \"a00441e4-210e-48f0-bd77-057db2fe849d\" (UID: \"a00441e4-210e-48f0-bd77-057db2fe849d\") " Oct 30 05:36:36.452858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.452831 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00441e4-210e-48f0-bd77-057db2fe849d-plugins-config-volume" pod "a00441e4-210e-48f0-bd77-057db2fe849d" (UID: "a00441e4-210e-48f0-bd77-057db2fe849d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:36.453015 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.452876 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwsrq\" (UniqueName: \"kubernetes.io/projected/f557a246-4b9a-4475-b2f1-5b7df159e4ac-kube-api-access-rwsrq\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:36.455889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.455860 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00441e4-210e-48f0-bd77-057db2fe849d-kube-api-access-2266h" pod "a00441e4-210e-48f0-bd77-057db2fe849d" (UID: "a00441e4-210e-48f0-bd77-057db2fe849d"). InnerVolumeSpecName "kube-api-access-2266h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:36.554944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.554909 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a00441e4-210e-48f0-bd77-057db2fe849d-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:36.554944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.554939 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2266h\" (UniqueName: \"kubernetes.io/projected/a00441e4-210e-48f0-bd77-057db2fe849d-kube-api-access-2266h\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:36.982414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.982377 6587 generic.go:356] "Generic (PLEG): container finished" podID="a00441e4-210e-48f0-bd77-057db2fe849d" containerID="65a69e5de952121f728ec6ec17b89b30295e41552e317b849de694e9574bc2f3" exitCode=0 Oct 30 05:36:36.982587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.982424 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" event={"ID":"a00441e4-210e-48f0-bd77-057db2fe849d","Type":"ContainerDied","Data":"65a69e5de952121f728ec6ec17b89b30295e41552e317b849de694e9574bc2f3"} Oct 30 05:36:36.982587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.982497 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" event={"ID":"a00441e4-210e-48f0-bd77-057db2fe849d","Type":"ContainerDied","Data":"04d6edbe680bc68a3750de3ce26d26512a64ceeec09e39884a866f159643b9de"} Oct 30 05:36:36.982587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.982443 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v" Oct 30 05:36:36.982587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.982528 6587 scope.go:122] "RemoveContainer" containerID="65a69e5de952121f728ec6ec17b89b30295e41552e317b849de694e9574bc2f3" Oct 30 05:36:36.984289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.984249 6587 generic.go:356] "Generic (PLEG): container finished" podID="f557a246-4b9a-4475-b2f1-5b7df159e4ac" containerID="df893f2c6caf6dcbafc1e2100e09b6625787783476c4a33f29f077cf43972896" exitCode=0 Oct 30 05:36:36.984446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.984295 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" Oct 30 05:36:36.984446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.984334 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" event={"ID":"f557a246-4b9a-4475-b2f1-5b7df159e4ac","Type":"ContainerDied","Data":"df893f2c6caf6dcbafc1e2100e09b6625787783476c4a33f29f077cf43972896"} Oct 30 05:36:36.984545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.984497 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2" event={"ID":"f557a246-4b9a-4475-b2f1-5b7df159e4ac","Type":"ContainerDied","Data":"e8c40b3773582c9b11b9ee8d5cde9d81945edd1cb528d930ca077bd745c9f359"} Oct 30 05:36:36.996969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.996944 6587 scope.go:122] "RemoveContainer" containerID="65a69e5de952121f728ec6ec17b89b30295e41552e317b849de694e9574bc2f3" Oct 30 05:36:36.997269 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:36.997239 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a69e5de952121f728ec6ec17b89b30295e41552e317b849de694e9574bc2f3\": container with ID starting with 65a69e5de952121f728ec6ec17b89b30295e41552e317b849de694e9574bc2f3 not found: ID does not exist" containerID="65a69e5de952121f728ec6ec17b89b30295e41552e317b849de694e9574bc2f3" Oct 30 05:36:36.997342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.997277 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a69e5de952121f728ec6ec17b89b30295e41552e317b849de694e9574bc2f3"} err="failed to get container status \"65a69e5de952121f728ec6ec17b89b30295e41552e317b849de694e9574bc2f3\": rpc error: code = NotFound desc = could not find container \"65a69e5de952121f728ec6ec17b89b30295e41552e317b849de694e9574bc2f3\": container with ID starting with 65a69e5de952121f728ec6ec17b89b30295e41552e317b849de694e9574bc2f3 not found: ID does not exist" Oct 30 05:36:36.997342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:36.997294 6587 scope.go:122] "RemoveContainer" containerID="df893f2c6caf6dcbafc1e2100e09b6625787783476c4a33f29f077cf43972896" Oct 30 05:36:37.007910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:37.007886 6587 scope.go:122] "RemoveContainer" containerID="df893f2c6caf6dcbafc1e2100e09b6625787783476c4a33f29f077cf43972896" Oct 30 05:36:37.008166 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:37.008148 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df893f2c6caf6dcbafc1e2100e09b6625787783476c4a33f29f077cf43972896\": container with ID starting with df893f2c6caf6dcbafc1e2100e09b6625787783476c4a33f29f077cf43972896 not found: ID does not exist" containerID="df893f2c6caf6dcbafc1e2100e09b6625787783476c4a33f29f077cf43972896" Oct 30 05:36:37.008223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:37.008174 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df893f2c6caf6dcbafc1e2100e09b6625787783476c4a33f29f077cf43972896"} err="failed to get container status \"df893f2c6caf6dcbafc1e2100e09b6625787783476c4a33f29f077cf43972896\": rpc error: code = NotFound desc = could not find container \"df893f2c6caf6dcbafc1e2100e09b6625787783476c4a33f29f077cf43972896\": container with ID starting with df893f2c6caf6dcbafc1e2100e09b6625787783476c4a33f29f077cf43972896 not found: ID does not exist" Oct 30 05:36:37.041510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:37.041479 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v"] Oct 30 05:36:37.055271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:37.055240 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v"] Oct 30 05:36:37.092334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:37.092302 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2"] Oct 30 05:36:37.096568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:37.096539 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2"] Oct 30 05:36:37.583088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:37.583046 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00441e4-210e-48f0-bd77-057db2fe849d" path="/var/lib/kubelet/pods/a00441e4-210e-48f0-bd77-057db2fe849d/volumes" Oct 30 05:36:37.583738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:37.583715 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f557a246-4b9a-4475-b2f1-5b7df159e4ac" path="/var/lib/kubelet/pods/f557a246-4b9a-4475-b2f1-5b7df159e4ac/volumes" Oct 30 05:36:40.035468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:40.035423 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5"] Oct 30 05:36:40.046239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:40.046207 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fccf51-9zfq5"] Oct 30 05:36:41.585431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:41.584651 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc2230a-a2ff-4826-811f-4920130cef0c" path="/var/lib/kubelet/pods/5dc2230a-a2ff-4826-811f-4920130cef0c/volumes" Oct 30 05:36:41.986431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:41.985629 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" Oct 30 05:36:42.029616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:42.029581 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" Oct 30 05:36:42.266483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:42.266391 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq"] Oct 30 05:36:42.271106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:42.271070 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-02cffa-dm2pq"] Oct 30 05:36:43.582462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:43.582426 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0835306-e772-4743-8dae-4e67254a4ff4" path="/var/lib/kubelet/pods/c0835306-e772-4743-8dae-4e67254a4ff4/volumes" Oct 30 05:36:43.765068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:43.765037 6587 scope.go:122] "RemoveContainer" containerID="9876c91ab14fa065bfcc00a3789af4addf5408532105818b3426fbc05be91949" Oct 30 05:36:43.777628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:43.777606 6587 scope.go:122] "RemoveContainer" containerID="41ea2fa53ad884247bc7ce3b2d6455cf0f288113cc749dead2c196e36d4705ed" Oct 30 05:36:43.789285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:43.789251 6587 scope.go:122] "RemoveContainer" containerID="3ce3264c072e25a95bdfcb45c66ed95e16100f317edc26d97599f51cde46770d" Oct 30 05:36:43.800956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:43.800932 6587 scope.go:122] "RemoveContainer" containerID="99fbffb904e126155b5d9cbfab7d40f2ab1d6d78a5ec2da918bafdfb76f2fccc" Oct 30 05:36:43.812063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:43.812042 6587 scope.go:122] "RemoveContainer" containerID="bde25f8dc69a19b73438e9c2272659d950942fed3de7a2c2ed82ab7d39316c78" Oct 30 05:36:43.823996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:43.823963 6587 scope.go:122] "RemoveContainer" containerID="81f6711d3f60e410db01a14a4f0e1adfe75df7b565263b31c6d1c0f4b8851197" Oct 30 05:36:43.863087 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:43.863048 6587 scope.go:122] "RemoveContainer" containerID="1aa5501be286111026ba70fd445789fc353d5152163fd44c3a1e51929702f3fb" Oct 30 05:36:43.874771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:43.874749 6587 scope.go:122] "RemoveContainer" containerID="7ba9f382f55354686026f40d50a8c5b5f0b455c0b92fd640fe31634540d3fadd" Oct 30 05:36:45.963478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:45.963443 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f"] Oct 30 05:36:46.000045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:46.000013 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm"] Oct 30 05:36:50.600458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:50.600423 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz"] Oct 30 05:36:50.600911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:50.600637 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" containerID="cri-o://a31d983a8f270d33976ca394ffa55180771242cbb78e8e543a9390e1a6cf34b8" gracePeriod=30 Oct 30 05:36:50.964919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:50.964830 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" podUID="4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e" containerName="payload-processing" containerID="cri-o://9e81a7c5810b05e439537be5bb091c6c8b4322955df2832ffb01b3f2a8751281" gracePeriod=25 Oct 30 05:36:51.000910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.000867 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" podUID="5d405eaf-ba36-4e25-a606-d94afb1d7ae9" containerName="payload-pre-processing" containerID="cri-o://e97cf30b8f1caf4c86f90b8d00892b8ac848cd9bb8ff430f91c48a110e0ae125" gracePeriod=25 Oct 30 05:36:51.099154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.099120 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6"] Oct 30 05:36:51.099931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.099914 6587 cpu_manager.go:414] "containerMap: removing container" podUID="0bc7fb10-7d71-443c-8251-c714fe13f358" containerName="revoke-keys" Oct 30 05:36:51.099991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.099932 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc7fb10-7d71-443c-8251-c714fe13f358" containerName="revoke-keys" Oct 30 05:36:51.099991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.099945 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f557a246-4b9a-4475-b2f1-5b7df159e4ac" containerName="payload-processing" Oct 30 05:36:51.099991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.099951 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f557a246-4b9a-4475-b2f1-5b7df159e4ac" containerName="payload-processing" Oct 30 05:36:51.100102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.099991 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a00441e4-210e-48f0-bd77-057db2fe849d" containerName="payload-pre-processing" Oct 30 05:36:51.100102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.099997 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00441e4-210e-48f0-bd77-057db2fe849d" containerName="payload-pre-processing" Oct 30 05:36:51.100194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.100110 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00441e4-210e-48f0-bd77-057db2fe849d" containerName="payload-pre-processing" Oct 30 05:36:51.100194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.100125 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc7fb10-7d71-443c-8251-c714fe13f358" containerName="revoke-keys" Oct 30 05:36:51.100194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.100131 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f557a246-4b9a-4475-b2f1-5b7df159e4ac" containerName="payload-processing" Oct 30 05:36:51.104057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.104040 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" Oct 30 05:36:51.128422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.128390 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6"] Oct 30 05:36:51.160418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.159674 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:36:51.160418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.159813 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:36:51.170382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.168801 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e947800-402f-424a-8598-3dd956d7086e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6\" (UID: \"4e947800-402f-424a-8598-3dd956d7086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" Oct 30 05:36:51.170382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.169087 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e947800-402f-424a-8598-3dd956d7086e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6\" (UID: \"4e947800-402f-424a-8598-3dd956d7086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" Oct 30 05:36:51.170382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.169142 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp899\" (UniqueName: \"kubernetes.io/projected/4e947800-402f-424a-8598-3dd956d7086e-kube-api-access-jp899\") pod \"maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6\" (UID: \"4e947800-402f-424a-8598-3dd956d7086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" Oct 30 05:36:51.204435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.203474 6587 generic.go:356] "Generic (PLEG): container finished" podID="4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e" containerID="9e81a7c5810b05e439537be5bb091c6c8b4322955df2832ffb01b3f2a8751281" exitCode=0 Oct 30 05:36:51.204435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.203713 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" event={"ID":"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e","Type":"ContainerDied","Data":"9e81a7c5810b05e439537be5bb091c6c8b4322955df2832ffb01b3f2a8751281"} Oct 30 05:36:51.206959 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.205977 6587 generic.go:356] "Generic (PLEG): container finished" podID="5d405eaf-ba36-4e25-a606-d94afb1d7ae9" containerID="e97cf30b8f1caf4c86f90b8d00892b8ac848cd9bb8ff430f91c48a110e0ae125" exitCode=0 Oct 30 05:36:51.206959 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.206093 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" event={"ID":"5d405eaf-ba36-4e25-a606-d94afb1d7ae9","Type":"ContainerDied","Data":"e97cf30b8f1caf4c86f90b8d00892b8ac848cd9bb8ff430f91c48a110e0ae125"} Oct 30 05:36:51.271193 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.271097 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e947800-402f-424a-8598-3dd956d7086e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6\" (UID: \"4e947800-402f-424a-8598-3dd956d7086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" Oct 30 05:36:51.271193 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.271174 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp899\" (UniqueName: \"kubernetes.io/projected/4e947800-402f-424a-8598-3dd956d7086e-kube-api-access-jp899\") pod \"maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6\" (UID: \"4e947800-402f-424a-8598-3dd956d7086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" Oct 30 05:36:51.271539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.271306 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e947800-402f-424a-8598-3dd956d7086e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6\" (UID: \"4e947800-402f-424a-8598-3dd956d7086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" Oct 30 05:36:51.272283 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.272257 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e947800-402f-424a-8598-3dd956d7086e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6\" (UID: \"4e947800-402f-424a-8598-3dd956d7086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" Oct 30 05:36:51.285648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.283769 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e947800-402f-424a-8598-3dd956d7086e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6\" (UID: \"4e947800-402f-424a-8598-3dd956d7086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" Oct 30 05:36:51.303346 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.303296 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp899\" (UniqueName: \"kubernetes.io/projected/4e947800-402f-424a-8598-3dd956d7086e-kube-api-access-jp899\") pod \"maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6\" (UID: \"4e947800-402f-424a-8598-3dd956d7086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" Oct 30 05:36:51.345522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.345491 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" Oct 30 05:36:51.385336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.385309 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" Oct 30 05:36:51.453469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.453434 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" Oct 30 05:36:51.473790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.473756 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-plugins-config-volume\") pod \"5d405eaf-ba36-4e25-a606-d94afb1d7ae9\" (UID: \"5d405eaf-ba36-4e25-a606-d94afb1d7ae9\") " Oct 30 05:36:51.473975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.473875 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-kube-api-access-clrl7\" (UniqueName: \"kubernetes.io/projected/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-kube-api-access-clrl7\") pod \"5d405eaf-ba36-4e25-a606-d94afb1d7ae9\" (UID: \"5d405eaf-ba36-4e25-a606-d94afb1d7ae9\") " Oct 30 05:36:51.473975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.473921 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-plugins-config-volume\") pod \"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e\" (UID: \"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e\") " Oct 30 05:36:51.473975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.473961 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-kube-api-access-k8j8p\" (UniqueName: \"kubernetes.io/projected/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-kube-api-access-k8j8p\") pod \"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e\" (UID: \"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e\") " Oct 30 05:36:51.475398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.475348 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-plugins-config-volume" pod "4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e" (UID: "4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:51.475496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.475398 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-plugins-config-volume" pod "5d405eaf-ba36-4e25-a606-d94afb1d7ae9" (UID: "5d405eaf-ba36-4e25-a606-d94afb1d7ae9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:51.478419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.478060 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-kube-api-access-clrl7" pod "5d405eaf-ba36-4e25-a606-d94afb1d7ae9" (UID: "5d405eaf-ba36-4e25-a606-d94afb1d7ae9"). InnerVolumeSpecName "kube-api-access-clrl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:51.478947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.478816 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-kube-api-access-k8j8p" pod "4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e" (UID: "4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e"). InnerVolumeSpecName "kube-api-access-k8j8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:51.579348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.578232 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:51.579348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.578269 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8j8p\" (UniqueName: \"kubernetes.io/projected/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e-kube-api-access-k8j8p\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:51.579348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.578283 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:51.579348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.578296 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clrl7\" (UniqueName: \"kubernetes.io/projected/5d405eaf-ba36-4e25-a606-d94afb1d7ae9-kube-api-access-clrl7\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:51.786057 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:36:51.786024 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e947800_402f_424a_8598_3dd956d7086e.slice/crio-e0b4c599cb40df9a005b1586111cbaf689d42857c3bc6f68e3367ac96f6edba6 WatchSource:0}: Error finding container e0b4c599cb40df9a005b1586111cbaf689d42857c3bc6f68e3367ac96f6edba6: Status 404 returned error can't find the container with id e0b4c599cb40df9a005b1586111cbaf689d42857c3bc6f68e3367ac96f6edba6 Oct 30 05:36:51.786498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:51.786205 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6"] Oct 30 05:36:52.216772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:52.216738 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" Oct 30 05:36:52.216963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:52.216737 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f" event={"ID":"4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e","Type":"ContainerDied","Data":"85787079b420ec273eb1e6eed26b591161414bbd478f30b932c890f5fe5d5883"} Oct 30 05:36:52.216963 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:52.216894 6587 scope.go:122] "RemoveContainer" containerID="9e81a7c5810b05e439537be5bb091c6c8b4322955df2832ffb01b3f2a8751281" Oct 30 05:36:52.229625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:52.229586 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" event={"ID":"4e947800-402f-424a-8598-3dd956d7086e","Type":"ContainerStarted","Data":"e0b4c599cb40df9a005b1586111cbaf689d42857c3bc6f68e3367ac96f6edba6"} Oct 30 05:36:52.231992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:52.231961 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" event={"ID":"5d405eaf-ba36-4e25-a606-d94afb1d7ae9","Type":"ContainerDied","Data":"ba007e360d2f3a829c3a53a5cfa41cdb4f8cbebd1cc143ddce37f0331854c041"} Oct 30 05:36:52.232148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:52.232007 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm" Oct 30 05:36:52.247438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:52.247412 6587 scope.go:122] "RemoveContainer" containerID="e97cf30b8f1caf4c86f90b8d00892b8ac848cd9bb8ff430f91c48a110e0ae125" Oct 30 05:36:52.292991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:52.292930 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f"] Oct 30 05:36:52.303341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:52.303311 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f"] Oct 30 05:36:52.346136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:52.346106 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm"] Oct 30 05:36:52.349994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:52.349968 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm"] Oct 30 05:36:53.243479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:53.243391 6587 generic.go:356] "Generic (PLEG): container finished" podID="4e947800-402f-424a-8598-3dd956d7086e" containerID="b7ab73ae8f17fc4d39b7cb872de8a046de4c72fbf9d04d12a3aea10beecc2dc8" exitCode=0 Oct 30 05:36:53.243479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:53.243427 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" event={"ID":"4e947800-402f-424a-8598-3dd956d7086e","Type":"ContainerDied","Data":"b7ab73ae8f17fc4d39b7cb872de8a046de4c72fbf9d04d12a3aea10beecc2dc8"} Oct 30 05:36:53.583582 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:53.583553 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e" path="/var/lib/kubelet/pods/4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e/volumes" Oct 30 05:36:53.584122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:53.584107 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d405eaf-ba36-4e25-a606-d94afb1d7ae9" path="/var/lib/kubelet/pods/5d405eaf-ba36-4e25-a606-d94afb1d7ae9/volumes" Oct 30 05:36:54.467665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:54.467632 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" Oct 30 05:36:54.626987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:54.626934 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e947800-402f-424a-8598-3dd956d7086e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e947800-402f-424a-8598-3dd956d7086e-maas-api-openshift-ca\") pod \"4e947800-402f-424a-8598-3dd956d7086e\" (UID: \"4e947800-402f-424a-8598-3dd956d7086e\") " Oct 30 05:36:54.627205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:54.627055 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e947800-402f-424a-8598-3dd956d7086e-kube-api-access-jp899\" (UniqueName: \"kubernetes.io/projected/4e947800-402f-424a-8598-3dd956d7086e-kube-api-access-jp899\") pod \"4e947800-402f-424a-8598-3dd956d7086e\" (UID: \"4e947800-402f-424a-8598-3dd956d7086e\") " Oct 30 05:36:54.627205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:54.627102 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e947800-402f-424a-8598-3dd956d7086e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e947800-402f-424a-8598-3dd956d7086e-maas-api-xks-ca\") pod \"4e947800-402f-424a-8598-3dd956d7086e\" (UID: \"4e947800-402f-424a-8598-3dd956d7086e\") " Oct 30 05:36:54.627928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:54.627308 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e947800-402f-424a-8598-3dd956d7086e-maas-api-openshift-ca" pod "4e947800-402f-424a-8598-3dd956d7086e" (UID: "4e947800-402f-424a-8598-3dd956d7086e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:54.628080 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:54.628052 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e947800-402f-424a-8598-3dd956d7086e-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:54.630005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:54.629977 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e947800-402f-424a-8598-3dd956d7086e-maas-api-xks-ca" pod "4e947800-402f-424a-8598-3dd956d7086e" (UID: "4e947800-402f-424a-8598-3dd956d7086e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:36:54.630491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:54.630460 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e947800-402f-424a-8598-3dd956d7086e-kube-api-access-jp899" pod "4e947800-402f-424a-8598-3dd956d7086e" (UID: "4e947800-402f-424a-8598-3dd956d7086e"). InnerVolumeSpecName "kube-api-access-jp899". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:54.730133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:54.730086 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp899\" (UniqueName: \"kubernetes.io/projected/4e947800-402f-424a-8598-3dd956d7086e-kube-api-access-jp899\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:54.730133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:54.730130 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e947800-402f-424a-8598-3dd956d7086e-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:55.282944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:55.282894 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" event={"ID":"4e947800-402f-424a-8598-3dd956d7086e","Type":"ContainerDied","Data":"e0b4c599cb40df9a005b1586111cbaf689d42857c3bc6f68e3367ac96f6edba6"} Oct 30 05:36:55.283171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:55.282960 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0b4c599cb40df9a005b1586111cbaf689d42857c3bc6f68e3367ac96f6edba6" Oct 30 05:36:55.283171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:55.283028 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6" Oct 30 05:36:55.914898 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:55.914871 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:36:55.996341 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:55.996303 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:36:56.057079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.057012 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-certs\") pod \"c9207797-cae2-4c27-9e1b-bdce5405161c\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " Oct 30 05:36:56.057760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.057713 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-certs" pod "c9207797-cae2-4c27-9e1b-bdce5405161c" (UID: "c9207797-cae2-4c27-9e1b-bdce5405161c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.058464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.057996 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-podinfo\") pod \"c9207797-cae2-4c27-9e1b-bdce5405161c\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " Oct 30 05:36:56.058464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.058088 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-data\") pod \"c9207797-cae2-4c27-9e1b-bdce5405161c\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " Oct 30 05:36:56.058464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.058148 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-kube-api-access-7p62g\" (UniqueName: \"kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-kube-api-access-7p62g\") pod \"c9207797-cae2-4c27-9e1b-bdce5405161c\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " Oct 30 05:36:56.058464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.058191 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-envoy\") pod \"c9207797-cae2-4c27-9e1b-bdce5405161c\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " Oct 30 05:36:56.061180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.060869 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-data" pod "c9207797-cae2-4c27-9e1b-bdce5405161c" (UID: "c9207797-cae2-4c27-9e1b-bdce5405161c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.062434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.062254 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-credential-socket\") pod \"c9207797-cae2-4c27-9e1b-bdce5405161c\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " Oct 30 05:36:56.063074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.062781 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-token\" (UniqueName: \"kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-token\") pod \"c9207797-cae2-4c27-9e1b-bdce5405161c\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " Oct 30 05:36:56.064004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.063340 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9207797-cae2-4c27-9e1b-bdce5405161c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9207797-cae2-4c27-9e1b-bdce5405161c-istiod-ca-cert\") pod \"c9207797-cae2-4c27-9e1b-bdce5405161c\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " Oct 30 05:36:56.064004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.063412 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-socket\") pod \"c9207797-cae2-4c27-9e1b-bdce5405161c\" (UID: \"c9207797-cae2-4c27-9e1b-bdce5405161c\") " Oct 30 05:36:56.064004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.063805 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9207797-cae2-4c27-9e1b-bdce5405161c-istiod-ca-cert" pod "c9207797-cae2-4c27-9e1b-bdce5405161c" (UID: "c9207797-cae2-4c27-9e1b-bdce5405161c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:36:56.064319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.064293 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-socket" pod "c9207797-cae2-4c27-9e1b-bdce5405161c" (UID: "c9207797-cae2-4c27-9e1b-bdce5405161c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.065463 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.065435 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-credential-socket" pod "c9207797-cae2-4c27-9e1b-bdce5405161c" (UID: "c9207797-cae2-4c27-9e1b-bdce5405161c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.069135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.065753 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.069135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.067236 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.069135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.067255 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9207797-cae2-4c27-9e1b-bdce5405161c-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.069135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.067269 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.069135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.067282 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.085649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.077807 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-podinfo" pod "c9207797-cae2-4c27-9e1b-bdce5405161c" (UID: "c9207797-cae2-4c27-9e1b-bdce5405161c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:36:56.119429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.118459 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-envoy" pod "c9207797-cae2-4c27-9e1b-bdce5405161c" (UID: "c9207797-cae2-4c27-9e1b-bdce5405161c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:56.119957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.119910 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-kube-api-access-7p62g" pod "c9207797-cae2-4c27-9e1b-bdce5405161c" (UID: "c9207797-cae2-4c27-9e1b-bdce5405161c"). InnerVolumeSpecName "kube-api-access-7p62g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:56.121977 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.121835 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-token" pod "c9207797-cae2-4c27-9e1b-bdce5405161c" (UID: "c9207797-cae2-4c27-9e1b-bdce5405161c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:56.169589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.169503 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.169589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.169536 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p62g\" (UniqueName: \"kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-kube-api-access-7p62g\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.169589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.169549 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.169589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.169564 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9207797-cae2-4c27-9e1b-bdce5405161c-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:36:56.296620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.296587 6587 generic.go:356] "Generic (PLEG): container finished" podID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerID="a31d983a8f270d33976ca394ffa55180771242cbb78e8e543a9390e1a6cf34b8" exitCode=0 Oct 30 05:36:56.296816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.296636 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" Oct 30 05:36:56.296816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.296682 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" event={"ID":"c9207797-cae2-4c27-9e1b-bdce5405161c","Type":"ContainerDied","Data":"a31d983a8f270d33976ca394ffa55180771242cbb78e8e543a9390e1a6cf34b8"} Oct 30 05:36:56.296816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.296750 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz" event={"ID":"c9207797-cae2-4c27-9e1b-bdce5405161c","Type":"ContainerDied","Data":"b9993c185b5ad20dc22bdc9809af6b0dd5b71efb072c44fd9ed1e2473ea79cb0"} Oct 30 05:36:56.296816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.296779 6587 scope.go:122] "RemoveContainer" containerID="a31d983a8f270d33976ca394ffa55180771242cbb78e8e543a9390e1a6cf34b8" Oct 30 05:36:56.310774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.310752 6587 scope.go:122] "RemoveContainer" containerID="0ac27b7a3b27da22512bbab85a70a89e84cfc05fd0dbce34b7256a6f4fde8346" Oct 30 05:36:56.325046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.325027 6587 scope.go:122] "RemoveContainer" containerID="a31d983a8f270d33976ca394ffa55180771242cbb78e8e543a9390e1a6cf34b8" Oct 30 05:36:56.325475 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:56.325454 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31d983a8f270d33976ca394ffa55180771242cbb78e8e543a9390e1a6cf34b8\": container with ID starting with a31d983a8f270d33976ca394ffa55180771242cbb78e8e543a9390e1a6cf34b8 not found: ID does not exist" containerID="a31d983a8f270d33976ca394ffa55180771242cbb78e8e543a9390e1a6cf34b8" Oct 30 05:36:56.325555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.325483 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31d983a8f270d33976ca394ffa55180771242cbb78e8e543a9390e1a6cf34b8"} err="failed to get container status \"a31d983a8f270d33976ca394ffa55180771242cbb78e8e543a9390e1a6cf34b8\": rpc error: code = NotFound desc = could not find container \"a31d983a8f270d33976ca394ffa55180771242cbb78e8e543a9390e1a6cf34b8\": container with ID starting with a31d983a8f270d33976ca394ffa55180771242cbb78e8e543a9390e1a6cf34b8 not found: ID does not exist" Oct 30 05:36:56.325555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.325501 6587 scope.go:122] "RemoveContainer" containerID="0ac27b7a3b27da22512bbab85a70a89e84cfc05fd0dbce34b7256a6f4fde8346" Oct 30 05:36:56.325795 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:36:56.325775 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac27b7a3b27da22512bbab85a70a89e84cfc05fd0dbce34b7256a6f4fde8346\": container with ID starting with 0ac27b7a3b27da22512bbab85a70a89e84cfc05fd0dbce34b7256a6f4fde8346 not found: ID does not exist" containerID="0ac27b7a3b27da22512bbab85a70a89e84cfc05fd0dbce34b7256a6f4fde8346" Oct 30 05:36:56.325867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.325801 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac27b7a3b27da22512bbab85a70a89e84cfc05fd0dbce34b7256a6f4fde8346"} err="failed to get container status \"0ac27b7a3b27da22512bbab85a70a89e84cfc05fd0dbce34b7256a6f4fde8346\": rpc error: code = NotFound desc = could not find container \"0ac27b7a3b27da22512bbab85a70a89e84cfc05fd0dbce34b7256a6f4fde8346\": container with ID starting with 0ac27b7a3b27da22512bbab85a70a89e84cfc05fd0dbce34b7256a6f4fde8346 not found: ID does not exist" Oct 30 05:36:56.356077 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.356053 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz"] Oct 30 05:36:56.369651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:56.369623 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz"] Oct 30 05:36:57.583137 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:36:57.583104 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" path="/var/lib/kubelet/pods/c9207797-cae2-4c27-9e1b-bdce5405161c/volumes" Oct 30 05:37:01.892642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:01.892609 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd"] Oct 30 05:37:01.906936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:01.906904 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d55037-d2vsd"] Oct 30 05:37:02.503906 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:02.503871 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7"] Oct 30 05:37:02.516296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:02.516262 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ff4760-wk7z7"] Oct 30 05:37:03.582740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:03.582706 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bc7fb10-7d71-443c-8251-c714fe13f358" path="/var/lib/kubelet/pods/0bc7fb10-7d71-443c-8251-c714fe13f358/volumes" Oct 30 05:37:03.583265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:03.583246 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4d86c6-2546-4520-9550-63c07f6069e6" path="/var/lib/kubelet/pods/4c4d86c6-2546-4520-9550-63c07f6069e6/volumes" Oct 30 05:37:08.941406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:08.941334 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd"] Oct 30 05:37:08.942425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:08.942385 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" podUID="8656d545-bb21-4db3-ade1-1e164a6c9744" containerName="istio-proxy" containerID="cri-o://f3204866f7893e97f91ede12e20418c31b632b3a6b3fd6f74a411b124cbd9f33" gracePeriod=30 Oct 30 05:37:10.026785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.026751 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j"] Oct 30 05:37:10.028222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.027387 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" containerID="cri-o://049b1b26089e05a2bdf2ea9a3f9cb5cb4c8d24be4a26316b3c6b09bf7b7e134a" gracePeriod=30 Oct 30 05:37:10.549567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.549532 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp"] Oct 30 05:37:10.550338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550317 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e" containerName="payload-processing" Oct 30 05:37:10.550417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550341 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e" containerName="payload-processing" Oct 30 05:37:10.550417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550389 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5d405eaf-ba36-4e25-a606-d94afb1d7ae9" containerName="payload-pre-processing" Oct 30 05:37:10.550417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550398 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d405eaf-ba36-4e25-a606-d94afb1d7ae9" containerName="payload-pre-processing" Oct 30 05:37:10.550539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550419 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" Oct 30 05:37:10.550539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550432 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" Oct 30 05:37:10.550539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550462 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" Oct 30 05:37:10.550539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550469 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" Oct 30 05:37:10.550539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550524 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4e947800-402f-424a-8598-3dd956d7086e" containerName="revoke-keys" Oct 30 05:37:10.550539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550532 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e947800-402f-424a-8598-3dd956d7086e" containerName="revoke-keys" Oct 30 05:37:10.550760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550690 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa5dc1f-4f2b-45b9-b74b-d177d8d9050e" containerName="payload-processing" Oct 30 05:37:10.550760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550707 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d405eaf-ba36-4e25-a606-d94afb1d7ae9" containerName="payload-pre-processing" Oct 30 05:37:10.550760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550721 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" Oct 30 05:37:10.550760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.550742 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e947800-402f-424a-8598-3dd956d7086e" containerName="revoke-keys" Oct 30 05:37:10.555202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.555180 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" Oct 30 05:37:10.588475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.588451 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:37:10.588580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.588476 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:37:10.601752 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.601729 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp"] Oct 30 05:37:10.732837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.732799 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp\" (UID: \"d70a7772-401c-41a6-be77-8a354e4c100a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" Oct 30 05:37:10.733050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.733022 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8xl2\" (UniqueName: \"kubernetes.io/projected/d70a7772-401c-41a6-be77-8a354e4c100a-kube-api-access-n8xl2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp\" (UID: \"d70a7772-401c-41a6-be77-8a354e4c100a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" Oct 30 05:37:10.733446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.733421 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp\" (UID: \"d70a7772-401c-41a6-be77-8a354e4c100a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" Oct 30 05:37:10.835293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.835194 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp\" (UID: \"d70a7772-401c-41a6-be77-8a354e4c100a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" Oct 30 05:37:10.835505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.835321 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8xl2\" (UniqueName: \"kubernetes.io/projected/d70a7772-401c-41a6-be77-8a354e4c100a-kube-api-access-n8xl2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp\" (UID: \"d70a7772-401c-41a6-be77-8a354e4c100a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" Oct 30 05:37:10.835505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.835482 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp\" (UID: \"d70a7772-401c-41a6-be77-8a354e4c100a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" Oct 30 05:37:10.836028 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.835993 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp\" (UID: \"d70a7772-401c-41a6-be77-8a354e4c100a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" Oct 30 05:37:10.838574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.838532 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp\" (UID: \"d70a7772-401c-41a6-be77-8a354e4c100a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" Oct 30 05:37:10.862264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.862235 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8xl2\" (UniqueName: \"kubernetes.io/projected/d70a7772-401c-41a6-be77-8a354e4c100a-kube-api-access-n8xl2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp\" (UID: \"d70a7772-401c-41a6-be77-8a354e4c100a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" Oct 30 05:37:10.870035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:10.870013 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" Oct 30 05:37:11.230604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:11.217297 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp"] Oct 30 05:37:11.437698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:11.437657 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" event={"ID":"d70a7772-401c-41a6-be77-8a354e4c100a","Type":"ContainerStarted","Data":"fb225819c54e85225687fbc9c7b8cd420a5e9c86cfee5c57d6033528df220655"} Oct 30 05:37:12.450127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:12.450023 6587 generic.go:356] "Generic (PLEG): container finished" podID="d70a7772-401c-41a6-be77-8a354e4c100a" containerID="77b40e88bc8985e5218ac1fa5dd5d5aee644bc8162e56472d1e0793017fc6157" exitCode=0 Oct 30 05:37:12.450127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:12.450077 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" event={"ID":"d70a7772-401c-41a6-be77-8a354e4c100a","Type":"ContainerDied","Data":"77b40e88bc8985e5218ac1fa5dd5d5aee644bc8162e56472d1e0793017fc6157"} Oct 30 05:37:12.660714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:12.660674 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g"] Oct 30 05:37:12.660940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:12.660912 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" podUID="6151042f-5d54-42f1-9e2c-83278374ee75" containerName="payload-processing" containerID="cri-o://978d4830c72bc84eafd062770c2af373aa1c84f0e7abc73b0028edbf3184983a" gracePeriod=30 Oct 30 05:37:12.687739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:12.687711 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5"] Oct 30 05:37:12.687918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:12.687896 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" podUID="84ba2105-85d3-4bd1-bbe6-07460d9ad35c" containerName="payload-pre-processing" containerID="cri-o://15cbc63f6830985416f8482bdaf4edcbd8dab9fa11e4931734e3540b9593d96f" gracePeriod=30 Oct 30 05:37:13.038672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.038621 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" Oct 30 05:37:13.070205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.070176 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6151042f-5d54-42f1-9e2c-83278374ee75-kube-api-access-5jxh9\" (UniqueName: \"kubernetes.io/projected/6151042f-5d54-42f1-9e2c-83278374ee75-kube-api-access-5jxh9\") pod \"6151042f-5d54-42f1-9e2c-83278374ee75\" (UID: \"6151042f-5d54-42f1-9e2c-83278374ee75\") " Oct 30 05:37:13.070389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.070335 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6151042f-5d54-42f1-9e2c-83278374ee75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6151042f-5d54-42f1-9e2c-83278374ee75-plugins-config-volume\") pod \"6151042f-5d54-42f1-9e2c-83278374ee75\" (UID: \"6151042f-5d54-42f1-9e2c-83278374ee75\") " Oct 30 05:37:13.070774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.070743 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6151042f-5d54-42f1-9e2c-83278374ee75-plugins-config-volume" pod "6151042f-5d54-42f1-9e2c-83278374ee75" (UID: "6151042f-5d54-42f1-9e2c-83278374ee75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:13.070968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.070942 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6151042f-5d54-42f1-9e2c-83278374ee75-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:13.074891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.074860 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6151042f-5d54-42f1-9e2c-83278374ee75-kube-api-access-5jxh9" pod "6151042f-5d54-42f1-9e2c-83278374ee75" (UID: "6151042f-5d54-42f1-9e2c-83278374ee75"). InnerVolumeSpecName "kube-api-access-5jxh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:13.076311 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.076286 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" Oct 30 05:37:13.186379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.176350 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-kube-api-access-9xzfd\" (UniqueName: \"kubernetes.io/projected/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-kube-api-access-9xzfd\") pod \"84ba2105-85d3-4bd1-bbe6-07460d9ad35c\" (UID: \"84ba2105-85d3-4bd1-bbe6-07460d9ad35c\") " Oct 30 05:37:13.186379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.176540 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-plugins-config-volume\") pod \"84ba2105-85d3-4bd1-bbe6-07460d9ad35c\" (UID: \"84ba2105-85d3-4bd1-bbe6-07460d9ad35c\") " Oct 30 05:37:13.186379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.177227 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jxh9\" (UniqueName: \"kubernetes.io/projected/6151042f-5d54-42f1-9e2c-83278374ee75-kube-api-access-5jxh9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:13.293429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.291632 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-plugins-config-volume" pod "84ba2105-85d3-4bd1-bbe6-07460d9ad35c" (UID: "84ba2105-85d3-4bd1-bbe6-07460d9ad35c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:13.293429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.293015 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:13.313380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.303561 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-kube-api-access-9xzfd" pod "84ba2105-85d3-4bd1-bbe6-07460d9ad35c" (UID: "84ba2105-85d3-4bd1-bbe6-07460d9ad35c"). InnerVolumeSpecName "kube-api-access-9xzfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:13.394288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.394253 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xzfd\" (UniqueName: \"kubernetes.io/projected/84ba2105-85d3-4bd1-bbe6-07460d9ad35c-kube-api-access-9xzfd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:13.462205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.462172 6587 generic.go:356] "Generic (PLEG): container finished" podID="84ba2105-85d3-4bd1-bbe6-07460d9ad35c" containerID="15cbc63f6830985416f8482bdaf4edcbd8dab9fa11e4931734e3540b9593d96f" exitCode=0 Oct 30 05:37:13.462677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.462230 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" Oct 30 05:37:13.462677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.462258 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" event={"ID":"84ba2105-85d3-4bd1-bbe6-07460d9ad35c","Type":"ContainerDied","Data":"15cbc63f6830985416f8482bdaf4edcbd8dab9fa11e4931734e3540b9593d96f"} Oct 30 05:37:13.462677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.462317 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5" event={"ID":"84ba2105-85d3-4bd1-bbe6-07460d9ad35c","Type":"ContainerDied","Data":"32023d27966a4746aaa73bc62708a3c76cfd6f688860a6f5f2c5c0378b010654"} Oct 30 05:37:13.462677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.462339 6587 scope.go:122] "RemoveContainer" containerID="15cbc63f6830985416f8482bdaf4edcbd8dab9fa11e4931734e3540b9593d96f" Oct 30 05:37:13.463916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.463895 6587 generic.go:356] "Generic (PLEG): container finished" podID="6151042f-5d54-42f1-9e2c-83278374ee75" containerID="978d4830c72bc84eafd062770c2af373aa1c84f0e7abc73b0028edbf3184983a" exitCode=0 Oct 30 05:37:13.464109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.463938 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" Oct 30 05:37:13.464109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.464048 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" event={"ID":"6151042f-5d54-42f1-9e2c-83278374ee75","Type":"ContainerDied","Data":"978d4830c72bc84eafd062770c2af373aa1c84f0e7abc73b0028edbf3184983a"} Oct 30 05:37:13.464109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.464100 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g" event={"ID":"6151042f-5d54-42f1-9e2c-83278374ee75","Type":"ContainerDied","Data":"8cdad52c7d4aba715da9dd182e7a45dfe549f3968cfa723a86fe91644ee40589"} Oct 30 05:37:13.477182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.477160 6587 scope.go:122] "RemoveContainer" containerID="15cbc63f6830985416f8482bdaf4edcbd8dab9fa11e4931734e3540b9593d96f" Oct 30 05:37:13.477526 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:37:13.477500 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15cbc63f6830985416f8482bdaf4edcbd8dab9fa11e4931734e3540b9593d96f\": container with ID starting with 15cbc63f6830985416f8482bdaf4edcbd8dab9fa11e4931734e3540b9593d96f not found: ID does not exist" containerID="15cbc63f6830985416f8482bdaf4edcbd8dab9fa11e4931734e3540b9593d96f" Oct 30 05:37:13.477651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.477535 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15cbc63f6830985416f8482bdaf4edcbd8dab9fa11e4931734e3540b9593d96f"} err="failed to get container status \"15cbc63f6830985416f8482bdaf4edcbd8dab9fa11e4931734e3540b9593d96f\": rpc error: code = NotFound desc = could not find container \"15cbc63f6830985416f8482bdaf4edcbd8dab9fa11e4931734e3540b9593d96f\": container with ID starting with 15cbc63f6830985416f8482bdaf4edcbd8dab9fa11e4931734e3540b9593d96f not found: ID does not exist" Oct 30 05:37:13.477651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.477558 6587 scope.go:122] "RemoveContainer" containerID="978d4830c72bc84eafd062770c2af373aa1c84f0e7abc73b0028edbf3184983a" Oct 30 05:37:13.488931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.488914 6587 scope.go:122] "RemoveContainer" containerID="978d4830c72bc84eafd062770c2af373aa1c84f0e7abc73b0028edbf3184983a" Oct 30 05:37:13.489213 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:37:13.489195 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978d4830c72bc84eafd062770c2af373aa1c84f0e7abc73b0028edbf3184983a\": container with ID starting with 978d4830c72bc84eafd062770c2af373aa1c84f0e7abc73b0028edbf3184983a not found: ID does not exist" containerID="978d4830c72bc84eafd062770c2af373aa1c84f0e7abc73b0028edbf3184983a" Oct 30 05:37:13.489295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.489221 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978d4830c72bc84eafd062770c2af373aa1c84f0e7abc73b0028edbf3184983a"} err="failed to get container status \"978d4830c72bc84eafd062770c2af373aa1c84f0e7abc73b0028edbf3184983a\": rpc error: code = NotFound desc = could not find container \"978d4830c72bc84eafd062770c2af373aa1c84f0e7abc73b0028edbf3184983a\": container with ID starting with 978d4830c72bc84eafd062770c2af373aa1c84f0e7abc73b0028edbf3184983a not found: ID does not exist" Oct 30 05:37:13.525208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.525171 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5"] Oct 30 05:37:13.528961 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.528935 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5"] Oct 30 05:37:13.574802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.568542 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g"] Oct 30 05:37:13.584897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.584766 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84ba2105-85d3-4bd1-bbe6-07460d9ad35c" path="/var/lib/kubelet/pods/84ba2105-85d3-4bd1-bbe6-07460d9ad35c/volumes" Oct 30 05:37:13.585490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.585465 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g"] Oct 30 05:37:13.605642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.605611 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6"] Oct 30 05:37:13.609461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.609437 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4bc875-qlzw6"] Oct 30 05:37:13.628897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.628874 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" Oct 30 05:37:13.799936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.799905 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70a7772-401c-41a6-be77-8a354e4c100a-kube-api-access-n8xl2\" (UniqueName: \"kubernetes.io/projected/d70a7772-401c-41a6-be77-8a354e4c100a-kube-api-access-n8xl2\") pod \"d70a7772-401c-41a6-be77-8a354e4c100a\" (UID: \"d70a7772-401c-41a6-be77-8a354e4c100a\") " Oct 30 05:37:13.800091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.800077 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-xks-ca\") pod \"d70a7772-401c-41a6-be77-8a354e4c100a\" (UID: \"d70a7772-401c-41a6-be77-8a354e4c100a\") " Oct 30 05:37:13.800244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.800229 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-openshift-ca\") pod \"d70a7772-401c-41a6-be77-8a354e4c100a\" (UID: \"d70a7772-401c-41a6-be77-8a354e4c100a\") " Oct 30 05:37:13.800586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.800560 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-openshift-ca" pod "d70a7772-401c-41a6-be77-8a354e4c100a" (UID: "d70a7772-401c-41a6-be77-8a354e4c100a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:13.800854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.800836 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:13.802036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.802011 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-xks-ca" pod "d70a7772-401c-41a6-be77-8a354e4c100a" (UID: "d70a7772-401c-41a6-be77-8a354e4c100a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:37:13.802197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.802177 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70a7772-401c-41a6-be77-8a354e4c100a-kube-api-access-n8xl2" pod "d70a7772-401c-41a6-be77-8a354e4c100a" (UID: "d70a7772-401c-41a6-be77-8a354e4c100a"). InnerVolumeSpecName "kube-api-access-n8xl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:13.898212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.898137 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:37:13.898212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.898198 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:37:13.902396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.902353 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70a7772-401c-41a6-be77-8a354e4c100a-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:13.902396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:13.902391 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8xl2\" (UniqueName: \"kubernetes.io/projected/d70a7772-401c-41a6-be77-8a354e4c100a-kube-api-access-n8xl2\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:14.234703 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.234677 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 05:37:14.409338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.409304 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-certs\") pod \"8656d545-bb21-4db3-ade1-1e164a6c9744\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " Oct 30 05:37:14.409547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.409492 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-envoy\") pod \"8656d545-bb21-4db3-ade1-1e164a6c9744\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " Oct 30 05:37:14.409643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.409581 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-socket\") pod \"8656d545-bb21-4db3-ade1-1e164a6c9744\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " Oct 30 05:37:14.409643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.409621 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-token\" (UniqueName: \"kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-token\") pod \"8656d545-bb21-4db3-ade1-1e164a6c9744\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " Oct 30 05:37:14.409772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.409658 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-certs" pod "8656d545-bb21-4db3-ade1-1e164a6c9744" (UID: "8656d545-bb21-4db3-ade1-1e164a6c9744"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:14.409772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.409680 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-podinfo\") pod \"8656d545-bb21-4db3-ade1-1e164a6c9744\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " Oct 30 05:37:14.409772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.409752 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-data\") pod \"8656d545-bb21-4db3-ade1-1e164a6c9744\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " Oct 30 05:37:14.409939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.409825 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-credential-socket\") pod \"8656d545-bb21-4db3-ade1-1e164a6c9744\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " Oct 30 05:37:14.409939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.409887 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-kube-api-access-hplw9\" (UniqueName: \"kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-kube-api-access-hplw9\") pod \"8656d545-bb21-4db3-ade1-1e164a6c9744\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " Oct 30 05:37:14.409939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.409929 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8656d545-bb21-4db3-ade1-1e164a6c9744-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8656d545-bb21-4db3-ade1-1e164a6c9744-istiod-ca-cert\") pod \"8656d545-bb21-4db3-ade1-1e164a6c9744\" (UID: \"8656d545-bb21-4db3-ade1-1e164a6c9744\") " Oct 30 05:37:14.410110 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.409935 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-socket" pod "8656d545-bb21-4db3-ade1-1e164a6c9744" (UID: "8656d545-bb21-4db3-ade1-1e164a6c9744"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:14.410177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.410109 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-data" pod "8656d545-bb21-4db3-ade1-1e164a6c9744" (UID: "8656d545-bb21-4db3-ade1-1e164a6c9744"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:14.410177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.410134 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-credential-socket" pod "8656d545-bb21-4db3-ade1-1e164a6c9744" (UID: "8656d545-bb21-4db3-ade1-1e164a6c9744"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:14.410518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.410484 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8656d545-bb21-4db3-ade1-1e164a6c9744-istiod-ca-cert" pod "8656d545-bb21-4db3-ade1-1e164a6c9744" (UID: "8656d545-bb21-4db3-ade1-1e164a6c9744"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:14.410755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.410730 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:14.410818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.410765 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:14.410818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.410780 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:14.410818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.410793 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8656d545-bb21-4db3-ade1-1e164a6c9744-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:14.410818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.410808 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:14.412072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.412048 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-podinfo" pod "8656d545-bb21-4db3-ade1-1e164a6c9744" (UID: "8656d545-bb21-4db3-ade1-1e164a6c9744"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:37:14.412192 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.412166 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-kube-api-access-hplw9" pod "8656d545-bb21-4db3-ade1-1e164a6c9744" (UID: "8656d545-bb21-4db3-ade1-1e164a6c9744"). InnerVolumeSpecName "kube-api-access-hplw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:14.412333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.412315 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-token" pod "8656d545-bb21-4db3-ade1-1e164a6c9744" (UID: "8656d545-bb21-4db3-ade1-1e164a6c9744"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:14.412418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.412337 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-envoy" pod "8656d545-bb21-4db3-ade1-1e164a6c9744" (UID: "8656d545-bb21-4db3-ade1-1e164a6c9744"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:14.476827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.476761 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" Oct 30 05:37:14.476827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.476777 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp" event={"ID":"d70a7772-401c-41a6-be77-8a354e4c100a","Type":"ContainerDied","Data":"fb225819c54e85225687fbc9c7b8cd420a5e9c86cfee5c57d6033528df220655"} Oct 30 05:37:14.476827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.476823 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb225819c54e85225687fbc9c7b8cd420a5e9c86cfee5c57d6033528df220655" Oct 30 05:37:14.479930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.479901 6587 generic.go:356] "Generic (PLEG): container finished" podID="8656d545-bb21-4db3-ade1-1e164a6c9744" containerID="f3204866f7893e97f91ede12e20418c31b632b3a6b3fd6f74a411b124cbd9f33" exitCode=0 Oct 30 05:37:14.480074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.479965 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" event={"ID":"8656d545-bb21-4db3-ade1-1e164a6c9744","Type":"ContainerDied","Data":"f3204866f7893e97f91ede12e20418c31b632b3a6b3fd6f74a411b124cbd9f33"} Oct 30 05:37:14.480074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.479994 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" event={"ID":"8656d545-bb21-4db3-ade1-1e164a6c9744","Type":"ContainerDied","Data":"7433b3e828d3158116484c4c2f0135c2a8df1f83523b3980154b8a22333bb537"} Oct 30 05:37:14.480074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.480020 6587 scope.go:122] "RemoveContainer" containerID="f3204866f7893e97f91ede12e20418c31b632b3a6b3fd6f74a411b124cbd9f33" Oct 30 05:37:14.480260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.480116 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd" Oct 30 05:37:14.495010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.494987 6587 scope.go:122] "RemoveContainer" containerID="f3204866f7893e97f91ede12e20418c31b632b3a6b3fd6f74a411b124cbd9f33" Oct 30 05:37:14.495272 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:37:14.495252 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3204866f7893e97f91ede12e20418c31b632b3a6b3fd6f74a411b124cbd9f33\": container with ID starting with f3204866f7893e97f91ede12e20418c31b632b3a6b3fd6f74a411b124cbd9f33 not found: ID does not exist" containerID="f3204866f7893e97f91ede12e20418c31b632b3a6b3fd6f74a411b124cbd9f33" Oct 30 05:37:14.495402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.495279 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3204866f7893e97f91ede12e20418c31b632b3a6b3fd6f74a411b124cbd9f33"} err="failed to get container status \"f3204866f7893e97f91ede12e20418c31b632b3a6b3fd6f74a411b124cbd9f33\": rpc error: code = NotFound desc = could not find container \"f3204866f7893e97f91ede12e20418c31b632b3a6b3fd6f74a411b124cbd9f33\": container with ID starting with f3204866f7893e97f91ede12e20418c31b632b3a6b3fd6f74a411b124cbd9f33 not found: ID does not exist" Oct 30 05:37:14.512083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.512046 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hplw9\" (UniqueName: \"kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-kube-api-access-hplw9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:14.512083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.512071 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:14.512083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.512083 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:14.512285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.512094 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8656d545-bb21-4db3-ade1-1e164a6c9744-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:14.555708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.555681 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd"] Oct 30 05:37:14.565509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:14.565478 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd"] Oct 30 05:37:15.362373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.362329 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:37:15.491603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.491571 6587 generic.go:356] "Generic (PLEG): container finished" podID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerID="049b1b26089e05a2bdf2ea9a3f9cb5cb4c8d24be4a26316b3c6b09bf7b7e134a" exitCode=0 Oct 30 05:37:15.492085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.491631 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" Oct 30 05:37:15.492085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.491657 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" event={"ID":"fc0a3d86-39e5-4015-baa3-6b09d8713f69","Type":"ContainerDied","Data":"049b1b26089e05a2bdf2ea9a3f9cb5cb4c8d24be4a26316b3c6b09bf7b7e134a"} Oct 30 05:37:15.492085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.491724 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j" event={"ID":"fc0a3d86-39e5-4015-baa3-6b09d8713f69","Type":"ContainerDied","Data":"68848be2d85d3770f59575cc2fffe9c86768bd4037ad01c6435184f77e6f5546"} Oct 30 05:37:15.492085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.491758 6587 scope.go:122] "RemoveContainer" containerID="049b1b26089e05a2bdf2ea9a3f9cb5cb4c8d24be4a26316b3c6b09bf7b7e134a" Oct 30 05:37:15.504277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.504259 6587 scope.go:122] "RemoveContainer" containerID="4da638fc4aa0cb8c8c051537dfbe05629172376895c3f0a3de7307e43948bb47" Oct 30 05:37:15.516178 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.516146 6587 scope.go:122] "RemoveContainer" containerID="049b1b26089e05a2bdf2ea9a3f9cb5cb4c8d24be4a26316b3c6b09bf7b7e134a" Oct 30 05:37:15.516474 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:37:15.516454 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"049b1b26089e05a2bdf2ea9a3f9cb5cb4c8d24be4a26316b3c6b09bf7b7e134a\": container with ID starting with 049b1b26089e05a2bdf2ea9a3f9cb5cb4c8d24be4a26316b3c6b09bf7b7e134a not found: ID does not exist" containerID="049b1b26089e05a2bdf2ea9a3f9cb5cb4c8d24be4a26316b3c6b09bf7b7e134a" Oct 30 05:37:15.516563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.516480 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"049b1b26089e05a2bdf2ea9a3f9cb5cb4c8d24be4a26316b3c6b09bf7b7e134a"} err="failed to get container status \"049b1b26089e05a2bdf2ea9a3f9cb5cb4c8d24be4a26316b3c6b09bf7b7e134a\": rpc error: code = NotFound desc = could not find container \"049b1b26089e05a2bdf2ea9a3f9cb5cb4c8d24be4a26316b3c6b09bf7b7e134a\": container with ID starting with 049b1b26089e05a2bdf2ea9a3f9cb5cb4c8d24be4a26316b3c6b09bf7b7e134a not found: ID does not exist" Oct 30 05:37:15.516563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.516497 6587 scope.go:122] "RemoveContainer" containerID="4da638fc4aa0cb8c8c051537dfbe05629172376895c3f0a3de7307e43948bb47" Oct 30 05:37:15.516775 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:37:15.516740 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da638fc4aa0cb8c8c051537dfbe05629172376895c3f0a3de7307e43948bb47\": container with ID starting with 4da638fc4aa0cb8c8c051537dfbe05629172376895c3f0a3de7307e43948bb47 not found: ID does not exist" containerID="4da638fc4aa0cb8c8c051537dfbe05629172376895c3f0a3de7307e43948bb47" Oct 30 05:37:15.516843 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.516785 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da638fc4aa0cb8c8c051537dfbe05629172376895c3f0a3de7307e43948bb47"} err="failed to get container status \"4da638fc4aa0cb8c8c051537dfbe05629172376895c3f0a3de7307e43948bb47\": rpc error: code = NotFound desc = could not find container \"4da638fc4aa0cb8c8c051537dfbe05629172376895c3f0a3de7307e43948bb47\": container with ID starting with 4da638fc4aa0cb8c8c051537dfbe05629172376895c3f0a3de7307e43948bb47 not found: ID does not exist" Oct 30 05:37:15.527381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.527342 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istiod-ca-cert\") pod \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " Oct 30 05:37:15.527522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.527397 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-kube-api-access-djzbj\" (UniqueName: \"kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-kube-api-access-djzbj\") pod \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " Oct 30 05:37:15.527522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.527466 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-socket\") pod \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " Oct 30 05:37:15.527628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.527583 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-envoy\") pod \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " Oct 30 05:37:15.527680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.527643 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-data\") pod \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " Oct 30 05:37:15.527732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.527700 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-certs\") pod \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " Oct 30 05:37:15.527779 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.527738 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-credential-socket\") pod \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " Oct 30 05:37:15.527833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.527816 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-podinfo\") pod \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " Oct 30 05:37:15.527886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.527836 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istiod-ca-cert" pod "fc0a3d86-39e5-4015-baa3-6b09d8713f69" (UID: "fc0a3d86-39e5-4015-baa3-6b09d8713f69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:15.527886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.527877 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-token\" (UniqueName: \"kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-token\") pod \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\" (UID: \"fc0a3d86-39e5-4015-baa3-6b09d8713f69\") " Oct 30 05:37:15.528353 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.528331 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-certs" pod "fc0a3d86-39e5-4015-baa3-6b09d8713f69" (UID: "fc0a3d86-39e5-4015-baa3-6b09d8713f69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:15.528656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.528630 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-data" pod "fc0a3d86-39e5-4015-baa3-6b09d8713f69" (UID: "fc0a3d86-39e5-4015-baa3-6b09d8713f69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:15.528773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.528645 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:15.528893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.528870 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:15.529477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.529450 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-socket" pod "fc0a3d86-39e5-4015-baa3-6b09d8713f69" (UID: "fc0a3d86-39e5-4015-baa3-6b09d8713f69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:15.530351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.530280 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-credential-socket" pod "fc0a3d86-39e5-4015-baa3-6b09d8713f69" (UID: "fc0a3d86-39e5-4015-baa3-6b09d8713f69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:15.531070 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.531041 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-envoy" pod "fc0a3d86-39e5-4015-baa3-6b09d8713f69" (UID: "fc0a3d86-39e5-4015-baa3-6b09d8713f69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:15.531163 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.531135 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-podinfo" pod "fc0a3d86-39e5-4015-baa3-6b09d8713f69" (UID: "fc0a3d86-39e5-4015-baa3-6b09d8713f69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:37:15.532872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.532849 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-token" pod "fc0a3d86-39e5-4015-baa3-6b09d8713f69" (UID: "fc0a3d86-39e5-4015-baa3-6b09d8713f69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:15.535493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.535469 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-kube-api-access-djzbj" pod "fc0a3d86-39e5-4015-baa3-6b09d8713f69" (UID: "fc0a3d86-39e5-4015-baa3-6b09d8713f69"). InnerVolumeSpecName "kube-api-access-djzbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:15.586508 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.586435 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e947800-402f-424a-8598-3dd956d7086e" path="/var/lib/kubelet/pods/4e947800-402f-424a-8598-3dd956d7086e/volumes" Oct 30 05:37:15.586962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.586939 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6151042f-5d54-42f1-9e2c-83278374ee75" path="/var/lib/kubelet/pods/6151042f-5d54-42f1-9e2c-83278374ee75/volumes" Oct 30 05:37:15.587516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.587495 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8656d545-bb21-4db3-ade1-1e164a6c9744" path="/var/lib/kubelet/pods/8656d545-bb21-4db3-ade1-1e164a6c9744/volumes" Oct 30 05:37:15.634213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.632388 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:15.635851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.635821 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djzbj\" (UniqueName: \"kubernetes.io/projected/fc0a3d86-39e5-4015-baa3-6b09d8713f69-kube-api-access-djzbj\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:15.636021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.635858 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:15.636021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.635875 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:15.636021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.635890 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:15.636021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.635903 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0a3d86-39e5-4015-baa3-6b09d8713f69-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:15.636021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.635915 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc0a3d86-39e5-4015-baa3-6b09d8713f69-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:15.840411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.840316 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j"] Oct 30 05:37:15.846302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:15.846271 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j"] Oct 30 05:37:17.583927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:17.583852 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" path="/var/lib/kubelet/pods/fc0a3d86-39e5-4015-baa3-6b09d8713f69/volumes" Oct 30 05:37:21.295111 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.295076 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd"] Oct 30 05:37:21.296089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296015 6587 cpu_manager.go:414] "containerMap: removing container" podUID="84ba2105-85d3-4bd1-bbe6-07460d9ad35c" containerName="payload-pre-processing" Oct 30 05:37:21.296089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296034 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ba2105-85d3-4bd1-bbe6-07460d9ad35c" containerName="payload-pre-processing" Oct 30 05:37:21.296089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296058 6587 cpu_manager.go:414] "containerMap: removing container" podUID="8656d545-bb21-4db3-ade1-1e164a6c9744" containerName="istio-proxy" Oct 30 05:37:21.296089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296067 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8656d545-bb21-4db3-ade1-1e164a6c9744" containerName="istio-proxy" Oct 30 05:37:21.296089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296087 6587 cpu_manager.go:414] "containerMap: removing container" podUID="6151042f-5d54-42f1-9e2c-83278374ee75" containerName="payload-processing" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296095 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6151042f-5d54-42f1-9e2c-83278374ee75" containerName="payload-processing" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296148 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296157 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296177 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d70a7772-401c-41a6-be77-8a354e4c100a" containerName="revoke-keys" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296183 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70a7772-401c-41a6-be77-8a354e4c100a" containerName="revoke-keys" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296195 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296202 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296317 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70a7772-401c-41a6-be77-8a354e4c100a" containerName="revoke-keys" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296329 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296336 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ba2105-85d3-4bd1-bbe6-07460d9ad35c" containerName="payload-pre-processing" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296344 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="8656d545-bb21-4db3-ade1-1e164a6c9744" containerName="istio-proxy" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296352 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="6151042f-5d54-42f1-9e2c-83278374ee75" containerName="payload-processing" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296387 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9207797-cae2-4c27-9e1b-bdce5405161c" containerName="istio-proxy" Oct 30 05:37:21.296389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.296395 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0a3d86-39e5-4015-baa3-6b09d8713f69" containerName="istio-proxy" Oct 30 05:37:21.301822 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.301798 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" Oct 30 05:37:21.335437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.334665 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd"] Oct 30 05:37:21.346252 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.346209 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c189456-89e2-404d-9ead-952f15064f24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd\" (UID: \"6c189456-89e2-404d-9ead-952f15064f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" Oct 30 05:37:21.346538 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.346498 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c189456-89e2-404d-9ead-952f15064f24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd\" (UID: \"6c189456-89e2-404d-9ead-952f15064f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" Oct 30 05:37:21.346625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.346584 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqlbh\" (UniqueName: \"kubernetes.io/projected/6c189456-89e2-404d-9ead-952f15064f24-kube-api-access-xqlbh\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd\" (UID: \"6c189456-89e2-404d-9ead-952f15064f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" Oct 30 05:37:21.399748 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.399715 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:37:21.399748 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.399733 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:37:21.449671 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.448559 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c189456-89e2-404d-9ead-952f15064f24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd\" (UID: \"6c189456-89e2-404d-9ead-952f15064f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" Oct 30 05:37:21.449671 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.448670 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqlbh\" (UniqueName: \"kubernetes.io/projected/6c189456-89e2-404d-9ead-952f15064f24-kube-api-access-xqlbh\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd\" (UID: \"6c189456-89e2-404d-9ead-952f15064f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" Oct 30 05:37:21.449671 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.448805 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c189456-89e2-404d-9ead-952f15064f24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd\" (UID: \"6c189456-89e2-404d-9ead-952f15064f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" Oct 30 05:37:21.452390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.450927 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c189456-89e2-404d-9ead-952f15064f24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd\" (UID: \"6c189456-89e2-404d-9ead-952f15064f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" Oct 30 05:37:21.456144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.456012 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c189456-89e2-404d-9ead-952f15064f24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd\" (UID: \"6c189456-89e2-404d-9ead-952f15064f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" Oct 30 05:37:21.471959 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.471934 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqlbh\" (UniqueName: \"kubernetes.io/projected/6c189456-89e2-404d-9ead-952f15064f24-kube-api-access-xqlbh\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd\" (UID: \"6c189456-89e2-404d-9ead-952f15064f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" Oct 30 05:37:21.637780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.637735 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" Oct 30 05:37:21.985585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:21.985558 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd"] Oct 30 05:37:21.987275 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:37:21.987236 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c189456_89e2_404d_9ead_952f15064f24.slice/crio-52a52349395d3f768a28d30a2c1e45dc6364b5a84250c1931fbabab3f1567323 WatchSource:0}: Error finding container 52a52349395d3f768a28d30a2c1e45dc6364b5a84250c1931fbabab3f1567323: Status 404 returned error can't find the container with id 52a52349395d3f768a28d30a2c1e45dc6364b5a84250c1931fbabab3f1567323 Oct 30 05:37:22.565080 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:22.565043 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" event={"ID":"6c189456-89e2-404d-9ead-952f15064f24","Type":"ContainerStarted","Data":"55e8a0277be3b5edbc785ed0a740a8be4e15f5e440cf246d22f8a34a4beb0298"} Oct 30 05:37:22.565655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:22.565104 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" event={"ID":"6c189456-89e2-404d-9ead-952f15064f24","Type":"ContainerStarted","Data":"52a52349395d3f768a28d30a2c1e45dc6364b5a84250c1931fbabab3f1567323"} Oct 30 05:37:22.611641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:22.611590 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" podStartSLOduration=1.611576614 podStartE2EDuration="1.611576614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:22.607699841 +0000 UTC m=+3167.135466628" watchObservedRunningTime="2026-10-30 05:37:22.611576614 +0000 UTC m=+3167.139343401" Oct 30 05:37:23.578289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:23.578251 6587 generic.go:356] "Generic (PLEG): container finished" podID="6c189456-89e2-404d-9ead-952f15064f24" containerID="55e8a0277be3b5edbc785ed0a740a8be4e15f5e440cf246d22f8a34a4beb0298" exitCode=0 Oct 30 05:37:23.584141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:23.584109 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" event={"ID":"6c189456-89e2-404d-9ead-952f15064f24","Type":"ContainerDied","Data":"55e8a0277be3b5edbc785ed0a740a8be4e15f5e440cf246d22f8a34a4beb0298"} Oct 30 05:37:24.757826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:24.757800 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" Oct 30 05:37:24.816067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:24.816025 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c189456-89e2-404d-9ead-952f15064f24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c189456-89e2-404d-9ead-952f15064f24-maas-api-xks-ca\") pod \"6c189456-89e2-404d-9ead-952f15064f24\" (UID: \"6c189456-89e2-404d-9ead-952f15064f24\") " Oct 30 05:37:24.816896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:24.816867 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c189456-89e2-404d-9ead-952f15064f24-kube-api-access-xqlbh\" (UniqueName: \"kubernetes.io/projected/6c189456-89e2-404d-9ead-952f15064f24-kube-api-access-xqlbh\") pod \"6c189456-89e2-404d-9ead-952f15064f24\" (UID: \"6c189456-89e2-404d-9ead-952f15064f24\") " Oct 30 05:37:24.817334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:24.817304 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c189456-89e2-404d-9ead-952f15064f24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c189456-89e2-404d-9ead-952f15064f24-maas-api-openshift-ca\") pod \"6c189456-89e2-404d-9ead-952f15064f24\" (UID: \"6c189456-89e2-404d-9ead-952f15064f24\") " Oct 30 05:37:24.817835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:24.817791 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c189456-89e2-404d-9ead-952f15064f24-maas-api-openshift-ca" pod "6c189456-89e2-404d-9ead-952f15064f24" (UID: "6c189456-89e2-404d-9ead-952f15064f24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:24.819020 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:24.818993 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c189456-89e2-404d-9ead-952f15064f24-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:24.819631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:24.819602 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c189456-89e2-404d-9ead-952f15064f24-maas-api-xks-ca" pod "6c189456-89e2-404d-9ead-952f15064f24" (UID: "6c189456-89e2-404d-9ead-952f15064f24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:37:24.820670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:24.820645 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c189456-89e2-404d-9ead-952f15064f24-kube-api-access-xqlbh" pod "6c189456-89e2-404d-9ead-952f15064f24" (UID: "6c189456-89e2-404d-9ead-952f15064f24"). InnerVolumeSpecName "kube-api-access-xqlbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:24.920615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:24.920539 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c189456-89e2-404d-9ead-952f15064f24-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:24.920615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:24.920567 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqlbh\" (UniqueName: \"kubernetes.io/projected/6c189456-89e2-404d-9ead-952f15064f24-kube-api-access-xqlbh\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:25.596549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:25.596521 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" Oct 30 05:37:25.596698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:25.596560 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd" event={"ID":"6c189456-89e2-404d-9ead-952f15064f24","Type":"ContainerDied","Data":"52a52349395d3f768a28d30a2c1e45dc6364b5a84250c1931fbabab3f1567323"} Oct 30 05:37:25.596698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:25.596610 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52a52349395d3f768a28d30a2c1e45dc6364b5a84250c1931fbabab3f1567323" Oct 30 05:37:25.988279 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:37:25.988182 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:37:35.861631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:35.861602 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vqsmz_a8739f61-fe1e-4c58-b535-4815d55e05f1/payload-processing/0.log" Oct 30 05:37:36.095735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:36.095708 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vqsmz_a8739f61-fe1e-4c58-b535-4815d55e05f1/payload-processing/0.log" Oct 30 05:37:37.003440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.003405 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp"] Oct 30 05:37:37.004490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.004419 6587 cpu_manager.go:414] "containerMap: removing container" podUID="6c189456-89e2-404d-9ead-952f15064f24" containerName="revoke-keys" Oct 30 05:37:37.004490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.004445 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c189456-89e2-404d-9ead-952f15064f24" containerName="revoke-keys" Oct 30 05:37:37.004688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.004664 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c189456-89e2-404d-9ead-952f15064f24" containerName="revoke-keys" Oct 30 05:37:37.008837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.008816 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" Oct 30 05:37:37.026780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.026758 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvzlb\" (UniqueName: \"kubernetes.io/projected/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-kube-api-access-rvzlb\") pod \"maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp\" (UID: \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" Oct 30 05:37:37.026901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.026882 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp\" (UID: \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" Oct 30 05:37:37.026959 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.026941 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp\" (UID: \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" Oct 30 05:37:37.033239 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.033212 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp"] Oct 30 05:37:37.041905 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.041878 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:37:37.042020 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.041936 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:37:37.128367 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.128317 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvzlb\" (UniqueName: \"kubernetes.io/projected/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-kube-api-access-rvzlb\") pod \"maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp\" (UID: \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" Oct 30 05:37:37.128553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.128439 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp\" (UID: \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" Oct 30 05:37:37.128553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.128481 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp\" (UID: \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" Oct 30 05:37:37.129245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.129214 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp\" (UID: \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" Oct 30 05:37:37.130996 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.130969 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp\" (UID: \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" Oct 30 05:37:37.151145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.151119 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvzlb\" (UniqueName: \"kubernetes.io/projected/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-kube-api-access-rvzlb\") pod \"maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp\" (UID: \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" Oct 30 05:37:37.323212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.323177 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" Oct 30 05:37:37.644064 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:37:37.644032 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44e16eb_96a0_4a2d_b35b_cd07ad1dded0.slice/crio-f358681f87b47c4a27ba2caf8c7a2da49ccf58dc34446ea210f873896f701623 WatchSource:0}: Error finding container f358681f87b47c4a27ba2caf8c7a2da49ccf58dc34446ea210f873896f701623: Status 404 returned error can't find the container with id f358681f87b47c4a27ba2caf8c7a2da49ccf58dc34446ea210f873896f701623 Oct 30 05:37:37.644309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.644283 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp"] Oct 30 05:37:37.700770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:37.700737 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" event={"ID":"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0","Type":"ContainerStarted","Data":"f358681f87b47c4a27ba2caf8c7a2da49ccf58dc34446ea210f873896f701623"} Oct 30 05:37:38.711767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:38.711731 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" event={"ID":"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0","Type":"ContainerStarted","Data":"25ed54c60798cc140d0ce5e9e8145c8dbdecbd08d128762c7ac2142b2d50b17c"} Oct 30 05:37:38.759831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:38.759778 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" podStartSLOduration=2.759764184 podStartE2EDuration="2.759764184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:38.756404902 +0000 UTC m=+3183.284171691" watchObservedRunningTime="2026-10-30 05:37:38.759764184 +0000 UTC m=+3183.287530998" Oct 30 05:37:39.722985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:39.722949 6587 generic.go:356] "Generic (PLEG): container finished" podID="a44e16eb-96a0-4a2d-b35b-cd07ad1dded0" containerID="25ed54c60798cc140d0ce5e9e8145c8dbdecbd08d128762c7ac2142b2d50b17c" exitCode=0 Oct 30 05:37:39.723502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:39.723037 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" event={"ID":"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0","Type":"ContainerDied","Data":"25ed54c60798cc140d0ce5e9e8145c8dbdecbd08d128762c7ac2142b2d50b17c"} Oct 30 05:37:40.903993 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:40.903965 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" Oct 30 05:37:40.985095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:40.985006 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-xks-ca\") pod \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\" (UID: \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\") " Oct 30 05:37:40.985095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:40.985068 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-kube-api-access-rvzlb\" (UniqueName: \"kubernetes.io/projected/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-kube-api-access-rvzlb\") pod \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\" (UID: \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\") " Oct 30 05:37:40.985371 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:40.985328 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-openshift-ca\") pod \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\" (UID: \"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0\") " Oct 30 05:37:40.985847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:40.985813 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-openshift-ca" pod "a44e16eb-96a0-4a2d-b35b-cd07ad1dded0" (UID: "a44e16eb-96a0-4a2d-b35b-cd07ad1dded0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:40.986058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:40.986032 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:40.987486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:40.987463 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-kube-api-access-rvzlb" pod "a44e16eb-96a0-4a2d-b35b-cd07ad1dded0" (UID: "a44e16eb-96a0-4a2d-b35b-cd07ad1dded0"). InnerVolumeSpecName "kube-api-access-rvzlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:40.987706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:40.987676 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-xks-ca" pod "a44e16eb-96a0-4a2d-b35b-cd07ad1dded0" (UID: "a44e16eb-96a0-4a2d-b35b-cd07ad1dded0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:37:41.087835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:41.087799 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:41.087835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:41.087831 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvzlb\" (UniqueName: \"kubernetes.io/projected/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0-kube-api-access-rvzlb\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:41.744675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:41.744643 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" Oct 30 05:37:41.744675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:41.744661 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp" event={"ID":"a44e16eb-96a0-4a2d-b35b-cd07ad1dded0","Type":"ContainerDied","Data":"f358681f87b47c4a27ba2caf8c7a2da49ccf58dc34446ea210f873896f701623"} Oct 30 05:37:41.744959 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:41.744712 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f358681f87b47c4a27ba2caf8c7a2da49ccf58dc34446ea210f873896f701623" Oct 30 05:37:43.583878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:43.583848 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc"] Oct 30 05:37:43.627423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:43.627389 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb"] Oct 30 05:37:44.051867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:44.051825 6587 scope.go:122] "RemoveContainer" containerID="eae619ff7a871863a55967672b5285626416cfe4618ef24e47ae222636b4ed06" Oct 30 05:37:44.064107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:44.064085 6587 scope.go:122] "RemoveContainer" containerID="768c21873b45826251da87c41e04650ce921d84ec756b14a89ca92a7144bade2" Oct 30 05:37:44.074596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:44.074572 6587 scope.go:122] "RemoveContainer" containerID="dbbaa1c95ec23aad52ed5bd74180cb91c111774b978f1fa7fb83fea1ee69691e" Oct 30 05:37:44.085164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:44.085142 6587 scope.go:122] "RemoveContainer" containerID="34752b18ecbcabbd1d8f01ba9d4bb59e6dc3182bb3b69b6e359f9e6a50591670" Oct 30 05:37:44.097934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:44.097912 6587 scope.go:122] "RemoveContainer" containerID="3b20d53248fb248a52bfac9de09cb6af938cc4c1a5a0f4753a63c11ac8f36272" Oct 30 05:37:44.107995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:44.107974 6587 scope.go:122] "RemoveContainer" containerID="5e9d2f5630bf2f7d42cc61ff547996080f54c5878720b1792188193b044b39d8" Oct 30 05:37:47.954670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:47.954633 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd"] Oct 30 05:37:47.966805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:47.966776 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44bc69-mf4dd"] Oct 30 05:37:48.584964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:48.584917 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" podUID="390e4b2a-b9a8-473c-8338-65eb60160014" containerName="payload-processing" containerID="cri-o://46ae69d8350a8d3ce711ebb6cbdb23ce6d5289ea16c9e75ad9aadbf13f1845d2" gracePeriod=25 Oct 30 05:37:48.628604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:48.628553 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" podUID="21371b98-c41e-4845-9e9b-596a87089643" containerName="payload-pre-processing" containerID="cri-o://1bec9fc7c3ddfb7c733861f387b1e3b57a6ff452be0e59ee154d9fe0b5771452" gracePeriod=25 Oct 30 05:37:48.630913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:48.630886 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp"] Oct 30 05:37:48.636405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:48.636382 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dfb231-78shp"] Oct 30 05:37:48.809548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:48.809335 6587 generic.go:356] "Generic (PLEG): container finished" podID="21371b98-c41e-4845-9e9b-596a87089643" containerID="1bec9fc7c3ddfb7c733861f387b1e3b57a6ff452be0e59ee154d9fe0b5771452" exitCode=0 Oct 30 05:37:48.809759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:48.809468 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" event={"ID":"21371b98-c41e-4845-9e9b-596a87089643","Type":"ContainerDied","Data":"1bec9fc7c3ddfb7c733861f387b1e3b57a6ff452be0e59ee154d9fe0b5771452"} Oct 30 05:37:48.818829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:48.818806 6587 generic.go:356] "Generic (PLEG): container finished" podID="390e4b2a-b9a8-473c-8338-65eb60160014" containerID="46ae69d8350a8d3ce711ebb6cbdb23ce6d5289ea16c9e75ad9aadbf13f1845d2" exitCode=0 Oct 30 05:37:48.819054 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:48.819031 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" event={"ID":"390e4b2a-b9a8-473c-8338-65eb60160014","Type":"ContainerDied","Data":"46ae69d8350a8d3ce711ebb6cbdb23ce6d5289ea16c9e75ad9aadbf13f1845d2"} Oct 30 05:37:48.954375 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:48.951812 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" Oct 30 05:37:48.986387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:48.986350 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" Oct 30 05:37:49.056050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.056010 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21371b98-c41e-4845-9e9b-596a87089643-kube-api-access-pjkb6\" (UniqueName: \"kubernetes.io/projected/21371b98-c41e-4845-9e9b-596a87089643-kube-api-access-pjkb6\") pod \"21371b98-c41e-4845-9e9b-596a87089643\" (UID: \"21371b98-c41e-4845-9e9b-596a87089643\") " Oct 30 05:37:49.056254 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.056150 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390e4b2a-b9a8-473c-8338-65eb60160014-kube-api-access-75nz2\" (UniqueName: \"kubernetes.io/projected/390e4b2a-b9a8-473c-8338-65eb60160014-kube-api-access-75nz2\") pod \"390e4b2a-b9a8-473c-8338-65eb60160014\" (UID: \"390e4b2a-b9a8-473c-8338-65eb60160014\") " Oct 30 05:37:49.056254 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.056205 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390e4b2a-b9a8-473c-8338-65eb60160014-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390e4b2a-b9a8-473c-8338-65eb60160014-plugins-config-volume\") pod \"390e4b2a-b9a8-473c-8338-65eb60160014\" (UID: \"390e4b2a-b9a8-473c-8338-65eb60160014\") " Oct 30 05:37:49.056871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.056842 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390e4b2a-b9a8-473c-8338-65eb60160014-plugins-config-volume" pod "390e4b2a-b9a8-473c-8338-65eb60160014" (UID: "390e4b2a-b9a8-473c-8338-65eb60160014"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:49.059458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.059422 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21371b98-c41e-4845-9e9b-596a87089643-kube-api-access-pjkb6" pod "21371b98-c41e-4845-9e9b-596a87089643" (UID: "21371b98-c41e-4845-9e9b-596a87089643"). InnerVolumeSpecName "kube-api-access-pjkb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:49.062138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.062110 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390e4b2a-b9a8-473c-8338-65eb60160014-kube-api-access-75nz2" pod "390e4b2a-b9a8-473c-8338-65eb60160014" (UID: "390e4b2a-b9a8-473c-8338-65eb60160014"). InnerVolumeSpecName "kube-api-access-75nz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:49.062266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.062248 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21371b98-c41e-4845-9e9b-596a87089643-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21371b98-c41e-4845-9e9b-596a87089643-plugins-config-volume\") pod \"21371b98-c41e-4845-9e9b-596a87089643\" (UID: \"21371b98-c41e-4845-9e9b-596a87089643\") " Oct 30 05:37:49.063045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.063013 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjkb6\" (UniqueName: \"kubernetes.io/projected/21371b98-c41e-4845-9e9b-596a87089643-kube-api-access-pjkb6\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:49.063045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.063047 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75nz2\" (UniqueName: \"kubernetes.io/projected/390e4b2a-b9a8-473c-8338-65eb60160014-kube-api-access-75nz2\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:49.063245 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.063063 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390e4b2a-b9a8-473c-8338-65eb60160014-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:49.063442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.063417 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21371b98-c41e-4845-9e9b-596a87089643-plugins-config-volume" pod "21371b98-c41e-4845-9e9b-596a87089643" (UID: "21371b98-c41e-4845-9e9b-596a87089643"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:49.164873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.164788 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21371b98-c41e-4845-9e9b-596a87089643-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:37:49.583563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.583523 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c189456-89e2-404d-9ead-952f15064f24" path="/var/lib/kubelet/pods/6c189456-89e2-404d-9ead-952f15064f24/volumes" Oct 30 05:37:49.584171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.584151 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70a7772-401c-41a6-be77-8a354e4c100a" path="/var/lib/kubelet/pods/d70a7772-401c-41a6-be77-8a354e4c100a/volumes" Oct 30 05:37:49.831556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.831517 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" event={"ID":"21371b98-c41e-4845-9e9b-596a87089643","Type":"ContainerDied","Data":"638f4161398cada24a0daa9d492a174759e3c350621094d28fd89117243e03f2"} Oct 30 05:37:49.831744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.831628 6587 scope.go:122] "RemoveContainer" containerID="1bec9fc7c3ddfb7c733861f387b1e3b57a6ff452be0e59ee154d9fe0b5771452" Oct 30 05:37:49.832295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.831862 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb" Oct 30 05:37:49.836171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.836077 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" event={"ID":"390e4b2a-b9a8-473c-8338-65eb60160014","Type":"ContainerDied","Data":"4c4b40f0ec8894ccc9f3a95eb33d0b72f7cf896d0287e9f79599211b78e51c73"} Oct 30 05:37:49.836308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.836287 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc" Oct 30 05:37:49.852429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.852407 6587 scope.go:122] "RemoveContainer" containerID="46ae69d8350a8d3ce711ebb6cbdb23ce6d5289ea16c9e75ad9aadbf13f1845d2" Oct 30 05:37:49.896499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.896424 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc"] Oct 30 05:37:49.906602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.906576 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc"] Oct 30 05:37:49.936587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.936550 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb"] Oct 30 05:37:49.940437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:49.940415 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb"] Oct 30 05:37:51.582865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:51.582833 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21371b98-c41e-4845-9e9b-596a87089643" path="/var/lib/kubelet/pods/21371b98-c41e-4845-9e9b-596a87089643/volumes" Oct 30 05:37:51.583345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:51.583328 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390e4b2a-b9a8-473c-8338-65eb60160014" path="/var/lib/kubelet/pods/390e4b2a-b9a8-473c-8338-65eb60160014/volumes" Oct 30 05:37:55.986837 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:37:55.986802 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:37:56.343766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.343730 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn"] Oct 30 05:37:56.344066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.344036 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" podUID="9ecec1c2-d569-4d47-af57-6c4398bd1abb" containerName="istio-proxy" containerID="cri-o://98bfa8c53a9d433f976b4973b3d20e02c13f6c80fbf12e8af057df3b5bcfffde" gracePeriod=30 Oct 30 05:37:56.846309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.846256 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg"] Oct 30 05:37:56.847013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.846992 6587 cpu_manager.go:414] "containerMap: removing container" podUID="21371b98-c41e-4845-9e9b-596a87089643" containerName="payload-pre-processing" Oct 30 05:37:56.847013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.847012 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21371b98-c41e-4845-9e9b-596a87089643" containerName="payload-pre-processing" Oct 30 05:37:56.847217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.847068 6587 cpu_manager.go:414] "containerMap: removing container" podUID="390e4b2a-b9a8-473c-8338-65eb60160014" containerName="payload-processing" Oct 30 05:37:56.847217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.847073 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390e4b2a-b9a8-473c-8338-65eb60160014" containerName="payload-processing" Oct 30 05:37:56.847217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.847085 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a44e16eb-96a0-4a2d-b35b-cd07ad1dded0" containerName="revoke-keys" Oct 30 05:37:56.847217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.847090 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44e16eb-96a0-4a2d-b35b-cd07ad1dded0" containerName="revoke-keys" Oct 30 05:37:56.847478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.847228 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="390e4b2a-b9a8-473c-8338-65eb60160014" containerName="payload-processing" Oct 30 05:37:56.847478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.847238 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44e16eb-96a0-4a2d-b35b-cd07ad1dded0" containerName="revoke-keys" Oct 30 05:37:56.847478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.847251 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="21371b98-c41e-4845-9e9b-596a87089643" containerName="payload-pre-processing" Oct 30 05:37:56.852983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.852959 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" Oct 30 05:37:56.870720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.870692 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg"] Oct 30 05:37:56.900207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.900064 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:37:56.900207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.900195 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:37:56.930285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.930250 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg\" (UID: \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" Oct 30 05:37:56.930451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.930291 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg\" (UID: \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" Oct 30 05:37:56.930527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:56.930516 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp65l\" (UniqueName: \"kubernetes.io/projected/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-kube-api-access-lp65l\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg\" (UID: \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" Oct 30 05:37:57.031687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:57.031652 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp65l\" (UniqueName: \"kubernetes.io/projected/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-kube-api-access-lp65l\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg\" (UID: \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" Oct 30 05:37:57.032093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:57.031842 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg\" (UID: \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" Oct 30 05:37:57.032093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:57.031880 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg\" (UID: \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" Oct 30 05:37:57.032574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:57.032555 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg\" (UID: \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" Oct 30 05:37:57.034392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:57.034369 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg\" (UID: \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" Oct 30 05:37:57.051370 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:57.051330 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp65l\" (UniqueName: \"kubernetes.io/projected/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-kube-api-access-lp65l\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg\" (UID: \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" Oct 30 05:37:57.180434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:57.171804 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" Oct 30 05:37:57.618132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:57.618099 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg"] Oct 30 05:37:57.907721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:57.907632 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" event={"ID":"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f","Type":"ContainerStarted","Data":"15f833608b66895326c5c307ac4a52b0f2cd451c9c09bb99e19ce88ae2e55203"} Oct 30 05:37:58.324649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.324613 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm"] Oct 30 05:37:58.328981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.328962 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" Oct 30 05:37:58.359920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.359891 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm"] Oct 30 05:37:58.459555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.459522 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm5zw\" (UniqueName: \"kubernetes.io/projected/43880e9c-de88-47e4-ae55-b1f1ea8bf329-kube-api-access-qm5zw\") pod \"maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm\" (UID: \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" Oct 30 05:37:58.459747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.459576 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm\" (UID: \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" Oct 30 05:37:58.459817 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.459795 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm\" (UID: \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" Oct 30 05:37:58.561597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.561562 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm\" (UID: \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" Oct 30 05:37:58.561761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.561741 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm\" (UID: \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" Oct 30 05:37:58.561888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.561872 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm5zw\" (UniqueName: \"kubernetes.io/projected/43880e9c-de88-47e4-ae55-b1f1ea8bf329-kube-api-access-qm5zw\") pod \"maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm\" (UID: \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" Oct 30 05:37:58.562302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.562262 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm\" (UID: \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" Oct 30 05:37:58.564246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.564221 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm\" (UID: \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" Oct 30 05:37:58.581891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.581837 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm5zw\" (UniqueName: \"kubernetes.io/projected/43880e9c-de88-47e4-ae55-b1f1ea8bf329-kube-api-access-qm5zw\") pod \"maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm\" (UID: \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" Oct 30 05:37:58.644891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.644859 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" Oct 30 05:37:58.926454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.926320 6587 generic.go:356] "Generic (PLEG): container finished" podID="fb4205bf-0b85-4788-9ee3-4e1867e8ee3f" containerID="254c6480b7931c694268bec3f0f3d2b3b92624f66a634c423cd8e9f5098e0b92" exitCode=0 Oct 30 05:37:58.926640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.926592 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" event={"ID":"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f","Type":"ContainerDied","Data":"254c6480b7931c694268bec3f0f3d2b3b92624f66a634c423cd8e9f5098e0b92"} Oct 30 05:37:58.962289 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:58.962254 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm"] Oct 30 05:37:58.963620 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:37:58.963579 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43880e9c_de88_47e4_ae55_b1f1ea8bf329.slice/crio-ab1a55729d58f07a352aed7b2868045fce2cd8cb19f4f4f400292e7791a44c1a WatchSource:0}: Error finding container ab1a55729d58f07a352aed7b2868045fce2cd8cb19f4f4f400292e7791a44c1a: Status 404 returned error can't find the container with id ab1a55729d58f07a352aed7b2868045fce2cd8cb19f4f4f400292e7791a44c1a Oct 30 05:37:59.936085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:59.936051 6587 generic.go:356] "Generic (PLEG): container finished" podID="43880e9c-de88-47e4-ae55-b1f1ea8bf329" containerID="6b74a2473060c9141f69741c60eafdeec102a705f6869f756b8bd1adab28438f" exitCode=0 Oct 30 05:37:59.936449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:59.936127 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" event={"ID":"43880e9c-de88-47e4-ae55-b1f1ea8bf329","Type":"ContainerDied","Data":"6b74a2473060c9141f69741c60eafdeec102a705f6869f756b8bd1adab28438f"} Oct 30 05:37:59.936449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:37:59.936176 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" event={"ID":"43880e9c-de88-47e4-ae55-b1f1ea8bf329","Type":"ContainerStarted","Data":"ab1a55729d58f07a352aed7b2868045fce2cd8cb19f4f4f400292e7791a44c1a"} Oct 30 05:38:00.132780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:00.132751 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" Oct 30 05:38:00.185665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:00.185629 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-xks-ca\") pod \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\" (UID: \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\") " Oct 30 05:38:00.186144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:00.186125 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-openshift-ca\") pod \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\" (UID: \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\") " Oct 30 05:38:00.186871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:00.186851 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-kube-api-access-lp65l\" (UniqueName: \"kubernetes.io/projected/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-kube-api-access-lp65l\") pod \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\" (UID: \"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f\") " Oct 30 05:38:00.188100 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:00.186671 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-openshift-ca" pod "fb4205bf-0b85-4788-9ee3-4e1867e8ee3f" (UID: "fb4205bf-0b85-4788-9ee3-4e1867e8ee3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:00.188707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:00.188673 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-xks-ca" pod "fb4205bf-0b85-4788-9ee3-4e1867e8ee3f" (UID: "fb4205bf-0b85-4788-9ee3-4e1867e8ee3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:00.195919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:00.195877 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-kube-api-access-lp65l" pod "fb4205bf-0b85-4788-9ee3-4e1867e8ee3f" (UID: "fb4205bf-0b85-4788-9ee3-4e1867e8ee3f"). InnerVolumeSpecName "kube-api-access-lp65l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:00.290027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:00.289987 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:00.290027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:00.290023 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp65l\" (UniqueName: \"kubernetes.io/projected/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-kube-api-access-lp65l\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:00.290027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:00.290034 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:00.945778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:00.945745 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" Oct 30 05:38:00.946228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:00.945786 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg" event={"ID":"fb4205bf-0b85-4788-9ee3-4e1867e8ee3f","Type":"ContainerDied","Data":"15f833608b66895326c5c307ac4a52b0f2cd451c9c09bb99e19ce88ae2e55203"} Oct 30 05:38:00.946228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:00.945843 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15f833608b66895326c5c307ac4a52b0f2cd451c9c09bb99e19ce88ae2e55203" Oct 30 05:38:01.007684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.007455 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:38:01.007684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.007504 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" podUID="9ecec1c2-d569-4d47-af57-6c4398bd1abb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:38:01.136482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.136455 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" Oct 30 05:38:01.202047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.201962 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-openshift-ca\") pod \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\" (UID: \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\") " Oct 30 05:38:01.202047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.202038 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43880e9c-de88-47e4-ae55-b1f1ea8bf329-kube-api-access-qm5zw\" (UniqueName: \"kubernetes.io/projected/43880e9c-de88-47e4-ae55-b1f1ea8bf329-kube-api-access-qm5zw\") pod \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\" (UID: \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\") " Oct 30 05:38:01.202307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.202133 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-xks-ca\") pod \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\" (UID: \"43880e9c-de88-47e4-ae55-b1f1ea8bf329\") " Oct 30 05:38:01.202453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.202424 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-openshift-ca" pod "43880e9c-de88-47e4-ae55-b1f1ea8bf329" (UID: "43880e9c-de88-47e4-ae55-b1f1ea8bf329"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:01.202952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.202927 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.215925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.215891 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43880e9c-de88-47e4-ae55-b1f1ea8bf329-kube-api-access-qm5zw" pod "43880e9c-de88-47e4-ae55-b1f1ea8bf329" (UID: "43880e9c-de88-47e4-ae55-b1f1ea8bf329"). InnerVolumeSpecName "kube-api-access-qm5zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:01.216061 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.216018 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-xks-ca" pod "43880e9c-de88-47e4-ae55-b1f1ea8bf329" (UID: "43880e9c-de88-47e4-ae55-b1f1ea8bf329"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:01.304742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.304701 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43880e9c-de88-47e4-ae55-b1f1ea8bf329-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.304742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.304743 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm5zw\" (UniqueName: \"kubernetes.io/projected/43880e9c-de88-47e4-ae55-b1f1ea8bf329-kube-api-access-qm5zw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.628882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.628855 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:38:01.712451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.712415 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istiod-ca-cert\") pod \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " Oct 30 05:38:01.712677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.712570 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-token\" (UniqueName: \"kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-token\") pod \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " Oct 30 05:38:01.712677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.712656 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-certs\") pod \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " Oct 30 05:38:01.712825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.712740 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-kube-api-access-q6698\" (UniqueName: \"kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-kube-api-access-q6698\") pod \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " Oct 30 05:38:01.712825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.712812 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-credential-socket\") pod \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " Oct 30 05:38:01.712931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.712880 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-socket\") pod \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " Oct 30 05:38:01.712931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.712916 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-data\") pod \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " Oct 30 05:38:01.714349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.712950 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-podinfo\") pod \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " Oct 30 05:38:01.714349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.712989 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-envoy\") pod \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\" (UID: \"9ecec1c2-d569-4d47-af57-6c4398bd1abb\") " Oct 30 05:38:01.714349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.713093 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-certs" pod "9ecec1c2-d569-4d47-af57-6c4398bd1abb" (UID: "9ecec1c2-d569-4d47-af57-6c4398bd1abb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:01.714349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.713294 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-socket" pod "9ecec1c2-d569-4d47-af57-6c4398bd1abb" (UID: "9ecec1c2-d569-4d47-af57-6c4398bd1abb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:01.714349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.713343 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-credential-socket" pod "9ecec1c2-d569-4d47-af57-6c4398bd1abb" (UID: "9ecec1c2-d569-4d47-af57-6c4398bd1abb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:01.714349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.713993 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istiod-ca-cert" pod "9ecec1c2-d569-4d47-af57-6c4398bd1abb" (UID: "9ecec1c2-d569-4d47-af57-6c4398bd1abb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:01.714349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.714085 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.714349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.714105 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.714349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.714118 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.714349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.714133 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-data" pod "9ecec1c2-d569-4d47-af57-6c4398bd1abb" (UID: "9ecec1c2-d569-4d47-af57-6c4398bd1abb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:01.716642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.716616 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-kube-api-access-q6698" pod "9ecec1c2-d569-4d47-af57-6c4398bd1abb" (UID: "9ecec1c2-d569-4d47-af57-6c4398bd1abb"). InnerVolumeSpecName "kube-api-access-q6698". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:01.716740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.716698 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-envoy" pod "9ecec1c2-d569-4d47-af57-6c4398bd1abb" (UID: "9ecec1c2-d569-4d47-af57-6c4398bd1abb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:01.716815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.716798 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-token" pod "9ecec1c2-d569-4d47-af57-6c4398bd1abb" (UID: "9ecec1c2-d569-4d47-af57-6c4398bd1abb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:01.716869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.716825 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-podinfo" pod "9ecec1c2-d569-4d47-af57-6c4398bd1abb" (UID: "9ecec1c2-d569-4d47-af57-6c4398bd1abb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:38:01.835132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.834796 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6698\" (UniqueName: \"kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-kube-api-access-q6698\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.835132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.835044 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.835132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.835061 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.835132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.835074 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.835132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.835089 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.835132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.835101 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ecec1c2-d569-4d47-af57-6c4398bd1abb-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:01.956038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.955998 6587 generic.go:356] "Generic (PLEG): container finished" podID="9ecec1c2-d569-4d47-af57-6c4398bd1abb" containerID="98bfa8c53a9d433f976b4973b3d20e02c13f6c80fbf12e8af057df3b5bcfffde" exitCode=0 Oct 30 05:38:01.956543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.956042 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" Oct 30 05:38:01.956543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.956077 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" event={"ID":"9ecec1c2-d569-4d47-af57-6c4398bd1abb","Type":"ContainerDied","Data":"98bfa8c53a9d433f976b4973b3d20e02c13f6c80fbf12e8af057df3b5bcfffde"} Oct 30 05:38:01.956543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.956131 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn" event={"ID":"9ecec1c2-d569-4d47-af57-6c4398bd1abb","Type":"ContainerDied","Data":"126cf3de91bc3430a9189f9ff433ec14e7cf892fcc429beec439b778b06bcd89"} Oct 30 05:38:01.956543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.956152 6587 scope.go:122] "RemoveContainer" containerID="98bfa8c53a9d433f976b4973b3d20e02c13f6c80fbf12e8af057df3b5bcfffde" Oct 30 05:38:01.957948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.957921 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" Oct 30 05:38:01.958046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.957927 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm" event={"ID":"43880e9c-de88-47e4-ae55-b1f1ea8bf329","Type":"ContainerDied","Data":"ab1a55729d58f07a352aed7b2868045fce2cd8cb19f4f4f400292e7791a44c1a"} Oct 30 05:38:01.958046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.958033 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab1a55729d58f07a352aed7b2868045fce2cd8cb19f4f4f400292e7791a44c1a" Oct 30 05:38:01.969290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.969274 6587 scope.go:122] "RemoveContainer" containerID="98bfa8c53a9d433f976b4973b3d20e02c13f6c80fbf12e8af057df3b5bcfffde" Oct 30 05:38:01.969619 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:01.969600 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98bfa8c53a9d433f976b4973b3d20e02c13f6c80fbf12e8af057df3b5bcfffde\": container with ID starting with 98bfa8c53a9d433f976b4973b3d20e02c13f6c80fbf12e8af057df3b5bcfffde not found: ID does not exist" containerID="98bfa8c53a9d433f976b4973b3d20e02c13f6c80fbf12e8af057df3b5bcfffde" Oct 30 05:38:01.969701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:01.969624 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98bfa8c53a9d433f976b4973b3d20e02c13f6c80fbf12e8af057df3b5bcfffde"} err="failed to get container status \"98bfa8c53a9d433f976b4973b3d20e02c13f6c80fbf12e8af057df3b5bcfffde\": rpc error: code = NotFound desc = could not find container \"98bfa8c53a9d433f976b4973b3d20e02c13f6c80fbf12e8af057df3b5bcfffde\": container with ID starting with 98bfa8c53a9d433f976b4973b3d20e02c13f6c80fbf12e8af057df3b5bcfffde not found: ID does not exist" Oct 30 05:38:02.011453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:02.011426 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn"] Oct 30 05:38:02.023134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:02.023111 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn"] Oct 30 05:38:03.102047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:03.102008 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr"] Oct 30 05:38:03.582512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:03.582477 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ecec1c2-d569-4d47-af57-6c4398bd1abb" path="/var/lib/kubelet/pods/9ecec1c2-d569-4d47-af57-6c4398bd1abb/volumes" Oct 30 05:38:07.396186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:07.396154 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk"] Oct 30 05:38:09.906707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:09.906668 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk"] Oct 30 05:38:09.907413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:09.906931 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" podUID="13378ef4-ea56-4f17-8536-5ace5c724717" containerName="maas-api" containerID="cri-o://51e6a204c79d07b8a56c94c10985e07094685a82d32ea490a1fd416ce840dc6f" gracePeriod=30 Oct 30 05:38:10.059039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.058998 6587 generic.go:356] "Generic (PLEG): container finished" podID="13378ef4-ea56-4f17-8536-5ace5c724717" containerID="51e6a204c79d07b8a56c94c10985e07094685a82d32ea490a1fd416ce840dc6f" exitCode=0 Oct 30 05:38:10.059213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.059065 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" event={"ID":"13378ef4-ea56-4f17-8536-5ace5c724717","Type":"ContainerDied","Data":"51e6a204c79d07b8a56c94c10985e07094685a82d32ea490a1fd416ce840dc6f"} Oct 30 05:38:10.118222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.118188 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8"] Oct 30 05:38:10.118474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.118451 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" podUID="973098f5-b7c1-42dc-934d-4cb248f7695b" containerName="payload-processing" containerID="cri-o://29595456f134e1ef150ac8116a71b4513d1ffd7d15ecfa218d5e65d77aadc849" gracePeriod=30 Oct 30 05:38:10.165957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.165863 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" podUID="973098f5-b7c1-42dc-934d-4cb248f7695b" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.1.127:9004: connect: connection refused" Oct 30 05:38:10.256583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.256555 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:38:10.311373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.311300 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13378ef4-ea56-4f17-8536-5ace5c724717-kube-api-access-hdq6z\" (UniqueName: \"kubernetes.io/projected/13378ef4-ea56-4f17-8536-5ace5c724717-kube-api-access-hdq6z\") pod \"13378ef4-ea56-4f17-8536-5ace5c724717\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " Oct 30 05:38:10.311897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.311825 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-maas-api-tls\") pod \"13378ef4-ea56-4f17-8536-5ace5c724717\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " Oct 30 05:38:10.312136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.312088 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13378ef4-ea56-4f17-8536-5ace5c724717-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13378ef4-ea56-4f17-8536-5ace5c724717-openshift-service-ca\") pod \"13378ef4-ea56-4f17-8536-5ace5c724717\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " Oct 30 05:38:10.312603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.312467 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-metrics-certs\") pod \"13378ef4-ea56-4f17-8536-5ace5c724717\" (UID: \"13378ef4-ea56-4f17-8536-5ace5c724717\") " Oct 30 05:38:10.315270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.313857 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13378ef4-ea56-4f17-8536-5ace5c724717-openshift-service-ca" pod "13378ef4-ea56-4f17-8536-5ace5c724717" (UID: "13378ef4-ea56-4f17-8536-5ace5c724717"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:10.315444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.315071 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-maas-api-tls" pod "13378ef4-ea56-4f17-8536-5ace5c724717" (UID: "13378ef4-ea56-4f17-8536-5ace5c724717"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:10.319761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.319736 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13378ef4-ea56-4f17-8536-5ace5c724717-kube-api-access-hdq6z" pod "13378ef4-ea56-4f17-8536-5ace5c724717" (UID: "13378ef4-ea56-4f17-8536-5ace5c724717"). InnerVolumeSpecName "kube-api-access-hdq6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:10.320849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.320817 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-metrics-certs" pod "13378ef4-ea56-4f17-8536-5ace5c724717" (UID: "13378ef4-ea56-4f17-8536-5ace5c724717"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:10.415911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.415882 6587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-metrics-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:10.415911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.415909 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdq6z\" (UniqueName: \"kubernetes.io/projected/13378ef4-ea56-4f17-8536-5ace5c724717-kube-api-access-hdq6z\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:10.416121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.415918 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13378ef4-ea56-4f17-8536-5ace5c724717-maas-api-tls\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:10.416121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.415929 6587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13378ef4-ea56-4f17-8536-5ace5c724717-openshift-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:10.416526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.416507 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" Oct 30 05:38:10.523889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.523844 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/973098f5-b7c1-42dc-934d-4cb248f7695b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973098f5-b7c1-42dc-934d-4cb248f7695b-plugins-config-volume\") pod \"973098f5-b7c1-42dc-934d-4cb248f7695b\" (UID: \"973098f5-b7c1-42dc-934d-4cb248f7695b\") " Oct 30 05:38:10.524286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.524263 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/973098f5-b7c1-42dc-934d-4cb248f7695b-kube-api-access-hzjdv\" (UniqueName: \"kubernetes.io/projected/973098f5-b7c1-42dc-934d-4cb248f7695b-kube-api-access-hzjdv\") pod \"973098f5-b7c1-42dc-934d-4cb248f7695b\" (UID: \"973098f5-b7c1-42dc-934d-4cb248f7695b\") " Oct 30 05:38:10.545414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.532126 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/973098f5-b7c1-42dc-934d-4cb248f7695b-kube-api-access-hzjdv" pod "973098f5-b7c1-42dc-934d-4cb248f7695b" (UID: "973098f5-b7c1-42dc-934d-4cb248f7695b"). InnerVolumeSpecName "kube-api-access-hzjdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:10.545414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.537452 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/973098f5-b7c1-42dc-934d-4cb248f7695b-plugins-config-volume" pod "973098f5-b7c1-42dc-934d-4cb248f7695b" (UID: "973098f5-b7c1-42dc-934d-4cb248f7695b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:10.627145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.627080 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973098f5-b7c1-42dc-934d-4cb248f7695b-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:10.627341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.627198 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzjdv\" (UniqueName: \"kubernetes.io/projected/973098f5-b7c1-42dc-934d-4cb248f7695b-kube-api-access-hzjdv\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:10.978591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.978564 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp"] Oct 30 05:38:10.989400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:10.989342 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ec1091-s6pdp"] Oct 30 05:38:11.076497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.076461 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" Oct 30 05:38:11.076497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.076484 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk" event={"ID":"13378ef4-ea56-4f17-8536-5ace5c724717","Type":"ContainerDied","Data":"2b7c9bf47dc8d1d9aaa646a1af6aaca80e35d616e0a20ec32ead83016c1f5131"} Oct 30 05:38:11.076742 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.076567 6587 scope.go:122] "RemoveContainer" containerID="51e6a204c79d07b8a56c94c10985e07094685a82d32ea490a1fd416ce840dc6f" Oct 30 05:38:11.079082 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.078740 6587 generic.go:356] "Generic (PLEG): container finished" podID="973098f5-b7c1-42dc-934d-4cb248f7695b" containerID="29595456f134e1ef150ac8116a71b4513d1ffd7d15ecfa218d5e65d77aadc849" exitCode=0 Oct 30 05:38:11.079082 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.078781 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" Oct 30 05:38:11.079082 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.078781 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" event={"ID":"973098f5-b7c1-42dc-934d-4cb248f7695b","Type":"ContainerDied","Data":"29595456f134e1ef150ac8116a71b4513d1ffd7d15ecfa218d5e65d77aadc849"} Oct 30 05:38:11.079082 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.078911 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8" event={"ID":"973098f5-b7c1-42dc-934d-4cb248f7695b","Type":"ContainerDied","Data":"d412c0dc712ae8bfbdd7ac3bb4ed82c4700f9dace26278a5dd6aad58fa7b4128"} Oct 30 05:38:11.090840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.090822 6587 scope.go:122] "RemoveContainer" containerID="29595456f134e1ef150ac8116a71b4513d1ffd7d15ecfa218d5e65d77aadc849" Oct 30 05:38:11.104116 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.104098 6587 scope.go:122] "RemoveContainer" containerID="29595456f134e1ef150ac8116a71b4513d1ffd7d15ecfa218d5e65d77aadc849" Oct 30 05:38:11.104466 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:11.104446 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29595456f134e1ef150ac8116a71b4513d1ffd7d15ecfa218d5e65d77aadc849\": container with ID starting with 29595456f134e1ef150ac8116a71b4513d1ffd7d15ecfa218d5e65d77aadc849 not found: ID does not exist" containerID="29595456f134e1ef150ac8116a71b4513d1ffd7d15ecfa218d5e65d77aadc849" Oct 30 05:38:11.104535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.104475 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29595456f134e1ef150ac8116a71b4513d1ffd7d15ecfa218d5e65d77aadc849"} err="failed to get container status \"29595456f134e1ef150ac8116a71b4513d1ffd7d15ecfa218d5e65d77aadc849\": rpc error: code = NotFound desc = could not find container \"29595456f134e1ef150ac8116a71b4513d1ffd7d15ecfa218d5e65d77aadc849\": container with ID starting with 29595456f134e1ef150ac8116a71b4513d1ffd7d15ecfa218d5e65d77aadc849 not found: ID does not exist" Oct 30 05:38:11.132284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.132254 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk"] Oct 30 05:38:11.136944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.136922 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ec1091-59464c5566-lvxlk"] Oct 30 05:38:11.191547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.191494 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8"] Oct 30 05:38:11.194933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.194908 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8"] Oct 30 05:38:11.582641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.582609 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13378ef4-ea56-4f17-8536-5ace5c724717" path="/var/lib/kubelet/pods/13378ef4-ea56-4f17-8536-5ace5c724717/volumes" Oct 30 05:38:11.583069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.583055 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="973098f5-b7c1-42dc-934d-4cb248f7695b" path="/var/lib/kubelet/pods/973098f5-b7c1-42dc-934d-4cb248f7695b/volumes" Oct 30 05:38:11.583481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:11.583467 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44e16eb-96a0-4a2d-b35b-cd07ad1dded0" path="/var/lib/kubelet/pods/a44e16eb-96a0-4a2d-b35b-cd07ad1dded0/volumes" Oct 30 05:38:12.396848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:12.396798 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" podUID="25ba8b43-1249-48f5-91b1-25bdb0f82dd5" containerName="payload-pre-processing" containerID="cri-o://725ba193af45deb48e58b9bb1195662d681c0cb43f781c3320077e60eebde23e" gracePeriod=25 Oct 30 05:38:14.628901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:14.628823 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" podUID="25ba8b43-1249-48f5-91b1-25bdb0f82dd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:38:17.035447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:17.035411 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6"] Oct 30 05:38:17.047087 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:17.047056 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5873f3d-g75f6"] Oct 30 05:38:17.583032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:17.582996 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126f3871-104a-4cd8-99a2-00e114b38ed5" path="/var/lib/kubelet/pods/126f3871-104a-4cd8-99a2-00e114b38ed5/volumes" Oct 30 05:38:18.125209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.125164 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" containerID="cri-o://b386800447a4fb9b835031d7305f652ffd41e2d7d12c4d4cbaa642dc8304df50" gracePeriod=45 Oct 30 05:38:18.338482 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.338445 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq"] Oct 30 05:38:18.339049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.339017 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerName="istio-proxy" containerID="cri-o://26120fe2eb888a7e7ba3da4f8dcec046228edd7dc692d46290c2fa82779a3335" gracePeriod=30 Oct 30 05:38:18.463571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.463537 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:38:18.567389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.564464 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3758b02b-300e-4659-a74e-f96bc8967080-tls-certs\" (UniqueName: \"kubernetes.io/secret/3758b02b-300e-4659-a74e-f96bc8967080-tls-certs\") pod \"3758b02b-300e-4659-a74e-f96bc8967080\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " Oct 30 05:38:18.567389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.564592 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-model-cache\") pod \"3758b02b-300e-4659-a74e-f96bc8967080\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " Oct 30 05:38:18.567389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.564640 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-home\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-home\") pod \"3758b02b-300e-4659-a74e-f96bc8967080\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " Oct 30 05:38:18.567389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.564726 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-dshm\") pod \"3758b02b-300e-4659-a74e-f96bc8967080\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " Oct 30 05:38:18.567389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.564865 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3758b02b-300e-4659-a74e-f96bc8967080-kube-api-access-tpx9c\" (UniqueName: \"kubernetes.io/projected/3758b02b-300e-4659-a74e-f96bc8967080-kube-api-access-tpx9c\") pod \"3758b02b-300e-4659-a74e-f96bc8967080\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " Oct 30 05:38:18.567389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.564937 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-tmp-dir\") pod \"3758b02b-300e-4659-a74e-f96bc8967080\" (UID: \"3758b02b-300e-4659-a74e-f96bc8967080\") " Oct 30 05:38:18.570381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.568697 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-home" pod "3758b02b-300e-4659-a74e-f96bc8967080" (UID: "3758b02b-300e-4659-a74e-f96bc8967080"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:18.570381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.569242 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-tmp-dir" pod "3758b02b-300e-4659-a74e-f96bc8967080" (UID: "3758b02b-300e-4659-a74e-f96bc8967080"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:18.576964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.571079 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-model-cache" pod "3758b02b-300e-4659-a74e-f96bc8967080" (UID: "3758b02b-300e-4659-a74e-f96bc8967080"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:18.621469 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.621425 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3758b02b-300e-4659-a74e-f96bc8967080-tls-certs" pod "3758b02b-300e-4659-a74e-f96bc8967080" (UID: "3758b02b-300e-4659-a74e-f96bc8967080"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:18.621816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.621789 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3758b02b-300e-4659-a74e-f96bc8967080-kube-api-access-tpx9c" pod "3758b02b-300e-4659-a74e-f96bc8967080" (UID: "3758b02b-300e-4659-a74e-f96bc8967080"). InnerVolumeSpecName "kube-api-access-tpx9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:18.622849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.622824 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-dshm" pod "3758b02b-300e-4659-a74e-f96bc8967080" (UID: "3758b02b-300e-4659-a74e-f96bc8967080"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:18.667291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.667212 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpx9c\" (UniqueName: \"kubernetes.io/projected/3758b02b-300e-4659-a74e-f96bc8967080-kube-api-access-tpx9c\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:18.667291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.667250 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:18.667291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.667266 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3758b02b-300e-4659-a74e-f96bc8967080-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:18.667291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.667278 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:18.667291 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.667290 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:18.667585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.667302 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3758b02b-300e-4659-a74e-f96bc8967080-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:18.898766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.898732 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p"] Oct 30 05:38:18.899512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899492 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" Oct 30 05:38:18.899512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899512 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" Oct 30 05:38:18.899708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899540 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fb4205bf-0b85-4788-9ee3-4e1867e8ee3f" containerName="revoke-keys" Oct 30 05:38:18.899708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899547 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb4205bf-0b85-4788-9ee3-4e1867e8ee3f" containerName="revoke-keys" Oct 30 05:38:18.899708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899571 6587 cpu_manager.go:414] "containerMap: removing container" podUID="43880e9c-de88-47e4-ae55-b1f1ea8bf329" containerName="revoke-keys" Oct 30 05:38:18.899708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899579 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43880e9c-de88-47e4-ae55-b1f1ea8bf329" containerName="revoke-keys" Oct 30 05:38:18.899708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899590 6587 cpu_manager.go:414] "containerMap: removing container" podUID="973098f5-b7c1-42dc-934d-4cb248f7695b" containerName="payload-processing" Oct 30 05:38:18.899708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899597 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="973098f5-b7c1-42dc-934d-4cb248f7695b" containerName="payload-processing" Oct 30 05:38:18.899708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899638 6587 cpu_manager.go:414] "containerMap: removing container" podUID="13378ef4-ea56-4f17-8536-5ace5c724717" containerName="maas-api" Oct 30 05:38:18.899708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899645 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13378ef4-ea56-4f17-8536-5ace5c724717" containerName="maas-api" Oct 30 05:38:18.899708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899664 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9ecec1c2-d569-4d47-af57-6c4398bd1abb" containerName="istio-proxy" Oct 30 05:38:18.899708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899671 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ecec1c2-d569-4d47-af57-6c4398bd1abb" containerName="istio-proxy" Oct 30 05:38:18.900231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899834 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="13378ef4-ea56-4f17-8536-5ace5c724717" containerName="maas-api" Oct 30 05:38:18.900231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899849 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ecec1c2-d569-4d47-af57-6c4398bd1abb" containerName="istio-proxy" Oct 30 05:38:18.900231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899865 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" Oct 30 05:38:18.900231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899879 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb4205bf-0b85-4788-9ee3-4e1867e8ee3f" containerName="revoke-keys" Oct 30 05:38:18.900231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899893 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="43880e9c-de88-47e4-ae55-b1f1ea8bf329" containerName="revoke-keys" Oct 30 05:38:18.900231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.899903 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="973098f5-b7c1-42dc-934d-4cb248f7695b" containerName="payload-processing" Oct 30 05:38:18.903880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.903860 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" Oct 30 05:38:18.927798 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.927725 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx"] Oct 30 05:38:18.928086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.928054 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" podUID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" containerName="istio-proxy" containerID="cri-o://2216adce2e21d9d9960b2e0982a6a89b3e6d2f41c0bd1ed8426c5658cf2f078d" gracePeriod=30 Oct 30 05:38:18.928199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.928125 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:38:18.928199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.928179 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:38:18.931749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:18.931717 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p"] Oct 30 05:38:19.072888 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.072853 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p\" (UID: \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" Oct 30 05:38:19.073125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.073091 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd566\" (UniqueName: \"kubernetes.io/projected/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-kube-api-access-kd566\") pod \"maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p\" (UID: \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" Oct 30 05:38:19.073206 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.073151 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p\" (UID: \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" Oct 30 05:38:19.157413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.157351 6587 generic.go:356] "Generic (PLEG): container finished" podID="3758b02b-300e-4659-a74e-f96bc8967080" containerID="b386800447a4fb9b835031d7305f652ffd41e2d7d12c4d4cbaa642dc8304df50" exitCode=0 Oct 30 05:38:19.157849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.157427 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" Oct 30 05:38:19.157849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.157443 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" event={"ID":"3758b02b-300e-4659-a74e-f96bc8967080","Type":"ContainerDied","Data":"b386800447a4fb9b835031d7305f652ffd41e2d7d12c4d4cbaa642dc8304df50"} Oct 30 05:38:19.157849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.157492 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" event={"ID":"3758b02b-300e-4659-a74e-f96bc8967080","Type":"ContainerDied","Data":"f6ecff9b44d52190920c76af403207954a191d9ba26b94abe0f1c7de3d8a03b2"} Oct 30 05:38:19.157849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.157513 6587 scope.go:122] "RemoveContainer" containerID="b386800447a4fb9b835031d7305f652ffd41e2d7d12c4d4cbaa642dc8304df50" Oct 30 05:38:19.171376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.169097 6587 scope.go:122] "RemoveContainer" containerID="b386800447a4fb9b835031d7305f652ffd41e2d7d12c4d4cbaa642dc8304df50" Oct 30 05:38:19.171376 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:19.169432 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b386800447a4fb9b835031d7305f652ffd41e2d7d12c4d4cbaa642dc8304df50\": container with ID starting with b386800447a4fb9b835031d7305f652ffd41e2d7d12c4d4cbaa642dc8304df50 not found: ID does not exist" containerID="b386800447a4fb9b835031d7305f652ffd41e2d7d12c4d4cbaa642dc8304df50" Oct 30 05:38:19.171376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.169464 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b386800447a4fb9b835031d7305f652ffd41e2d7d12c4d4cbaa642dc8304df50"} err="failed to get container status \"b386800447a4fb9b835031d7305f652ffd41e2d7d12c4d4cbaa642dc8304df50\": rpc error: code = NotFound desc = could not find container \"b386800447a4fb9b835031d7305f652ffd41e2d7d12c4d4cbaa642dc8304df50\": container with ID starting with b386800447a4fb9b835031d7305f652ffd41e2d7d12c4d4cbaa642dc8304df50 not found: ID does not exist" Oct 30 05:38:19.174966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.174940 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p\" (UID: \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" Oct 30 05:38:19.175072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.175057 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd566\" (UniqueName: \"kubernetes.io/projected/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-kube-api-access-kd566\") pod \"maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p\" (UID: \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" Oct 30 05:38:19.175136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.175084 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p\" (UID: \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" Oct 30 05:38:19.175668 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.175649 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p\" (UID: \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" Oct 30 05:38:19.177542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.177523 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p\" (UID: \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" Oct 30 05:38:19.195121 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.195066 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd566\" (UniqueName: \"kubernetes.io/projected/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-kube-api-access-kd566\") pod \"maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p\" (UID: \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" Oct 30 05:38:19.214730 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.214707 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr"] Oct 30 05:38:19.218696 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.218674 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr"] Oct 30 05:38:19.218811 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.218739 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" Oct 30 05:38:19.383331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.383281 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-2-w0-b21507-kserve-7b5cdb4f5c-zsglr" podUID="3758b02b-300e-4659-a74e-f96bc8967080" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:38:19.560624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.560594 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p"] Oct 30 05:38:19.561415 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:38:19.561347 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba66369_7e9c_46a7_ba63_2d08bc3187c5.slice/crio-62241df8667cf06ae1c1e00d8dc4480b065a16dde12834a4129f5beacfb5349f WatchSource:0}: Error finding container 62241df8667cf06ae1c1e00d8dc4480b065a16dde12834a4129f5beacfb5349f: Status 404 returned error can't find the container with id 62241df8667cf06ae1c1e00d8dc4480b065a16dde12834a4129f5beacfb5349f Oct 30 05:38:19.582456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.582432 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3758b02b-300e-4659-a74e-f96bc8967080" path="/var/lib/kubelet/pods/3758b02b-300e-4659-a74e-f96bc8967080/volumes" Oct 30 05:38:19.885956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.885925 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:38:19.886087 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:19.885976 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:38:20.166497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:20.166388 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" event={"ID":"7ba66369-7e9c-46a7-ba63-2d08bc3187c5","Type":"ContainerStarted","Data":"47bac404da756ec776adb9993488d446ed2e11f453a92eca6bb8b5f4c2efa29d"} Oct 30 05:38:20.166497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:20.166458 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" event={"ID":"7ba66369-7e9c-46a7-ba63-2d08bc3187c5","Type":"ContainerStarted","Data":"62241df8667cf06ae1c1e00d8dc4480b065a16dde12834a4129f5beacfb5349f"} Oct 30 05:38:20.209536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:20.209473 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" podStartSLOduration=2.209457547 podStartE2EDuration="2.209457547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:20.208149062 +0000 UTC m=+3224.735915849" watchObservedRunningTime="2026-10-30 05:38:20.209457547 +0000 UTC m=+3224.737224335" Oct 30 05:38:20.783314 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:20.783226 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw"] Oct 30 05:38:20.783523 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:20.783445 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" podUID="664e59b0-fa1f-498c-960e-94457fdcf3f1" containerName="istio-proxy" containerID="cri-o://3fff937c00339736e5c0c4041c070b7f4c47d1f1febabe285f6c853c02bbb5c7" gracePeriod=30 Oct 30 05:38:20.796513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:20.796480 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh"] Oct 30 05:38:20.796789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:20.796731 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" podUID="5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" containerName="istio-proxy" containerID="cri-o://954b2850487489ca5c770cb0efd8c655bc6fb5f36b7d9b6e149d5c95f3ecd6b5" gracePeriod=30 Oct 30 05:38:20.802493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:20.802461 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf"] Oct 30 05:38:20.802793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:20.802719 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" podUID="54424663-3cda-49f5-a298-88beadbf9343" containerName="istio-proxy" containerID="cri-o://da1715bd43f17a5c90bba01971076e9c84130f2c0847739b93da8bbafb6bd3a9" gracePeriod=30 Oct 30 05:38:21.180398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:21.180334 6587 generic.go:356] "Generic (PLEG): container finished" podID="7ba66369-7e9c-46a7-ba63-2d08bc3187c5" containerID="47bac404da756ec776adb9993488d446ed2e11f453a92eca6bb8b5f4c2efa29d" exitCode=0 Oct 30 05:38:21.180819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:21.180425 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" event={"ID":"7ba66369-7e9c-46a7-ba63-2d08bc3187c5","Type":"ContainerDied","Data":"47bac404da756ec776adb9993488d446ed2e11f453a92eca6bb8b5f4c2efa29d"} Oct 30 05:38:22.356202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:22.356174 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" Oct 30 05:38:22.427599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:22.427569 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-xks-ca\") pod \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\" (UID: \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\") " Oct 30 05:38:22.427791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:22.427695 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-openshift-ca\") pod \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\" (UID: \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\") " Oct 30 05:38:22.427870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:22.427843 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-kube-api-access-kd566\" (UniqueName: \"kubernetes.io/projected/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-kube-api-access-kd566\") pod \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\" (UID: \"7ba66369-7e9c-46a7-ba63-2d08bc3187c5\") " Oct 30 05:38:22.428054 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:22.428025 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-openshift-ca" pod "7ba66369-7e9c-46a7-ba63-2d08bc3187c5" (UID: "7ba66369-7e9c-46a7-ba63-2d08bc3187c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:22.428506 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:22.428481 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:22.430321 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:22.430294 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-xks-ca" pod "7ba66369-7e9c-46a7-ba63-2d08bc3187c5" (UID: "7ba66369-7e9c-46a7-ba63-2d08bc3187c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:22.430766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:22.430740 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-kube-api-access-kd566" pod "7ba66369-7e9c-46a7-ba63-2d08bc3187c5" (UID: "7ba66369-7e9c-46a7-ba63-2d08bc3187c5"). InnerVolumeSpecName "kube-api-access-kd566". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:22.529606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:22.529565 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:22.529606 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:22.529602 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd566\" (UniqueName: \"kubernetes.io/projected/7ba66369-7e9c-46a7-ba63-2d08bc3187c5-kube-api-access-kd566\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:23.199841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.199753 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" event={"ID":"7ba66369-7e9c-46a7-ba63-2d08bc3187c5","Type":"ContainerDied","Data":"62241df8667cf06ae1c1e00d8dc4480b065a16dde12834a4129f5beacfb5349f"} Oct 30 05:38:23.199841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.199807 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p" Oct 30 05:38:23.199841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.199815 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62241df8667cf06ae1c1e00d8dc4480b065a16dde12834a4129f5beacfb5349f" Oct 30 05:38:23.344280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.344244 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:38:23.344489 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.344303 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" podUID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:38:23.654292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.654263 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:38:23.747029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.746941 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-credential-socket\") pod \"c5530d72-64d2-48b1-9f51-362f4a41b916\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " Oct 30 05:38:23.747218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.747077 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-data\") pod \"c5530d72-64d2-48b1-9f51-362f4a41b916\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " Oct 30 05:38:23.747218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.747115 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-envoy\") pod \"c5530d72-64d2-48b1-9f51-362f4a41b916\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " Oct 30 05:38:23.747372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.747276 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-credential-socket" pod "c5530d72-64d2-48b1-9f51-362f4a41b916" (UID: "c5530d72-64d2-48b1-9f51-362f4a41b916"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:23.747372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.747283 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5530d72-64d2-48b1-9f51-362f4a41b916-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5530d72-64d2-48b1-9f51-362f4a41b916-istiod-ca-cert\") pod \"c5530d72-64d2-48b1-9f51-362f4a41b916\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " Oct 30 05:38:23.747511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.747398 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-kube-api-access-pwzpx\" (UniqueName: \"kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-kube-api-access-pwzpx\") pod \"c5530d72-64d2-48b1-9f51-362f4a41b916\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " Oct 30 05:38:23.747511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.747450 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-podinfo\") pod \"c5530d72-64d2-48b1-9f51-362f4a41b916\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " Oct 30 05:38:23.747634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.747600 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-socket\") pod \"c5530d72-64d2-48b1-9f51-362f4a41b916\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " Oct 30 05:38:23.747694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.747634 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-certs\") pod \"c5530d72-64d2-48b1-9f51-362f4a41b916\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " Oct 30 05:38:23.747694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.747665 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5530d72-64d2-48b1-9f51-362f4a41b916-istiod-ca-cert" pod "c5530d72-64d2-48b1-9f51-362f4a41b916" (UID: "c5530d72-64d2-48b1-9f51-362f4a41b916"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:23.747694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.747678 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-token\" (UniqueName: \"kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-token\") pod \"c5530d72-64d2-48b1-9f51-362f4a41b916\" (UID: \"c5530d72-64d2-48b1-9f51-362f4a41b916\") " Oct 30 05:38:23.748201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.747908 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-data" pod "c5530d72-64d2-48b1-9f51-362f4a41b916" (UID: "c5530d72-64d2-48b1-9f51-362f4a41b916"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:23.748837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.748497 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5530d72-64d2-48b1-9f51-362f4a41b916-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:23.748837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.748524 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:23.748837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.748537 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:23.750109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.749937 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-socket" pod "c5530d72-64d2-48b1-9f51-362f4a41b916" (UID: "c5530d72-64d2-48b1-9f51-362f4a41b916"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:23.750320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.750271 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-certs" pod "c5530d72-64d2-48b1-9f51-362f4a41b916" (UID: "c5530d72-64d2-48b1-9f51-362f4a41b916"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:23.752483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.752457 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-envoy" pod "c5530d72-64d2-48b1-9f51-362f4a41b916" (UID: "c5530d72-64d2-48b1-9f51-362f4a41b916"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:23.752915 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.752882 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-token" pod "c5530d72-64d2-48b1-9f51-362f4a41b916" (UID: "c5530d72-64d2-48b1-9f51-362f4a41b916"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:23.753196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.753162 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-kube-api-access-pwzpx" pod "c5530d72-64d2-48b1-9f51-362f4a41b916" (UID: "c5530d72-64d2-48b1-9f51-362f4a41b916"). InnerVolumeSpecName "kube-api-access-pwzpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:23.753648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.753614 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-podinfo" pod "c5530d72-64d2-48b1-9f51-362f4a41b916" (UID: "c5530d72-64d2-48b1-9f51-362f4a41b916"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:38:23.850484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.850453 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwzpx\" (UniqueName: \"kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-kube-api-access-pwzpx\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:23.850484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.850482 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:23.850484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.850492 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:23.850739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.850502 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:23.850739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.850510 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:23.850739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:23.850518 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5530d72-64d2-48b1-9f51-362f4a41b916-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:24.037965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.037874 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc"] Oct 30 05:38:24.041420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.041389 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc91e775-2tffc"] Oct 30 05:38:24.207675 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.207649 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:38:24.211260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.211231 6587 generic.go:356] "Generic (PLEG): container finished" podID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" containerID="2216adce2e21d9d9960b2e0982a6a89b3e6d2f41c0bd1ed8426c5658cf2f078d" exitCode=0 Oct 30 05:38:24.211462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.211277 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" Oct 30 05:38:24.211462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.211316 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" event={"ID":"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e","Type":"ContainerDied","Data":"2216adce2e21d9d9960b2e0982a6a89b3e6d2f41c0bd1ed8426c5658cf2f078d"} Oct 30 05:38:24.211462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.211387 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx" event={"ID":"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e","Type":"ContainerDied","Data":"5de641351b1f0913b63b7218838b386bbcade255e4deb89dbc77f2e650b5677b"} Oct 30 05:38:24.211462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.211418 6587 scope.go:122] "RemoveContainer" containerID="2216adce2e21d9d9960b2e0982a6a89b3e6d2f41c0bd1ed8426c5658cf2f078d" Oct 30 05:38:24.213400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.213376 6587 generic.go:356] "Generic (PLEG): container finished" podID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerID="26120fe2eb888a7e7ba3da4f8dcec046228edd7dc692d46290c2fa82779a3335" exitCode=0 Oct 30 05:38:24.213539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.213422 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" event={"ID":"c5530d72-64d2-48b1-9f51-362f4a41b916","Type":"ContainerDied","Data":"26120fe2eb888a7e7ba3da4f8dcec046228edd7dc692d46290c2fa82779a3335"} Oct 30 05:38:24.213539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.213457 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" event={"ID":"c5530d72-64d2-48b1-9f51-362f4a41b916","Type":"ContainerDied","Data":"906f10d5f38d6b7a0c1479377ffa208a5728b386335260ea9bdf0b01600ecfda"} Oct 30 05:38:24.213539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.213506 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq" Oct 30 05:38:24.227306 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.227278 6587 scope.go:122] "RemoveContainer" containerID="2216adce2e21d9d9960b2e0982a6a89b3e6d2f41c0bd1ed8426c5658cf2f078d" Oct 30 05:38:24.227679 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:24.227657 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2216adce2e21d9d9960b2e0982a6a89b3e6d2f41c0bd1ed8426c5658cf2f078d\": container with ID starting with 2216adce2e21d9d9960b2e0982a6a89b3e6d2f41c0bd1ed8426c5658cf2f078d not found: ID does not exist" containerID="2216adce2e21d9d9960b2e0982a6a89b3e6d2f41c0bd1ed8426c5658cf2f078d" Oct 30 05:38:24.227772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.227686 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2216adce2e21d9d9960b2e0982a6a89b3e6d2f41c0bd1ed8426c5658cf2f078d"} err="failed to get container status \"2216adce2e21d9d9960b2e0982a6a89b3e6d2f41c0bd1ed8426c5658cf2f078d\": rpc error: code = NotFound desc = could not find container \"2216adce2e21d9d9960b2e0982a6a89b3e6d2f41c0bd1ed8426c5658cf2f078d\": container with ID starting with 2216adce2e21d9d9960b2e0982a6a89b3e6d2f41c0bd1ed8426c5658cf2f078d not found: ID does not exist" Oct 30 05:38:24.227772 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.227704 6587 scope.go:122] "RemoveContainer" containerID="26120fe2eb888a7e7ba3da4f8dcec046228edd7dc692d46290c2fa82779a3335" Oct 30 05:38:24.247102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.247082 6587 scope.go:122] "RemoveContainer" containerID="2e57348e3f31ad7d3b65738d60f17caddbb1ed44c802e1e3679ceac58dfca93f" Oct 30 05:38:24.258244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.258224 6587 scope.go:122] "RemoveContainer" containerID="26120fe2eb888a7e7ba3da4f8dcec046228edd7dc692d46290c2fa82779a3335" Oct 30 05:38:24.258554 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:24.258533 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26120fe2eb888a7e7ba3da4f8dcec046228edd7dc692d46290c2fa82779a3335\": container with ID starting with 26120fe2eb888a7e7ba3da4f8dcec046228edd7dc692d46290c2fa82779a3335 not found: ID does not exist" containerID="26120fe2eb888a7e7ba3da4f8dcec046228edd7dc692d46290c2fa82779a3335" Oct 30 05:38:24.258634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.258563 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26120fe2eb888a7e7ba3da4f8dcec046228edd7dc692d46290c2fa82779a3335"} err="failed to get container status \"26120fe2eb888a7e7ba3da4f8dcec046228edd7dc692d46290c2fa82779a3335\": rpc error: code = NotFound desc = could not find container \"26120fe2eb888a7e7ba3da4f8dcec046228edd7dc692d46290c2fa82779a3335\": container with ID starting with 26120fe2eb888a7e7ba3da4f8dcec046228edd7dc692d46290c2fa82779a3335 not found: ID does not exist" Oct 30 05:38:24.258634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.258581 6587 scope.go:122] "RemoveContainer" containerID="2e57348e3f31ad7d3b65738d60f17caddbb1ed44c802e1e3679ceac58dfca93f" Oct 30 05:38:24.258862 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:24.258845 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e57348e3f31ad7d3b65738d60f17caddbb1ed44c802e1e3679ceac58dfca93f\": container with ID starting with 2e57348e3f31ad7d3b65738d60f17caddbb1ed44c802e1e3679ceac58dfca93f not found: ID does not exist" containerID="2e57348e3f31ad7d3b65738d60f17caddbb1ed44c802e1e3679ceac58dfca93f" Oct 30 05:38:24.258904 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.258870 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e57348e3f31ad7d3b65738d60f17caddbb1ed44c802e1e3679ceac58dfca93f"} err="failed to get container status \"2e57348e3f31ad7d3b65738d60f17caddbb1ed44c802e1e3679ceac58dfca93f\": rpc error: code = NotFound desc = could not find container \"2e57348e3f31ad7d3b65738d60f17caddbb1ed44c802e1e3679ceac58dfca93f\": container with ID starting with 2e57348e3f31ad7d3b65738d60f17caddbb1ed44c802e1e3679ceac58dfca93f not found: ID does not exist" Oct 30 05:38:24.259045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.259028 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-token\" (UniqueName: \"kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-token\") pod \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " Oct 30 05:38:24.259089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.259070 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-envoy\") pod \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " Oct 30 05:38:24.259137 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.259126 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-podinfo\") pod \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " Oct 30 05:38:24.259194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.259159 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-kube-api-access-gmrnx\" (UniqueName: \"kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-kube-api-access-gmrnx\") pod \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " Oct 30 05:38:24.259307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.259198 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-socket\") pod \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " Oct 30 05:38:24.259307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.259299 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istiod-ca-cert\") pod \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " Oct 30 05:38:24.259448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.259397 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-credential-socket\") pod \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " Oct 30 05:38:24.259507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.259463 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-certs\") pod \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " Oct 30 05:38:24.259604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.259583 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-data\") pod \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\" (UID: \"c8590e3a-bbaa-46f0-ae63-a47d18f6b55e\") " Oct 30 05:38:24.260560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.259794 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-socket" pod "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" (UID: "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:24.260560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.259992 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-certs" pod "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" (UID: "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:24.260560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.260033 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istiod-ca-cert" pod "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" (UID: "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:24.260560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.260208 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-data" pod "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" (UID: "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:24.260560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.260475 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:24.260560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.260511 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:24.260560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.260524 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:24.260560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.260537 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:24.261463 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.261441 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-credential-socket" pod "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" (UID: "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:24.262539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.262487 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-podinfo" pod "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" (UID: "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:38:24.262924 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.262897 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-kube-api-access-gmrnx" pod "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" (UID: "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e"). InnerVolumeSpecName "kube-api-access-gmrnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:24.263153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.263132 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-envoy" pod "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" (UID: "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:24.264527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.264502 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-token" pod "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" (UID: "c8590e3a-bbaa-46f0-ae63-a47d18f6b55e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:24.312281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.312243 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq"] Oct 30 05:38:24.317601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.317570 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq"] Oct 30 05:38:24.361870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.361833 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:24.361870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.361859 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:24.361870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.361870 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:24.361870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.361880 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:24.362232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.361890 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmrnx\" (UniqueName: \"kubernetes.io/projected/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e-kube-api-access-gmrnx\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:24.567861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.567769 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx"] Oct 30 05:38:24.571667 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.571638 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx"] Oct 30 05:38:24.628689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:24.628653 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" podUID="25ba8b43-1249-48f5-91b1-25bdb0f82dd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:38:25.581928 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:25.581896 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" path="/var/lib/kubelet/pods/c5530d72-64d2-48b1-9f51-362f4a41b916/volumes" Oct 30 05:38:25.582386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:25.582340 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7fd2f46-2be0-483e-a865-af37dcea59be" path="/var/lib/kubelet/pods/c7fd2f46-2be0-483e-a865-af37dcea59be/volumes" Oct 30 05:38:25.582809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:25.582794 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" path="/var/lib/kubelet/pods/c8590e3a-bbaa-46f0-ae63-a47d18f6b55e/volumes" Oct 30 05:38:26.018197 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:26.017680 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:38:26.152163 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.151791 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:38:26.154551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.154529 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:38:26.185389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.185354 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:38:26.194084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194060 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-socket\") pod \"664e59b0-fa1f-498c-960e-94457fdcf3f1\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " Oct 30 05:38:26.194205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194104 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-certs\") pod \"664e59b0-fa1f-498c-960e-94457fdcf3f1\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " Oct 30 05:38:26.194205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194131 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-credential-socket\") pod \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " Oct 30 05:38:26.194205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194186 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/664e59b0-fa1f-498c-960e-94457fdcf3f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/664e59b0-fa1f-498c-960e-94457fdcf3f1-istiod-ca-cert\") pod \"664e59b0-fa1f-498c-960e-94457fdcf3f1\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " Oct 30 05:38:26.194397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194242 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-socket\") pod \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " Oct 30 05:38:26.194397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194281 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istiod-ca-cert\") pod \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " Oct 30 05:38:26.194397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194302 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-socket" pod "664e59b0-fa1f-498c-960e-94457fdcf3f1" (UID: "664e59b0-fa1f-498c-960e-94457fdcf3f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.194397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194311 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-token\" (UniqueName: \"kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-token\") pod \"664e59b0-fa1f-498c-960e-94457fdcf3f1\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " Oct 30 05:38:26.194397 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194381 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-data\") pod \"664e59b0-fa1f-498c-960e-94457fdcf3f1\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " Oct 30 05:38:26.194685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194410 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-kube-api-access-tf6w6\" (UniqueName: \"kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-kube-api-access-tf6w6\") pod \"664e59b0-fa1f-498c-960e-94457fdcf3f1\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " Oct 30 05:38:26.194685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194455 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-certs" pod "664e59b0-fa1f-498c-960e-94457fdcf3f1" (UID: "664e59b0-fa1f-498c-960e-94457fdcf3f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.194685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194448 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-podinfo\") pod \"664e59b0-fa1f-498c-960e-94457fdcf3f1\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " Oct 30 05:38:26.194685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194615 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-socket" pod "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" (UID: "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.194685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194629 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-credential-socket\") pod \"664e59b0-fa1f-498c-960e-94457fdcf3f1\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " Oct 30 05:38:26.194970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194696 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-data" pod "664e59b0-fa1f-498c-960e-94457fdcf3f1" (UID: "664e59b0-fa1f-498c-960e-94457fdcf3f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.194970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194707 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-podinfo\") pod \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " Oct 30 05:38:26.194970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194769 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-kube-api-access-7fs5l\" (UniqueName: \"kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-kube-api-access-7fs5l\") pod \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " Oct 30 05:38:26.194970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194859 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-envoy\") pod \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " Oct 30 05:38:26.194970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194860 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-credential-socket" pod "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" (UID: "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.194970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194891 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-data\") pod \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " Oct 30 05:38:26.194970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194897 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664e59b0-fa1f-498c-960e-94457fdcf3f1-istiod-ca-cert" pod "664e59b0-fa1f-498c-960e-94457fdcf3f1" (UID: "664e59b0-fa1f-498c-960e-94457fdcf3f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:26.194970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194906 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istiod-ca-cert" pod "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" (UID: "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:26.194970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194960 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-credential-socket" pod "664e59b0-fa1f-498c-960e-94457fdcf3f1" (UID: "664e59b0-fa1f-498c-960e-94457fdcf3f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.195506 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.194994 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-envoy\") pod \"664e59b0-fa1f-498c-960e-94457fdcf3f1\" (UID: \"664e59b0-fa1f-498c-960e-94457fdcf3f1\") " Oct 30 05:38:26.195506 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.195096 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-token\" (UniqueName: \"kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-token\") pod \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " Oct 30 05:38:26.195506 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.195146 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-certs\") pod \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\" (UID: \"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf\") " Oct 30 05:38:26.195967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.195145 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-data" pod "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" (UID: "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.195967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.195812 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-certs" pod "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" (UID: "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.195967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.195910 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.195967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.195934 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.195967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.195949 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.195967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.195962 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/664e59b0-fa1f-498c-960e-94457fdcf3f1-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.196383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.195976 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.196383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.195989 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.196383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.196002 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.196383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.196014 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.197546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.197518 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-envoy" pod "664e59b0-fa1f-498c-960e-94457fdcf3f1" (UID: "664e59b0-fa1f-498c-960e-94457fdcf3f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.197650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.197576 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-podinfo" pod "664e59b0-fa1f-498c-960e-94457fdcf3f1" (UID: "664e59b0-fa1f-498c-960e-94457fdcf3f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:38:26.197955 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.197928 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-kube-api-access-tf6w6" pod "664e59b0-fa1f-498c-960e-94457fdcf3f1" (UID: "664e59b0-fa1f-498c-960e-94457fdcf3f1"). InnerVolumeSpecName "kube-api-access-tf6w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:26.198172 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.198146 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-kube-api-access-7fs5l" pod "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" (UID: "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf"). InnerVolumeSpecName "kube-api-access-7fs5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:26.198266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.198247 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-token" pod "664e59b0-fa1f-498c-960e-94457fdcf3f1" (UID: "664e59b0-fa1f-498c-960e-94457fdcf3f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:26.198464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.198441 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-token" pod "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" (UID: "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:26.198654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.198626 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-envoy" pod "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" (UID: "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.198984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.198968 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-podinfo" pod "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" (UID: "5d5fe84e-9ee4-4c57-bbbc-d37b491564bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:38:26.234984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.234949 6587 generic.go:356] "Generic (PLEG): container finished" podID="664e59b0-fa1f-498c-960e-94457fdcf3f1" containerID="3fff937c00339736e5c0c4041c070b7f4c47d1f1febabe285f6c853c02bbb5c7" exitCode=0 Oct 30 05:38:26.235175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.235083 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" event={"ID":"664e59b0-fa1f-498c-960e-94457fdcf3f1","Type":"ContainerDied","Data":"3fff937c00339736e5c0c4041c070b7f4c47d1f1febabe285f6c853c02bbb5c7"} Oct 30 05:38:26.235175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.235134 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" event={"ID":"664e59b0-fa1f-498c-960e-94457fdcf3f1","Type":"ContainerDied","Data":"b53a0a2dce17cbf62859c065261b6c85f2aada99005a235706f33917a2353b93"} Oct 30 05:38:26.235175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.235163 6587 scope.go:122] "RemoveContainer" containerID="3fff937c00339736e5c0c4041c070b7f4c47d1f1febabe285f6c853c02bbb5c7" Oct 30 05:38:26.235379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.235311 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw" Oct 30 05:38:26.244444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.243956 6587 generic.go:356] "Generic (PLEG): container finished" podID="5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" containerID="954b2850487489ca5c770cb0efd8c655bc6fb5f36b7d9b6e149d5c95f3ecd6b5" exitCode=0 Oct 30 05:38:26.244444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.244031 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" event={"ID":"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf","Type":"ContainerDied","Data":"954b2850487489ca5c770cb0efd8c655bc6fb5f36b7d9b6e149d5c95f3ecd6b5"} Oct 30 05:38:26.244444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.244094 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" event={"ID":"5d5fe84e-9ee4-4c57-bbbc-d37b491564bf","Type":"ContainerDied","Data":"9a098357b056579cfecac4dbe583d28b68540ff823c57bd142945446f89c2848"} Oct 30 05:38:26.244444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.244182 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh" Oct 30 05:38:26.247957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.247928 6587 generic.go:356] "Generic (PLEG): container finished" podID="54424663-3cda-49f5-a298-88beadbf9343" containerID="da1715bd43f17a5c90bba01971076e9c84130f2c0847739b93da8bbafb6bd3a9" exitCode=0 Oct 30 05:38:26.248100 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.248075 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" event={"ID":"54424663-3cda-49f5-a298-88beadbf9343","Type":"ContainerDied","Data":"da1715bd43f17a5c90bba01971076e9c84130f2c0847739b93da8bbafb6bd3a9"} Oct 30 05:38:26.248169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.248117 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" event={"ID":"54424663-3cda-49f5-a298-88beadbf9343","Type":"ContainerDied","Data":"d5c0cb254fa906b751467e32d14f3546bb215a3357ea78c72dc7c704bc7848c7"} Oct 30 05:38:26.248235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.248177 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf" Oct 30 05:38:26.263407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.263326 6587 scope.go:122] "RemoveContainer" containerID="3fff937c00339736e5c0c4041c070b7f4c47d1f1febabe285f6c853c02bbb5c7" Oct 30 05:38:26.263666 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:26.263635 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fff937c00339736e5c0c4041c070b7f4c47d1f1febabe285f6c853c02bbb5c7\": container with ID starting with 3fff937c00339736e5c0c4041c070b7f4c47d1f1febabe285f6c853c02bbb5c7 not found: ID does not exist" containerID="3fff937c00339736e5c0c4041c070b7f4c47d1f1febabe285f6c853c02bbb5c7" Oct 30 05:38:26.263827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.263678 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fff937c00339736e5c0c4041c070b7f4c47d1f1febabe285f6c853c02bbb5c7"} err="failed to get container status \"3fff937c00339736e5c0c4041c070b7f4c47d1f1febabe285f6c853c02bbb5c7\": rpc error: code = NotFound desc = could not find container \"3fff937c00339736e5c0c4041c070b7f4c47d1f1febabe285f6c853c02bbb5c7\": container with ID starting with 3fff937c00339736e5c0c4041c070b7f4c47d1f1febabe285f6c853c02bbb5c7 not found: ID does not exist" Oct 30 05:38:26.263827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.263706 6587 scope.go:122] "RemoveContainer" containerID="954b2850487489ca5c770cb0efd8c655bc6fb5f36b7d9b6e149d5c95f3ecd6b5" Oct 30 05:38:26.281422 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.281395 6587 scope.go:122] "RemoveContainer" containerID="954b2850487489ca5c770cb0efd8c655bc6fb5f36b7d9b6e149d5c95f3ecd6b5" Oct 30 05:38:26.281740 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:26.281710 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954b2850487489ca5c770cb0efd8c655bc6fb5f36b7d9b6e149d5c95f3ecd6b5\": container with ID starting with 954b2850487489ca5c770cb0efd8c655bc6fb5f36b7d9b6e149d5c95f3ecd6b5 not found: ID does not exist" containerID="954b2850487489ca5c770cb0efd8c655bc6fb5f36b7d9b6e149d5c95f3ecd6b5" Oct 30 05:38:26.281819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.281739 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954b2850487489ca5c770cb0efd8c655bc6fb5f36b7d9b6e149d5c95f3ecd6b5"} err="failed to get container status \"954b2850487489ca5c770cb0efd8c655bc6fb5f36b7d9b6e149d5c95f3ecd6b5\": rpc error: code = NotFound desc = could not find container \"954b2850487489ca5c770cb0efd8c655bc6fb5f36b7d9b6e149d5c95f3ecd6b5\": container with ID starting with 954b2850487489ca5c770cb0efd8c655bc6fb5f36b7d9b6e149d5c95f3ecd6b5 not found: ID does not exist" Oct 30 05:38:26.281819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.281757 6587 scope.go:122] "RemoveContainer" containerID="da1715bd43f17a5c90bba01971076e9c84130f2c0847739b93da8bbafb6bd3a9" Oct 30 05:38:26.294198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.294146 6587 scope.go:122] "RemoveContainer" containerID="da1715bd43f17a5c90bba01971076e9c84130f2c0847739b93da8bbafb6bd3a9" Oct 30 05:38:26.294530 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:26.294504 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1715bd43f17a5c90bba01971076e9c84130f2c0847739b93da8bbafb6bd3a9\": container with ID starting with da1715bd43f17a5c90bba01971076e9c84130f2c0847739b93da8bbafb6bd3a9 not found: ID does not exist" containerID="da1715bd43f17a5c90bba01971076e9c84130f2c0847739b93da8bbafb6bd3a9" Oct 30 05:38:26.294631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.294541 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1715bd43f17a5c90bba01971076e9c84130f2c0847739b93da8bbafb6bd3a9"} err="failed to get container status \"da1715bd43f17a5c90bba01971076e9c84130f2c0847739b93da8bbafb6bd3a9\": rpc error: code = NotFound desc = could not find container \"da1715bd43f17a5c90bba01971076e9c84130f2c0847739b93da8bbafb6bd3a9\": container with ID starting with da1715bd43f17a5c90bba01971076e9c84130f2c0847739b93da8bbafb6bd3a9 not found: ID does not exist" Oct 30 05:38:26.296999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.296954 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-socket\") pod \"54424663-3cda-49f5-a298-88beadbf9343\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " Oct 30 05:38:26.297106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.297062 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-istio-token\" (UniqueName: \"kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-istio-token\") pod \"54424663-3cda-49f5-a298-88beadbf9343\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " Oct 30 05:38:26.297174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.297106 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-data\") pod \"54424663-3cda-49f5-a298-88beadbf9343\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " Oct 30 05:38:26.297174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.297150 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-kube-api-access-zbm28\" (UniqueName: \"kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-kube-api-access-zbm28\") pod \"54424663-3cda-49f5-a298-88beadbf9343\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " Oct 30 05:38:26.297302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.297245 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-socket" pod "54424663-3cda-49f5-a298-88beadbf9343" (UID: "54424663-3cda-49f5-a298-88beadbf9343"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.297377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.297300 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-envoy\") pod \"54424663-3cda-49f5-a298-88beadbf9343\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.297546 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-data" pod "54424663-3cda-49f5-a298-88beadbf9343" (UID: "54424663-3cda-49f5-a298-88beadbf9343"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.297556 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-credential-socket\") pod \"54424663-3cda-49f5-a298-88beadbf9343\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.297639 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-certs\") pod \"54424663-3cda-49f5-a298-88beadbf9343\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.297671 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54424663-3cda-49f5-a298-88beadbf9343-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54424663-3cda-49f5-a298-88beadbf9343-istio-podinfo\") pod \"54424663-3cda-49f5-a298-88beadbf9343\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.297860 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54424663-3cda-49f5-a298-88beadbf9343-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54424663-3cda-49f5-a298-88beadbf9343-istiod-ca-cert\") pod \"54424663-3cda-49f5-a298-88beadbf9343\" (UID: \"54424663-3cda-49f5-a298-88beadbf9343\") " Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.297916 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-credential-socket" pod "54424663-3cda-49f5-a298-88beadbf9343" (UID: "54424663-3cda-49f5-a298-88beadbf9343"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.297936 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-certs" pod "54424663-3cda-49f5-a298-88beadbf9343" (UID: "54424663-3cda-49f5-a298-88beadbf9343"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298249 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54424663-3cda-49f5-a298-88beadbf9343-istiod-ca-cert" pod "54424663-3cda-49f5-a298-88beadbf9343" (UID: "54424663-3cda-49f5-a298-88beadbf9343"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298609 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298636 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298652 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf6w6\" (UniqueName: \"kubernetes.io/projected/664e59b0-fa1f-498c-960e-94457fdcf3f1-kube-api-access-tf6w6\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298666 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298682 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298699 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298714 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fs5l\" (UniqueName: \"kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-kube-api-access-7fs5l\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298730 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298745 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298758 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298771 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54424663-3cda-49f5-a298-88beadbf9343-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298785 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/664e59b0-fa1f-498c-960e-94457fdcf3f1-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298798 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298811 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.299101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.298823 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.300525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.300435 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-envoy" pod "54424663-3cda-49f5-a298-88beadbf9343" (UID: "54424663-3cda-49f5-a298-88beadbf9343"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:26.300723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.300698 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54424663-3cda-49f5-a298-88beadbf9343-istio-podinfo" pod "54424663-3cda-49f5-a298-88beadbf9343" (UID: "54424663-3cda-49f5-a298-88beadbf9343"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:38:26.300782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.300735 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-kube-api-access-zbm28" pod "54424663-3cda-49f5-a298-88beadbf9343" (UID: "54424663-3cda-49f5-a298-88beadbf9343"). InnerVolumeSpecName "kube-api-access-zbm28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:26.300823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.300805 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-istio-token" pod "54424663-3cda-49f5-a298-88beadbf9343" (UID: "54424663-3cda-49f5-a298-88beadbf9343"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:26.369616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.369584 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx"] Oct 30 05:38:26.370220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370206 6587 cpu_manager.go:414] "containerMap: removing container" podUID="54424663-3cda-49f5-a298-88beadbf9343" containerName="istio-proxy" Oct 30 05:38:26.370271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370223 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54424663-3cda-49f5-a298-88beadbf9343" containerName="istio-proxy" Oct 30 05:38:26.370271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370244 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerName="istio-proxy" Oct 30 05:38:26.370271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370249 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerName="istio-proxy" Oct 30 05:38:26.370394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370271 6587 cpu_manager.go:414] "containerMap: removing container" podUID="664e59b0-fa1f-498c-960e-94457fdcf3f1" containerName="istio-proxy" Oct 30 05:38:26.370394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370276 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664e59b0-fa1f-498c-960e-94457fdcf3f1" containerName="istio-proxy" Oct 30 05:38:26.370394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370291 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7ba66369-7e9c-46a7-ba63-2d08bc3187c5" containerName="revoke-keys" Oct 30 05:38:26.370394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370310 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba66369-7e9c-46a7-ba63-2d08bc3187c5" containerName="revoke-keys" Oct 30 05:38:26.370394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370320 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" containerName="istio-proxy" Oct 30 05:38:26.370394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370325 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" containerName="istio-proxy" Oct 30 05:38:26.370394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370331 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" containerName="istio-proxy" Oct 30 05:38:26.370394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370336 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" containerName="istio-proxy" Oct 30 05:38:26.370647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370458 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" containerName="istio-proxy" Oct 30 05:38:26.370647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370470 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8590e3a-bbaa-46f0-ae63-a47d18f6b55e" containerName="istio-proxy" Oct 30 05:38:26.370647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370479 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerName="istio-proxy" Oct 30 05:38:26.370647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370487 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="54424663-3cda-49f5-a298-88beadbf9343" containerName="istio-proxy" Oct 30 05:38:26.370647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370496 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerName="istio-proxy" Oct 30 05:38:26.370647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370505 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="664e59b0-fa1f-498c-960e-94457fdcf3f1" containerName="istio-proxy" Oct 30 05:38:26.370647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.370513 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba66369-7e9c-46a7-ba63-2d08bc3187c5" containerName="revoke-keys" Oct 30 05:38:26.375937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.375902 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" Oct 30 05:38:26.378947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.378924 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw"] Oct 30 05:38:26.382075 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.382048 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw"] Oct 30 05:38:26.395061 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.395008 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx"] Oct 30 05:38:26.396049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.396026 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:38:26.396133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.396111 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:38:26.400261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.400234 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54424663-3cda-49f5-a298-88beadbf9343-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.400394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.400265 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.400394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.400282 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbm28\" (UniqueName: \"kubernetes.io/projected/54424663-3cda-49f5-a298-88beadbf9343-kube-api-access-zbm28\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.400394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.400297 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54424663-3cda-49f5-a298-88beadbf9343-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:26.417459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.417417 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh"] Oct 30 05:38:26.420839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.420813 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh"] Oct 30 05:38:26.501517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.501482 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjpzf\" (UniqueName: \"kubernetes.io/projected/d57116f6-cce0-4902-8619-89c358dcab6f-kube-api-access-gjpzf\") pod \"maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx\" (UID: \"d57116f6-cce0-4902-8619-89c358dcab6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" Oct 30 05:38:26.501693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.501524 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx\" (UID: \"d57116f6-cce0-4902-8619-89c358dcab6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" Oct 30 05:38:26.501693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.501666 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx\" (UID: \"d57116f6-cce0-4902-8619-89c358dcab6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" Oct 30 05:38:26.603275 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.603222 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjpzf\" (UniqueName: \"kubernetes.io/projected/d57116f6-cce0-4902-8619-89c358dcab6f-kube-api-access-gjpzf\") pod \"maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx\" (UID: \"d57116f6-cce0-4902-8619-89c358dcab6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" Oct 30 05:38:26.603741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.603411 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx\" (UID: \"d57116f6-cce0-4902-8619-89c358dcab6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" Oct 30 05:38:26.603741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.603661 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx\" (UID: \"d57116f6-cce0-4902-8619-89c358dcab6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" Oct 30 05:38:26.604459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.604434 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf"] Oct 30 05:38:26.604719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.604658 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx\" (UID: \"d57116f6-cce0-4902-8619-89c358dcab6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" Oct 30 05:38:26.606352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.606327 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx\" (UID: \"d57116f6-cce0-4902-8619-89c358dcab6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" Oct 30 05:38:26.608841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.608817 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf"] Oct 30 05:38:26.622952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.622928 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjpzf\" (UniqueName: \"kubernetes.io/projected/d57116f6-cce0-4902-8619-89c358dcab6f-kube-api-access-gjpzf\") pod \"maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx\" (UID: \"d57116f6-cce0-4902-8619-89c358dcab6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" Oct 30 05:38:26.690851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:26.690770 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" Oct 30 05:38:27.003557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:27.003528 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx"] Oct 30 05:38:27.005641 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:38:27.005612 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57116f6_cce0_4902_8619_89c358dcab6f.slice/crio-661e5d878da8d6c36d9882d08e1cce2bb5933645481bc35c3d1231abce2d397a WatchSource:0}: Error finding container 661e5d878da8d6c36d9882d08e1cce2bb5933645481bc35c3d1231abce2d397a: Status 404 returned error can't find the container with id 661e5d878da8d6c36d9882d08e1cce2bb5933645481bc35c3d1231abce2d397a Oct 30 05:38:27.265034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:27.264954 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" event={"ID":"d57116f6-cce0-4902-8619-89c358dcab6f","Type":"ContainerStarted","Data":"661e5d878da8d6c36d9882d08e1cce2bb5933645481bc35c3d1231abce2d397a"} Oct 30 05:38:27.582541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:27.582505 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54424663-3cda-49f5-a298-88beadbf9343" path="/var/lib/kubelet/pods/54424663-3cda-49f5-a298-88beadbf9343/volumes" Oct 30 05:38:27.583049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:27.583035 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d5fe84e-9ee4-4c57-bbbc-d37b491564bf" path="/var/lib/kubelet/pods/5d5fe84e-9ee4-4c57-bbbc-d37b491564bf/volumes" Oct 30 05:38:27.583488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:27.583475 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="664e59b0-fa1f-498c-960e-94457fdcf3f1" path="/var/lib/kubelet/pods/664e59b0-fa1f-498c-960e-94457fdcf3f1/volumes" Oct 30 05:38:28.275448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:28.275338 6587 generic.go:356] "Generic (PLEG): container finished" podID="d57116f6-cce0-4902-8619-89c358dcab6f" containerID="ca3631566ced55011fb45e0b96c4714618a1b30e989bdc336e2fecd92acea575" exitCode=0 Oct 30 05:38:28.275448 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:28.275428 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" event={"ID":"d57116f6-cce0-4902-8619-89c358dcab6f","Type":"ContainerDied","Data":"ca3631566ced55011fb45e0b96c4714618a1b30e989bdc336e2fecd92acea575"} Oct 30 05:38:29.432114 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:29.432085 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" Oct 30 05:38:29.548444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:29.548091 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-openshift-ca\") pod \"d57116f6-cce0-4902-8619-89c358dcab6f\" (UID: \"d57116f6-cce0-4902-8619-89c358dcab6f\") " Oct 30 05:38:29.548444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:29.548173 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-xks-ca\") pod \"d57116f6-cce0-4902-8619-89c358dcab6f\" (UID: \"d57116f6-cce0-4902-8619-89c358dcab6f\") " Oct 30 05:38:29.548444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:29.548216 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57116f6-cce0-4902-8619-89c358dcab6f-kube-api-access-gjpzf\" (UniqueName: \"kubernetes.io/projected/d57116f6-cce0-4902-8619-89c358dcab6f-kube-api-access-gjpzf\") pod \"d57116f6-cce0-4902-8619-89c358dcab6f\" (UID: \"d57116f6-cce0-4902-8619-89c358dcab6f\") " Oct 30 05:38:29.556402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:29.551833 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57116f6-cce0-4902-8619-89c358dcab6f-kube-api-access-gjpzf" pod "d57116f6-cce0-4902-8619-89c358dcab6f" (UID: "d57116f6-cce0-4902-8619-89c358dcab6f"). InnerVolumeSpecName "kube-api-access-gjpzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:29.556402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:29.552185 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-openshift-ca" pod "d57116f6-cce0-4902-8619-89c358dcab6f" (UID: "d57116f6-cce0-4902-8619-89c358dcab6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:29.572982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:29.572932 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-xks-ca" pod "d57116f6-cce0-4902-8619-89c358dcab6f" (UID: "d57116f6-cce0-4902-8619-89c358dcab6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:29.649883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:29.649845 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:29.649883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:29.649877 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57116f6-cce0-4902-8619-89c358dcab6f-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:29.649883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:29.649887 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjpzf\" (UniqueName: \"kubernetes.io/projected/d57116f6-cce0-4902-8619-89c358dcab6f-kube-api-access-gjpzf\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:29.998488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:29.998406 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg"] Oct 30 05:38:30.002806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:30.002773 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-93bad9-bg2sg"] Oct 30 05:38:30.296158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:30.296075 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" Oct 30 05:38:30.296158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:30.296095 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx" event={"ID":"d57116f6-cce0-4902-8619-89c358dcab6f","Type":"ContainerDied","Data":"661e5d878da8d6c36d9882d08e1cce2bb5933645481bc35c3d1231abce2d397a"} Oct 30 05:38:30.296158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:30.296145 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="661e5d878da8d6c36d9882d08e1cce2bb5933645481bc35c3d1231abce2d397a" Oct 30 05:38:31.581488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.581454 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb4205bf-0b85-4788-9ee3-4e1867e8ee3f" path="/var/lib/kubelet/pods/fb4205bf-0b85-4788-9ee3-4e1867e8ee3f/volumes" Oct 30 05:38:31.924716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.924636 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf"] Oct 30 05:38:31.925315 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.925299 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d57116f6-cce0-4902-8619-89c358dcab6f" containerName="revoke-keys" Oct 30 05:38:31.925406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.925315 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57116f6-cce0-4902-8619-89c358dcab6f" containerName="revoke-keys" Oct 30 05:38:31.925406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.925371 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerName="istio-proxy" Oct 30 05:38:31.925406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.925377 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5530d72-64d2-48b1-9f51-362f4a41b916" containerName="istio-proxy" Oct 30 05:38:31.925578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.925474 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57116f6-cce0-4902-8619-89c358dcab6f" containerName="revoke-keys" Oct 30 05:38:31.930565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.930543 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" Oct 30 05:38:31.959209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.959181 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf"] Oct 30 05:38:31.961004 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.960984 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:38:31.961153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.961028 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:38:31.980232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.980200 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5"] Oct 30 05:38:31.985792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.985765 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" Oct 30 05:38:31.996980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.996951 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59094d-26e4-486b-b338-920802b35481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bb45835-8bnqf\" (UID: \"3f59094d-26e4-486b-b338-920802b35481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" Oct 30 05:38:31.997127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.997104 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59094d-26e4-486b-b338-920802b35481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bb45835-8bnqf\" (UID: \"3f59094d-26e4-486b-b338-920802b35481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" Oct 30 05:38:31.997290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:31.997267 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jssxj\" (UniqueName: \"kubernetes.io/projected/3f59094d-26e4-486b-b338-920802b35481-kube-api-access-jssxj\") pod \"maas-api-revoke-sub-1bb45835-8bnqf\" (UID: \"3f59094d-26e4-486b-b338-920802b35481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" Oct 30 05:38:32.007726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.007695 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5"] Oct 30 05:38:32.099205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.099171 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jssxj\" (UniqueName: \"kubernetes.io/projected/3f59094d-26e4-486b-b338-920802b35481-kube-api-access-jssxj\") pod \"maas-api-revoke-sub-1bb45835-8bnqf\" (UID: \"3f59094d-26e4-486b-b338-920802b35481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" Oct 30 05:38:32.099440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.099297 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b96s\" (UniqueName: \"kubernetes.io/projected/2deccbff-8088-44e8-9c67-10f1165213b5-kube-api-access-2b96s\") pod \"maas-api-revoke-sub-9276e572-9cwg5\" (UID: \"2deccbff-8088-44e8-9c67-10f1165213b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" Oct 30 05:38:32.099440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.099366 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59094d-26e4-486b-b338-920802b35481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bb45835-8bnqf\" (UID: \"3f59094d-26e4-486b-b338-920802b35481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" Oct 30 05:38:32.099529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.099475 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59094d-26e4-486b-b338-920802b35481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bb45835-8bnqf\" (UID: \"3f59094d-26e4-486b-b338-920802b35481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" Oct 30 05:38:32.099529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.099502 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9276e572-9cwg5\" (UID: \"2deccbff-8088-44e8-9c67-10f1165213b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" Oct 30 05:38:32.099616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.099565 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9276e572-9cwg5\" (UID: \"2deccbff-8088-44e8-9c67-10f1165213b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" Oct 30 05:38:32.100148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.100118 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59094d-26e4-486b-b338-920802b35481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bb45835-8bnqf\" (UID: \"3f59094d-26e4-486b-b338-920802b35481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" Oct 30 05:38:32.101987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.101963 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59094d-26e4-486b-b338-920802b35481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bb45835-8bnqf\" (UID: \"3f59094d-26e4-486b-b338-920802b35481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" Oct 30 05:38:32.120951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.120927 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jssxj\" (UniqueName: \"kubernetes.io/projected/3f59094d-26e4-486b-b338-920802b35481-kube-api-access-jssxj\") pod \"maas-api-revoke-sub-1bb45835-8bnqf\" (UID: \"3f59094d-26e4-486b-b338-920802b35481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" Oct 30 05:38:32.201263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.201164 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b96s\" (UniqueName: \"kubernetes.io/projected/2deccbff-8088-44e8-9c67-10f1165213b5-kube-api-access-2b96s\") pod \"maas-api-revoke-sub-9276e572-9cwg5\" (UID: \"2deccbff-8088-44e8-9c67-10f1165213b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" Oct 30 05:38:32.201474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.201286 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9276e572-9cwg5\" (UID: \"2deccbff-8088-44e8-9c67-10f1165213b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" Oct 30 05:38:32.201474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.201340 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9276e572-9cwg5\" (UID: \"2deccbff-8088-44e8-9c67-10f1165213b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" Oct 30 05:38:32.202034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.202012 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9276e572-9cwg5\" (UID: \"2deccbff-8088-44e8-9c67-10f1165213b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" Oct 30 05:38:32.203664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.203642 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9276e572-9cwg5\" (UID: \"2deccbff-8088-44e8-9c67-10f1165213b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" Oct 30 05:38:32.230315 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.230284 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b96s\" (UniqueName: \"kubernetes.io/projected/2deccbff-8088-44e8-9c67-10f1165213b5-kube-api-access-2b96s\") pod \"maas-api-revoke-sub-9276e572-9cwg5\" (UID: \"2deccbff-8088-44e8-9c67-10f1165213b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" Oct 30 05:38:32.245137 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.245113 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" Oct 30 05:38:32.302602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.302565 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" Oct 30 05:38:32.585212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.585173 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf"] Oct 30 05:38:32.668031 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.668001 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5"] Oct 30 05:38:32.671419 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:38:32.671349 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2deccbff_8088_44e8_9c67_10f1165213b5.slice/crio-31b21a73d06b2c60253a3dd871b5ce6cd7ebcc366311c5659b18260b345ff21d WatchSource:0}: Error finding container 31b21a73d06b2c60253a3dd871b5ce6cd7ebcc366311c5659b18260b345ff21d: Status 404 returned error can't find the container with id 31b21a73d06b2c60253a3dd871b5ce6cd7ebcc366311c5659b18260b345ff21d Oct 30 05:38:32.717318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.717288 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" Oct 30 05:38:32.841455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.841375 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-plugins-config-volume\") pod \"25ba8b43-1249-48f5-91b1-25bdb0f82dd5\" (UID: \"25ba8b43-1249-48f5-91b1-25bdb0f82dd5\") " Oct 30 05:38:32.841455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.841422 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-kube-api-access-b6p5w\" (UniqueName: \"kubernetes.io/projected/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-kube-api-access-b6p5w\") pod \"25ba8b43-1249-48f5-91b1-25bdb0f82dd5\" (UID: \"25ba8b43-1249-48f5-91b1-25bdb0f82dd5\") " Oct 30 05:38:32.841845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.841815 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-plugins-config-volume" pod "25ba8b43-1249-48f5-91b1-25bdb0f82dd5" (UID: "25ba8b43-1249-48f5-91b1-25bdb0f82dd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:32.842485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.842083 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:32.843654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.843632 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-kube-api-access-b6p5w" pod "25ba8b43-1249-48f5-91b1-25bdb0f82dd5" (UID: "25ba8b43-1249-48f5-91b1-25bdb0f82dd5"). InnerVolumeSpecName "kube-api-access-b6p5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:32.943802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:32.943771 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6p5w\" (UniqueName: \"kubernetes.io/projected/25ba8b43-1249-48f5-91b1-25bdb0f82dd5-kube-api-access-b6p5w\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:33.325079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.325040 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" event={"ID":"3f59094d-26e4-486b-b338-920802b35481","Type":"ContainerStarted","Data":"98bd08e4b65b3e15154c0fcff11752a5c87eb87cf21e008989ec8ec5d288014a"} Oct 30 05:38:33.325270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.325106 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" event={"ID":"3f59094d-26e4-486b-b338-920802b35481","Type":"ContainerStarted","Data":"e973a531ebdb56489c0a1d81e850a97ef89d93d9fceed0801b3140b7cdcdb114"} Oct 30 05:38:33.326780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.326748 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" event={"ID":"2deccbff-8088-44e8-9c67-10f1165213b5","Type":"ContainerStarted","Data":"520e0f3fc412ba204cf2b53d25bd4bd08ce6e9b5fd921799407f3cc13fd42e6f"} Oct 30 05:38:33.326934 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.326798 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" event={"ID":"2deccbff-8088-44e8-9c67-10f1165213b5","Type":"ContainerStarted","Data":"31b21a73d06b2c60253a3dd871b5ce6cd7ebcc366311c5659b18260b345ff21d"} Oct 30 05:38:33.328284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.328258 6587 generic.go:356] "Generic (PLEG): container finished" podID="25ba8b43-1249-48f5-91b1-25bdb0f82dd5" containerID="725ba193af45deb48e58b9bb1195662d681c0cb43f781c3320077e60eebde23e" exitCode=0 Oct 30 05:38:33.328423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.328293 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" Oct 30 05:38:33.328423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.328306 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" event={"ID":"25ba8b43-1249-48f5-91b1-25bdb0f82dd5","Type":"ContainerDied","Data":"725ba193af45deb48e58b9bb1195662d681c0cb43f781c3320077e60eebde23e"} Oct 30 05:38:33.328423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.328343 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk" event={"ID":"25ba8b43-1249-48f5-91b1-25bdb0f82dd5","Type":"ContainerDied","Data":"b26e2e16bd281cbc878817edb620e974488917caa85bc9a608af18fc5bb2e308"} Oct 30 05:38:33.328423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.328381 6587 scope.go:122] "RemoveContainer" containerID="725ba193af45deb48e58b9bb1195662d681c0cb43f781c3320077e60eebde23e" Oct 30 05:38:33.340190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.340174 6587 scope.go:122] "RemoveContainer" containerID="725ba193af45deb48e58b9bb1195662d681c0cb43f781c3320077e60eebde23e" Oct 30 05:38:33.340484 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:33.340461 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725ba193af45deb48e58b9bb1195662d681c0cb43f781c3320077e60eebde23e\": container with ID starting with 725ba193af45deb48e58b9bb1195662d681c0cb43f781c3320077e60eebde23e not found: ID does not exist" containerID="725ba193af45deb48e58b9bb1195662d681c0cb43f781c3320077e60eebde23e" Oct 30 05:38:33.340585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.340489 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725ba193af45deb48e58b9bb1195662d681c0cb43f781c3320077e60eebde23e"} err="failed to get container status \"725ba193af45deb48e58b9bb1195662d681c0cb43f781c3320077e60eebde23e\": rpc error: code = NotFound desc = could not find container \"725ba193af45deb48e58b9bb1195662d681c0cb43f781c3320077e60eebde23e\": container with ID starting with 725ba193af45deb48e58b9bb1195662d681c0cb43f781c3320077e60eebde23e not found: ID does not exist" Oct 30 05:38:33.370814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.370737 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" podStartSLOduration=2.370724803 podStartE2EDuration="2.370724803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:33.367462118 +0000 UTC m=+3237.895228904" watchObservedRunningTime="2026-10-30 05:38:33.370724803 +0000 UTC m=+3237.898491588" Oct 30 05:38:33.412206 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.412158 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" podStartSLOduration=2.412143849 podStartE2EDuration="2.412143849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:33.411653282 +0000 UTC m=+3237.939420069" watchObservedRunningTime="2026-10-30 05:38:33.412143849 +0000 UTC m=+3237.939910634" Oct 30 05:38:33.452261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.452233 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk"] Oct 30 05:38:33.455550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.455527 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk"] Oct 30 05:38:33.583235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:33.583196 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ba8b43-1249-48f5-91b1-25bdb0f82dd5" path="/var/lib/kubelet/pods/25ba8b43-1249-48f5-91b1-25bdb0f82dd5/volumes" Oct 30 05:38:34.338887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:34.338845 6587 generic.go:356] "Generic (PLEG): container finished" podID="2deccbff-8088-44e8-9c67-10f1165213b5" containerID="520e0f3fc412ba204cf2b53d25bd4bd08ce6e9b5fd921799407f3cc13fd42e6f" exitCode=0 Oct 30 05:38:34.339349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:34.338962 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" event={"ID":"2deccbff-8088-44e8-9c67-10f1165213b5","Type":"ContainerDied","Data":"520e0f3fc412ba204cf2b53d25bd4bd08ce6e9b5fd921799407f3cc13fd42e6f"} Oct 30 05:38:34.342428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:34.342399 6587 generic.go:356] "Generic (PLEG): container finished" podID="3f59094d-26e4-486b-b338-920802b35481" containerID="98bd08e4b65b3e15154c0fcff11752a5c87eb87cf21e008989ec8ec5d288014a" exitCode=0 Oct 30 05:38:34.342562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:34.342432 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" event={"ID":"3f59094d-26e4-486b-b338-920802b35481","Type":"ContainerDied","Data":"98bd08e4b65b3e15154c0fcff11752a5c87eb87cf21e008989ec8ec5d288014a"} Oct 30 05:38:35.539860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.539832 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" Oct 30 05:38:35.575601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.575568 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f59094d-26e4-486b-b338-920802b35481-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59094d-26e4-486b-b338-920802b35481-maas-api-openshift-ca\") pod \"3f59094d-26e4-486b-b338-920802b35481\" (UID: \"3f59094d-26e4-486b-b338-920802b35481\") " Oct 30 05:38:35.575757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.575701 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f59094d-26e4-486b-b338-920802b35481-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59094d-26e4-486b-b338-920802b35481-maas-api-xks-ca\") pod \"3f59094d-26e4-486b-b338-920802b35481\" (UID: \"3f59094d-26e4-486b-b338-920802b35481\") " Oct 30 05:38:35.575827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.575792 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f59094d-26e4-486b-b338-920802b35481-kube-api-access-jssxj\" (UniqueName: \"kubernetes.io/projected/3f59094d-26e4-486b-b338-920802b35481-kube-api-access-jssxj\") pod \"3f59094d-26e4-486b-b338-920802b35481\" (UID: \"3f59094d-26e4-486b-b338-920802b35481\") " Oct 30 05:38:35.576243 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.575972 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f59094d-26e4-486b-b338-920802b35481-maas-api-openshift-ca" pod "3f59094d-26e4-486b-b338-920802b35481" (UID: "3f59094d-26e4-486b-b338-920802b35481"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:35.576441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.576333 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59094d-26e4-486b-b338-920802b35481-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:35.578349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.578308 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f59094d-26e4-486b-b338-920802b35481-kube-api-access-jssxj" pod "3f59094d-26e4-486b-b338-920802b35481" (UID: "3f59094d-26e4-486b-b338-920802b35481"). InnerVolumeSpecName "kube-api-access-jssxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:35.578651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.578616 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f59094d-26e4-486b-b338-920802b35481-maas-api-xks-ca" pod "3f59094d-26e4-486b-b338-920802b35481" (UID: "3f59094d-26e4-486b-b338-920802b35481"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:35.608818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.608790 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" Oct 30 05:38:35.677093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.677059 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2deccbff-8088-44e8-9c67-10f1165213b5-kube-api-access-2b96s\" (UniqueName: \"kubernetes.io/projected/2deccbff-8088-44e8-9c67-10f1165213b5-kube-api-access-2b96s\") pod \"2deccbff-8088-44e8-9c67-10f1165213b5\" (UID: \"2deccbff-8088-44e8-9c67-10f1165213b5\") " Oct 30 05:38:35.677309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.677128 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-openshift-ca\") pod \"2deccbff-8088-44e8-9c67-10f1165213b5\" (UID: \"2deccbff-8088-44e8-9c67-10f1165213b5\") " Oct 30 05:38:35.677309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.677161 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-xks-ca\") pod \"2deccbff-8088-44e8-9c67-10f1165213b5\" (UID: \"2deccbff-8088-44e8-9c67-10f1165213b5\") " Oct 30 05:38:35.677573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.677545 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-openshift-ca" pod "2deccbff-8088-44e8-9c67-10f1165213b5" (UID: "2deccbff-8088-44e8-9c67-10f1165213b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:35.677859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.677834 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:35.677974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.677867 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59094d-26e4-486b-b338-920802b35481-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:35.677974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.677882 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jssxj\" (UniqueName: \"kubernetes.io/projected/3f59094d-26e4-486b-b338-920802b35481-kube-api-access-jssxj\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:35.679719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.679678 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-xks-ca" pod "2deccbff-8088-44e8-9c67-10f1165213b5" (UID: "2deccbff-8088-44e8-9c67-10f1165213b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:35.680030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.680003 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2deccbff-8088-44e8-9c67-10f1165213b5-kube-api-access-2b96s" pod "2deccbff-8088-44e8-9c67-10f1165213b5" (UID: "2deccbff-8088-44e8-9c67-10f1165213b5"). InnerVolumeSpecName "kube-api-access-2b96s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:35.779602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.779567 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b96s\" (UniqueName: \"kubernetes.io/projected/2deccbff-8088-44e8-9c67-10f1165213b5-kube-api-access-2b96s\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:35.779602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:35.779597 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2deccbff-8088-44e8-9c67-10f1165213b5-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:36.361213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:36.361185 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" Oct 30 05:38:36.361441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:36.361213 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf" event={"ID":"3f59094d-26e4-486b-b338-920802b35481","Type":"ContainerDied","Data":"e973a531ebdb56489c0a1d81e850a97ef89d93d9fceed0801b3140b7cdcdb114"} Oct 30 05:38:36.361441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:36.361279 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e973a531ebdb56489c0a1d81e850a97ef89d93d9fceed0801b3140b7cdcdb114" Oct 30 05:38:36.363092 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:36.363073 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" Oct 30 05:38:36.363217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:36.363088 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5" event={"ID":"2deccbff-8088-44e8-9c67-10f1165213b5","Type":"ContainerDied","Data":"31b21a73d06b2c60253a3dd871b5ce6cd7ebcc366311c5659b18260b345ff21d"} Oct 30 05:38:36.363217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:36.363133 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31b21a73d06b2c60253a3dd871b5ce6cd7ebcc366311c5659b18260b345ff21d" Oct 30 05:38:37.584367 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:37.584314 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n"] Oct 30 05:38:37.584850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:37.584587 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" containerID="cri-o://4981368cf600221dad417e44ddbb9d6350f43e2f928833ddf1c045533d75f6f7" gracePeriod=30 Oct 30 05:38:38.237273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.237232 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78"] Oct 30 05:38:38.237952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.237932 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3f59094d-26e4-486b-b338-920802b35481" containerName="revoke-keys" Oct 30 05:38:38.237952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.237953 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f59094d-26e4-486b-b338-920802b35481" containerName="revoke-keys" Oct 30 05:38:38.238122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.237982 6587 cpu_manager.go:414] "containerMap: removing container" podUID="25ba8b43-1249-48f5-91b1-25bdb0f82dd5" containerName="payload-pre-processing" Oct 30 05:38:38.238122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.237988 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ba8b43-1249-48f5-91b1-25bdb0f82dd5" containerName="payload-pre-processing" Oct 30 05:38:38.238122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.238009 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2deccbff-8088-44e8-9c67-10f1165213b5" containerName="revoke-keys" Oct 30 05:38:38.238122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.238014 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2deccbff-8088-44e8-9c67-10f1165213b5" containerName="revoke-keys" Oct 30 05:38:38.238122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.238109 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f59094d-26e4-486b-b338-920802b35481" containerName="revoke-keys" Oct 30 05:38:38.238122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.238122 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ba8b43-1249-48f5-91b1-25bdb0f82dd5" containerName="payload-pre-processing" Oct 30 05:38:38.238456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.238132 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2deccbff-8088-44e8-9c67-10f1165213b5" containerName="revoke-keys" Oct 30 05:38:38.241667 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.241644 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" Oct 30 05:38:38.265107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.265080 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78"] Oct 30 05:38:38.306479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.306448 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:38:38.306639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.306525 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:38:38.324233 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.324201 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-397179-6xs78\" (UID: \"29fbeaee-26a9-400f-a520-2c55782d2fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" Oct 30 05:38:38.324451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.324427 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-397179-6xs78\" (UID: \"29fbeaee-26a9-400f-a520-2c55782d2fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" Oct 30 05:38:38.324635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.324611 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxlcx\" (UniqueName: \"kubernetes.io/projected/29fbeaee-26a9-400f-a520-2c55782d2fc4-kube-api-access-pxlcx\") pod \"maas-api-revoke-keys-e2e-worker-w2-397179-6xs78\" (UID: \"29fbeaee-26a9-400f-a520-2c55782d2fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" Oct 30 05:38:38.426235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.426199 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-397179-6xs78\" (UID: \"29fbeaee-26a9-400f-a520-2c55782d2fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" Oct 30 05:38:38.426465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.426320 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxlcx\" (UniqueName: \"kubernetes.io/projected/29fbeaee-26a9-400f-a520-2c55782d2fc4-kube-api-access-pxlcx\") pod \"maas-api-revoke-keys-e2e-worker-w2-397179-6xs78\" (UID: \"29fbeaee-26a9-400f-a520-2c55782d2fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" Oct 30 05:38:38.426465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.426417 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-397179-6xs78\" (UID: \"29fbeaee-26a9-400f-a520-2c55782d2fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" Oct 30 05:38:38.427246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.427217 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-397179-6xs78\" (UID: \"29fbeaee-26a9-400f-a520-2c55782d2fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" Oct 30 05:38:38.428926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.428901 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-397179-6xs78\" (UID: \"29fbeaee-26a9-400f-a520-2c55782d2fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" Oct 30 05:38:38.446813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.446789 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxlcx\" (UniqueName: \"kubernetes.io/projected/29fbeaee-26a9-400f-a520-2c55782d2fc4-kube-api-access-pxlcx\") pod \"maas-api-revoke-keys-e2e-worker-w2-397179-6xs78\" (UID: \"29fbeaee-26a9-400f-a520-2c55782d2fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" Oct 30 05:38:38.557112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.557079 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" Oct 30 05:38:38.698421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.698347 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:38:38.698788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.698445 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:38:38.876654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:38.876617 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78"] Oct 30 05:38:38.878056 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:38:38.878031 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fbeaee_26a9_400f_a520_2c55782d2fc4.slice/crio-3f81c708bfb89d6985f4cc573844efd1961a1bb9130a3424e18606f62a615935 WatchSource:0}: Error finding container 3f81c708bfb89d6985f4cc573844efd1961a1bb9130a3424e18606f62a615935: Status 404 returned error can't find the container with id 3f81c708bfb89d6985f4cc573844efd1961a1bb9130a3424e18606f62a615935 Oct 30 05:38:39.392480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:39.392393 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" event={"ID":"29fbeaee-26a9-400f-a520-2c55782d2fc4","Type":"ContainerStarted","Data":"b46ccc8c32421f7397cd9de7d6cdc73c926fcf9d10b004722ad8e24ee159dfe9"} Oct 30 05:38:39.392480 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:39.392453 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" event={"ID":"29fbeaee-26a9-400f-a520-2c55782d2fc4","Type":"ContainerStarted","Data":"3f81c708bfb89d6985f4cc573844efd1961a1bb9130a3424e18606f62a615935"} Oct 30 05:38:39.441987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:39.441936 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" podStartSLOduration=1.441918107 podStartE2EDuration="1.441918107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:39.436924411 +0000 UTC m=+3243.964691216" watchObservedRunningTime="2026-10-30 05:38:39.441918107 +0000 UTC m=+3243.969684895" Oct 30 05:38:40.404341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:40.404296 6587 generic.go:356] "Generic (PLEG): container finished" podID="29fbeaee-26a9-400f-a520-2c55782d2fc4" containerID="b46ccc8c32421f7397cd9de7d6cdc73c926fcf9d10b004722ad8e24ee159dfe9" exitCode=0 Oct 30 05:38:40.404341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:40.404408 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" event={"ID":"29fbeaee-26a9-400f-a520-2c55782d2fc4","Type":"ContainerDied","Data":"b46ccc8c32421f7397cd9de7d6cdc73c926fcf9d10b004722ad8e24ee159dfe9"} Oct 30 05:38:41.047595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.047556 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t"] Oct 30 05:38:41.413227 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.413190 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" podUID="34b792b9-3587-447d-8e6a-a3d21bc4131b" containerName="maas-api" containerID="cri-o://b57e8bcaee26c7336f41e695739e97af1f202a0809ba42a0c9a1a7a2979fee2e" gracePeriod=30 Oct 30 05:38:41.571296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.571266 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" Oct 30 05:38:41.690604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.679216 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-xks-ca\") pod \"29fbeaee-26a9-400f-a520-2c55782d2fc4\" (UID: \"29fbeaee-26a9-400f-a520-2c55782d2fc4\") " Oct 30 05:38:41.690604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.680893 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-openshift-ca\") pod \"29fbeaee-26a9-400f-a520-2c55782d2fc4\" (UID: \"29fbeaee-26a9-400f-a520-2c55782d2fc4\") " Oct 30 05:38:41.690604 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.680974 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29fbeaee-26a9-400f-a520-2c55782d2fc4-kube-api-access-pxlcx\" (UniqueName: \"kubernetes.io/projected/29fbeaee-26a9-400f-a520-2c55782d2fc4-kube-api-access-pxlcx\") pod \"29fbeaee-26a9-400f-a520-2c55782d2fc4\" (UID: \"29fbeaee-26a9-400f-a520-2c55782d2fc4\") " Oct 30 05:38:41.702444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.693702 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-openshift-ca" pod "29fbeaee-26a9-400f-a520-2c55782d2fc4" (UID: "29fbeaee-26a9-400f-a520-2c55782d2fc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:41.716858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.714631 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-xks-ca" pod "29fbeaee-26a9-400f-a520-2c55782d2fc4" (UID: "29fbeaee-26a9-400f-a520-2c55782d2fc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:41.716858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.714769 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29fbeaee-26a9-400f-a520-2c55782d2fc4-kube-api-access-pxlcx" pod "29fbeaee-26a9-400f-a520-2c55782d2fc4" (UID: "29fbeaee-26a9-400f-a520-2c55782d2fc4"). InnerVolumeSpecName "kube-api-access-pxlcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:41.754851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.754661 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:38:41.783543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.783509 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:41.783543 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.783546 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fbeaee-26a9-400f-a520-2c55782d2fc4-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:41.783743 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.783563 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxlcx\" (UniqueName: \"kubernetes.io/projected/29fbeaee-26a9-400f-a520-2c55782d2fc4-kube-api-access-pxlcx\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:41.884453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.884422 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-metrics-certs\") pod \"34b792b9-3587-447d-8e6a-a3d21bc4131b\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " Oct 30 05:38:41.884616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.884461 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b792b9-3587-447d-8e6a-a3d21bc4131b-kube-api-access-f6jwr\" (UniqueName: \"kubernetes.io/projected/34b792b9-3587-447d-8e6a-a3d21bc4131b-kube-api-access-f6jwr\") pod \"34b792b9-3587-447d-8e6a-a3d21bc4131b\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " Oct 30 05:38:41.884661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.884616 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b792b9-3587-447d-8e6a-a3d21bc4131b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b792b9-3587-447d-8e6a-a3d21bc4131b-openshift-service-ca\") pod \"34b792b9-3587-447d-8e6a-a3d21bc4131b\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " Oct 30 05:38:41.884661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.884646 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-maas-api-tls\") pod \"34b792b9-3587-447d-8e6a-a3d21bc4131b\" (UID: \"34b792b9-3587-447d-8e6a-a3d21bc4131b\") " Oct 30 05:38:41.885033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.885000 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b792b9-3587-447d-8e6a-a3d21bc4131b-openshift-service-ca" pod "34b792b9-3587-447d-8e6a-a3d21bc4131b" (UID: "34b792b9-3587-447d-8e6a-a3d21bc4131b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:41.885315 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.885291 6587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b792b9-3587-447d-8e6a-a3d21bc4131b-openshift-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:41.886799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.886775 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-maas-api-tls" pod "34b792b9-3587-447d-8e6a-a3d21bc4131b" (UID: "34b792b9-3587-447d-8e6a-a3d21bc4131b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:41.887189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.887172 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-metrics-certs" pod "34b792b9-3587-447d-8e6a-a3d21bc4131b" (UID: "34b792b9-3587-447d-8e6a-a3d21bc4131b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:41.887271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.887235 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b792b9-3587-447d-8e6a-a3d21bc4131b-kube-api-access-f6jwr" pod "34b792b9-3587-447d-8e6a-a3d21bc4131b" (UID: "34b792b9-3587-447d-8e6a-a3d21bc4131b"). InnerVolumeSpecName "kube-api-access-f6jwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:41.928787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.928755 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9"] Oct 30 05:38:41.928973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.928951 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" podUID="4f145154-169d-4566-a3a9-8dec41416ceb" containerName="payload-processing" containerID="cri-o://733356db3d653e9a1a383fbf6522313ffd9062f0c7126ea45705aed186c567cf" gracePeriod=30 Oct 30 05:38:41.986188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.986154 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-maas-api-tls\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:41.986188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.986185 6587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b792b9-3587-447d-8e6a-a3d21bc4131b-metrics-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:41.986188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:41.986195 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6jwr\" (UniqueName: \"kubernetes.io/projected/34b792b9-3587-447d-8e6a-a3d21bc4131b-kube-api-access-f6jwr\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:42.100123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.100085 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p"] Oct 30 05:38:42.103141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.103115 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7a4600-tfk4p"] Oct 30 05:38:42.193531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.193505 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" Oct 30 05:38:42.303435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.302834 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f145154-169d-4566-a3a9-8dec41416ceb-kube-api-access-47zvh\" (UniqueName: \"kubernetes.io/projected/4f145154-169d-4566-a3a9-8dec41416ceb-kube-api-access-47zvh\") pod \"4f145154-169d-4566-a3a9-8dec41416ceb\" (UID: \"4f145154-169d-4566-a3a9-8dec41416ceb\") " Oct 30 05:38:42.303435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.302936 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f145154-169d-4566-a3a9-8dec41416ceb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f145154-169d-4566-a3a9-8dec41416ceb-plugins-config-volume\") pod \"4f145154-169d-4566-a3a9-8dec41416ceb\" (UID: \"4f145154-169d-4566-a3a9-8dec41416ceb\") " Oct 30 05:38:42.303900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.303741 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f145154-169d-4566-a3a9-8dec41416ceb-plugins-config-volume" pod "4f145154-169d-4566-a3a9-8dec41416ceb" (UID: "4f145154-169d-4566-a3a9-8dec41416ceb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:42.304479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.304441 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f145154-169d-4566-a3a9-8dec41416ceb-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:42.325202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.310103 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f145154-169d-4566-a3a9-8dec41416ceb-kube-api-access-47zvh" pod "4f145154-169d-4566-a3a9-8dec41416ceb" (UID: "4f145154-169d-4566-a3a9-8dec41416ceb"). InnerVolumeSpecName "kube-api-access-47zvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:42.405771 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.405736 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47zvh\" (UniqueName: \"kubernetes.io/projected/4f145154-169d-4566-a3a9-8dec41416ceb-kube-api-access-47zvh\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:42.422019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.421974 6587 generic.go:356] "Generic (PLEG): container finished" podID="34b792b9-3587-447d-8e6a-a3d21bc4131b" containerID="b57e8bcaee26c7336f41e695739e97af1f202a0809ba42a0c9a1a7a2979fee2e" exitCode=0 Oct 30 05:38:42.422513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.422029 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" Oct 30 05:38:42.422513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.422070 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" event={"ID":"34b792b9-3587-447d-8e6a-a3d21bc4131b","Type":"ContainerDied","Data":"b57e8bcaee26c7336f41e695739e97af1f202a0809ba42a0c9a1a7a2979fee2e"} Oct 30 05:38:42.422513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.422132 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t" event={"ID":"34b792b9-3587-447d-8e6a-a3d21bc4131b","Type":"ContainerDied","Data":"0e8ecd218fa787d4579a3c56ed67b0c500467890a7c93dd0a7243710b4baf47d"} Oct 30 05:38:42.422513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.422165 6587 scope.go:122] "RemoveContainer" containerID="b57e8bcaee26c7336f41e695739e97af1f202a0809ba42a0c9a1a7a2979fee2e" Oct 30 05:38:42.425134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.424980 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" Oct 30 05:38:42.425134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.425043 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78" event={"ID":"29fbeaee-26a9-400f-a520-2c55782d2fc4","Type":"ContainerDied","Data":"3f81c708bfb89d6985f4cc573844efd1961a1bb9130a3424e18606f62a615935"} Oct 30 05:38:42.425134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.425084 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f81c708bfb89d6985f4cc573844efd1961a1bb9130a3424e18606f62a615935" Oct 30 05:38:42.426544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.426519 6587 generic.go:356] "Generic (PLEG): container finished" podID="4f145154-169d-4566-a3a9-8dec41416ceb" containerID="733356db3d653e9a1a383fbf6522313ffd9062f0c7126ea45705aed186c567cf" exitCode=0 Oct 30 05:38:42.426641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.426567 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" event={"ID":"4f145154-169d-4566-a3a9-8dec41416ceb","Type":"ContainerDied","Data":"733356db3d653e9a1a383fbf6522313ffd9062f0c7126ea45705aed186c567cf"} Oct 30 05:38:42.426641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.426606 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" event={"ID":"4f145154-169d-4566-a3a9-8dec41416ceb","Type":"ContainerDied","Data":"55ae0e0ddcb94a1b42113cef0a8095c875a11d75b5b3516b59bd2574eb889256"} Oct 30 05:38:42.426738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.426669 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9" Oct 30 05:38:42.458201 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.458173 6587 scope.go:122] "RemoveContainer" containerID="b57e8bcaee26c7336f41e695739e97af1f202a0809ba42a0c9a1a7a2979fee2e" Oct 30 05:38:42.458655 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:42.458630 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b57e8bcaee26c7336f41e695739e97af1f202a0809ba42a0c9a1a7a2979fee2e\": container with ID starting with b57e8bcaee26c7336f41e695739e97af1f202a0809ba42a0c9a1a7a2979fee2e not found: ID does not exist" containerID="b57e8bcaee26c7336f41e695739e97af1f202a0809ba42a0c9a1a7a2979fee2e" Oct 30 05:38:42.458802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.458663 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b57e8bcaee26c7336f41e695739e97af1f202a0809ba42a0c9a1a7a2979fee2e"} err="failed to get container status \"b57e8bcaee26c7336f41e695739e97af1f202a0809ba42a0c9a1a7a2979fee2e\": rpc error: code = NotFound desc = could not find container \"b57e8bcaee26c7336f41e695739e97af1f202a0809ba42a0c9a1a7a2979fee2e\": container with ID starting with b57e8bcaee26c7336f41e695739e97af1f202a0809ba42a0c9a1a7a2979fee2e not found: ID does not exist" Oct 30 05:38:42.458802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.458684 6587 scope.go:122] "RemoveContainer" containerID="733356db3d653e9a1a383fbf6522313ffd9062f0c7126ea45705aed186c567cf" Oct 30 05:38:42.474227 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.474201 6587 scope.go:122] "RemoveContainer" containerID="733356db3d653e9a1a383fbf6522313ffd9062f0c7126ea45705aed186c567cf" Oct 30 05:38:42.474567 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:42.474542 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733356db3d653e9a1a383fbf6522313ffd9062f0c7126ea45705aed186c567cf\": container with ID starting with 733356db3d653e9a1a383fbf6522313ffd9062f0c7126ea45705aed186c567cf not found: ID does not exist" containerID="733356db3d653e9a1a383fbf6522313ffd9062f0c7126ea45705aed186c567cf" Oct 30 05:38:42.474674 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.474577 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733356db3d653e9a1a383fbf6522313ffd9062f0c7126ea45705aed186c567cf"} err="failed to get container status \"733356db3d653e9a1a383fbf6522313ffd9062f0c7126ea45705aed186c567cf\": rpc error: code = NotFound desc = could not find container \"733356db3d653e9a1a383fbf6522313ffd9062f0c7126ea45705aed186c567cf\": container with ID starting with 733356db3d653e9a1a383fbf6522313ffd9062f0c7126ea45705aed186c567cf not found: ID does not exist" Oct 30 05:38:42.503490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.503427 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t"] Oct 30 05:38:42.510892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.510854 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a4600-7486b47f6d-ncp8t"] Oct 30 05:38:42.558455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.558374 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9"] Oct 30 05:38:42.562953 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.562930 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9"] Oct 30 05:38:42.890456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.887917 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:38:42.931164 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.931113 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx"] Oct 30 05:38:42.934373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:42.934326 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7226a8-4phxx"] Oct 30 05:38:43.015692 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.015658 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-podinfo\") pod \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " Oct 30 05:38:43.015875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.015703 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-token\" (UniqueName: \"kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-token\") pod \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " Oct 30 05:38:43.015875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.015751 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-data\") pod \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " Oct 30 05:38:43.015875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.015836 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-credential-socket\") pod \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " Oct 30 05:38:43.015875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.015852 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-envoy\") pod \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " Oct 30 05:38:43.016148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.015880 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-certs\") pod \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " Oct 30 05:38:43.016148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.015953 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-socket\") pod \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " Oct 30 05:38:43.016148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.016050 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-kube-api-access-cl5hb\" (UniqueName: \"kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-kube-api-access-cl5hb\") pod \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " Oct 30 05:38:43.016148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.016081 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istiod-ca-cert\") pod \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\" (UID: \"33684ba3-e578-4f9f-bafd-7f9faefb39ad\") " Oct 30 05:38:43.016450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.016393 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-credential-socket" pod "33684ba3-e578-4f9f-bafd-7f9faefb39ad" (UID: "33684ba3-e578-4f9f-bafd-7f9faefb39ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:43.016622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.016590 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-certs" pod "33684ba3-e578-4f9f-bafd-7f9faefb39ad" (UID: "33684ba3-e578-4f9f-bafd-7f9faefb39ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:43.016622 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.016599 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-socket" pod "33684ba3-e578-4f9f-bafd-7f9faefb39ad" (UID: "33684ba3-e578-4f9f-bafd-7f9faefb39ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:43.016828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.016710 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istiod-ca-cert" pod "33684ba3-e578-4f9f-bafd-7f9faefb39ad" (UID: "33684ba3-e578-4f9f-bafd-7f9faefb39ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:43.016952 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.016926 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-data" pod "33684ba3-e578-4f9f-bafd-7f9faefb39ad" (UID: "33684ba3-e578-4f9f-bafd-7f9faefb39ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:43.017066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.017043 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.017125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.017067 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.017125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.017083 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.017125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.017096 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.018297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.018271 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-podinfo" pod "33684ba3-e578-4f9f-bafd-7f9faefb39ad" (UID: "33684ba3-e578-4f9f-bafd-7f9faefb39ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:38:43.018710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.018685 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-token" pod "33684ba3-e578-4f9f-bafd-7f9faefb39ad" (UID: "33684ba3-e578-4f9f-bafd-7f9faefb39ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:43.018814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.018685 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-envoy" pod "33684ba3-e578-4f9f-bafd-7f9faefb39ad" (UID: "33684ba3-e578-4f9f-bafd-7f9faefb39ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:43.019433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.019407 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-kube-api-access-cl5hb" pod "33684ba3-e578-4f9f-bafd-7f9faefb39ad" (UID: "33684ba3-e578-4f9f-bafd-7f9faefb39ad"). InnerVolumeSpecName "kube-api-access-cl5hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:43.042108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.042084 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv"] Oct 30 05:38:43.118481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.118406 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl5hb\" (UniqueName: \"kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-kube-api-access-cl5hb\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.118481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.118443 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.118481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.118459 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.118481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.118468 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.118481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.118476 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33684ba3-e578-4f9f-bafd-7f9faefb39ad-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.438493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.438409 6587 generic.go:356] "Generic (PLEG): container finished" podID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerID="4981368cf600221dad417e44ddbb9d6350f43e2f928833ddf1c045533d75f6f7" exitCode=0 Oct 30 05:38:43.438493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.438462 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" Oct 30 05:38:43.438987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.438488 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" event={"ID":"33684ba3-e578-4f9f-bafd-7f9faefb39ad","Type":"ContainerDied","Data":"4981368cf600221dad417e44ddbb9d6350f43e2f928833ddf1c045533d75f6f7"} Oct 30 05:38:43.438987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.438560 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n" event={"ID":"33684ba3-e578-4f9f-bafd-7f9faefb39ad","Type":"ContainerDied","Data":"b23a79f08e75994c4f75e16a565f5cfc328d9356f08f3e0ca04e784b155c3882"} Oct 30 05:38:43.438987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.438591 6587 scope.go:122] "RemoveContainer" containerID="4981368cf600221dad417e44ddbb9d6350f43e2f928833ddf1c045533d75f6f7" Oct 30 05:38:43.450789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.450770 6587 scope.go:122] "RemoveContainer" containerID="dbea4f0a03d664e8fedc1643d39031ffb12a762d11a6b57d5017dd93102c5568" Oct 30 05:38:43.462005 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.461989 6587 scope.go:122] "RemoveContainer" containerID="4981368cf600221dad417e44ddbb9d6350f43e2f928833ddf1c045533d75f6f7" Oct 30 05:38:43.462308 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:43.462292 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4981368cf600221dad417e44ddbb9d6350f43e2f928833ddf1c045533d75f6f7\": container with ID starting with 4981368cf600221dad417e44ddbb9d6350f43e2f928833ddf1c045533d75f6f7 not found: ID does not exist" containerID="4981368cf600221dad417e44ddbb9d6350f43e2f928833ddf1c045533d75f6f7" Oct 30 05:38:43.462391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.462317 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4981368cf600221dad417e44ddbb9d6350f43e2f928833ddf1c045533d75f6f7"} err="failed to get container status \"4981368cf600221dad417e44ddbb9d6350f43e2f928833ddf1c045533d75f6f7\": rpc error: code = NotFound desc = could not find container \"4981368cf600221dad417e44ddbb9d6350f43e2f928833ddf1c045533d75f6f7\": container with ID starting with 4981368cf600221dad417e44ddbb9d6350f43e2f928833ddf1c045533d75f6f7 not found: ID does not exist" Oct 30 05:38:43.462391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.462334 6587 scope.go:122] "RemoveContainer" containerID="dbea4f0a03d664e8fedc1643d39031ffb12a762d11a6b57d5017dd93102c5568" Oct 30 05:38:43.462643 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:43.462613 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbea4f0a03d664e8fedc1643d39031ffb12a762d11a6b57d5017dd93102c5568\": container with ID starting with dbea4f0a03d664e8fedc1643d39031ffb12a762d11a6b57d5017dd93102c5568 not found: ID does not exist" containerID="dbea4f0a03d664e8fedc1643d39031ffb12a762d11a6b57d5017dd93102c5568" Oct 30 05:38:43.462698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.462643 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbea4f0a03d664e8fedc1643d39031ffb12a762d11a6b57d5017dd93102c5568"} err="failed to get container status \"dbea4f0a03d664e8fedc1643d39031ffb12a762d11a6b57d5017dd93102c5568\": rpc error: code = NotFound desc = could not find container \"dbea4f0a03d664e8fedc1643d39031ffb12a762d11a6b57d5017dd93102c5568\": container with ID starting with dbea4f0a03d664e8fedc1643d39031ffb12a762d11a6b57d5017dd93102c5568 not found: ID does not exist" Oct 30 05:38:43.516649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.516617 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n"] Oct 30 05:38:43.523247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.523215 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n"] Oct 30 05:38:43.582318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.582287 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" path="/var/lib/kubelet/pods/33684ba3-e578-4f9f-bafd-7f9faefb39ad/volumes" Oct 30 05:38:43.582807 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.582792 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b792b9-3587-447d-8e6a-a3d21bc4131b" path="/var/lib/kubelet/pods/34b792b9-3587-447d-8e6a-a3d21bc4131b/volumes" Oct 30 05:38:43.583167 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.583153 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f145154-169d-4566-a3a9-8dec41416ceb" path="/var/lib/kubelet/pods/4f145154-169d-4566-a3a9-8dec41416ceb/volumes" Oct 30 05:38:43.583949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.583932 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba66369-7e9c-46a7-ba63-2d08bc3187c5" path="/var/lib/kubelet/pods/7ba66369-7e9c-46a7-ba63-2d08bc3187c5/volumes" Oct 30 05:38:43.584377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:43.584349 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d57116f6-cce0-4902-8619-89c358dcab6f" path="/var/lib/kubelet/pods/d57116f6-cce0-4902-8619-89c358dcab6f/volumes" Oct 30 05:38:48.755589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:48.755503 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4"] Oct 30 05:38:48.792874 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:48.792841 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72"] Oct 30 05:38:50.068429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.065418 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs"] Oct 30 05:38:50.093411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.089766 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2690b27e-vdxcs"] Oct 30 05:38:50.785052 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.785009 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx"] Oct 30 05:38:50.785766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.785741 6587 cpu_manager.go:414] "containerMap: removing container" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" Oct 30 05:38:50.785766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.785765 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" Oct 30 05:38:50.785974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.785776 6587 cpu_manager.go:414] "containerMap: removing container" podUID="4f145154-169d-4566-a3a9-8dec41416ceb" containerName="payload-processing" Oct 30 05:38:50.785974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.785784 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f145154-169d-4566-a3a9-8dec41416ceb" containerName="payload-processing" Oct 30 05:38:50.785974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.785803 6587 cpu_manager.go:414] "containerMap: removing container" podUID="29fbeaee-26a9-400f-a520-2c55782d2fc4" containerName="revoke-keys" Oct 30 05:38:50.785974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.785809 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fbeaee-26a9-400f-a520-2c55782d2fc4" containerName="revoke-keys" Oct 30 05:38:50.785974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.785822 6587 cpu_manager.go:414] "containerMap: removing container" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" Oct 30 05:38:50.785974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.785829 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" Oct 30 05:38:50.785974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.785866 6587 cpu_manager.go:414] "containerMap: removing container" podUID="34b792b9-3587-447d-8e6a-a3d21bc4131b" containerName="maas-api" Oct 30 05:38:50.785974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.785873 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b792b9-3587-447d-8e6a-a3d21bc4131b" containerName="maas-api" Oct 30 05:38:50.786410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.786027 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" Oct 30 05:38:50.786410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.786040 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f145154-169d-4566-a3a9-8dec41416ceb" containerName="payload-processing" Oct 30 05:38:50.786410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.786051 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="29fbeaee-26a9-400f-a520-2c55782d2fc4" containerName="revoke-keys" Oct 30 05:38:50.786410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.786062 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="33684ba3-e578-4f9f-bafd-7f9faefb39ad" containerName="istio-proxy" Oct 30 05:38:50.786410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.786082 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b792b9-3587-447d-8e6a-a3d21bc4131b" containerName="maas-api" Oct 30 05:38:50.792069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.792040 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" Oct 30 05:38:50.809064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.809041 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx"] Oct 30 05:38:50.816731 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.816708 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:38:50.817026 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.816971 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:38:50.827273 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.827247 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx\" (UID: \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" Oct 30 05:38:50.827486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.827456 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx\" (UID: \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" Oct 30 05:38:50.827600 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.827511 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dckbh\" (UniqueName: \"kubernetes.io/projected/b42c0cb6-4501-45ad-b0a2-5af41a49f204-kube-api-access-dckbh\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx\" (UID: \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" Oct 30 05:38:50.837989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.837944 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:38:50.837989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.837947 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:38:50.930325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.929054 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx\" (UID: \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" Oct 30 05:38:50.930325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.929183 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx\" (UID: \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" Oct 30 05:38:50.930325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.929237 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dckbh\" (UniqueName: \"kubernetes.io/projected/b42c0cb6-4501-45ad-b0a2-5af41a49f204-kube-api-access-dckbh\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx\" (UID: \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" Oct 30 05:38:50.930325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.929961 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx\" (UID: \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" Oct 30 05:38:50.931579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.931551 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx\" (UID: \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" Oct 30 05:38:50.979277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:50.979205 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dckbh\" (UniqueName: \"kubernetes.io/projected/b42c0cb6-4501-45ad-b0a2-5af41a49f204-kube-api-access-dckbh\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx\" (UID: \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" Oct 30 05:38:51.106847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:51.106808 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" Oct 30 05:38:51.449617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:51.449590 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx"] Oct 30 05:38:51.452717 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:38:51.452687 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42c0cb6_4501_45ad_b0a2_5af41a49f204.slice/crio-4d4a77ee9355c9b178c5f26c112321a431c522e962d3d97cb0e73c788b1a9832 WatchSource:0}: Error finding container 4d4a77ee9355c9b178c5f26c112321a431c522e962d3d97cb0e73c788b1a9832: Status 404 returned error can't find the container with id 4d4a77ee9355c9b178c5f26c112321a431c522e962d3d97cb0e73c788b1a9832 Oct 30 05:38:51.511335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:51.511303 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" event={"ID":"b42c0cb6-4501-45ad-b0a2-5af41a49f204","Type":"ContainerStarted","Data":"4d4a77ee9355c9b178c5f26c112321a431c522e962d3d97cb0e73c788b1a9832"} Oct 30 05:38:51.581829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:51.581795 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="009cdc21-8578-4132-bdfa-55d149c7984b" path="/var/lib/kubelet/pods/009cdc21-8578-4132-bdfa-55d149c7984b/volumes" Oct 30 05:38:52.143373 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.143327 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w"] Oct 30 05:38:52.143756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.143561 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" containerID="cri-o://647fc8243b3bc774146f26476db37b07471a5420259af7c0066a45e7d503d6e8" gracePeriod=30 Oct 30 05:38:52.520699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.520659 6587 generic.go:356] "Generic (PLEG): container finished" podID="b42c0cb6-4501-45ad-b0a2-5af41a49f204" containerID="ade5c365169b0ad6ecb2522146adcbe80dacaff3dcf02c25bfd5b0015e4613f2" exitCode=0 Oct 30 05:38:52.520909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.520711 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" event={"ID":"b42c0cb6-4501-45ad-b0a2-5af41a49f204","Type":"ContainerDied","Data":"ade5c365169b0ad6ecb2522146adcbe80dacaff3dcf02c25bfd5b0015e4613f2"} Oct 30 05:38:52.674611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.674529 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s"] Oct 30 05:38:52.679866 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.679841 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" Oct 30 05:38:52.698403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.698342 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s"] Oct 30 05:38:52.758205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.758165 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096248dd-a845-4351-8b36-03d6178fac68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s\" (UID: \"096248dd-a845-4351-8b36-03d6178fac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" Oct 30 05:38:52.758437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.758317 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096248dd-a845-4351-8b36-03d6178fac68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s\" (UID: \"096248dd-a845-4351-8b36-03d6178fac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" Oct 30 05:38:52.758437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.758411 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-775rk\" (UniqueName: \"kubernetes.io/projected/096248dd-a845-4351-8b36-03d6178fac68-kube-api-access-775rk\") pod \"maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s\" (UID: \"096248dd-a845-4351-8b36-03d6178fac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" Oct 30 05:38:52.859633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.859594 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096248dd-a845-4351-8b36-03d6178fac68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s\" (UID: \"096248dd-a845-4351-8b36-03d6178fac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" Oct 30 05:38:52.859808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.859725 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096248dd-a845-4351-8b36-03d6178fac68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s\" (UID: \"096248dd-a845-4351-8b36-03d6178fac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" Oct 30 05:38:52.859858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.859814 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-775rk\" (UniqueName: \"kubernetes.io/projected/096248dd-a845-4351-8b36-03d6178fac68-kube-api-access-775rk\") pod \"maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s\" (UID: \"096248dd-a845-4351-8b36-03d6178fac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" Oct 30 05:38:52.860345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.860315 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096248dd-a845-4351-8b36-03d6178fac68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s\" (UID: \"096248dd-a845-4351-8b36-03d6178fac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" Oct 30 05:38:52.862103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.862080 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096248dd-a845-4351-8b36-03d6178fac68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s\" (UID: \"096248dd-a845-4351-8b36-03d6178fac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" Oct 30 05:38:52.880737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.880714 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-775rk\" (UniqueName: \"kubernetes.io/projected/096248dd-a845-4351-8b36-03d6178fac68-kube-api-access-775rk\") pod \"maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s\" (UID: \"096248dd-a845-4351-8b36-03d6178fac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" Oct 30 05:38:52.994107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:52.994030 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" Oct 30 05:38:53.305686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.305660 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s"] Oct 30 05:38:53.307943 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:38:53.307911 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096248dd_a845_4351_8b36_03d6178fac68.slice/crio-fc928e6b56792764c5054a50003100d86f227b5af73244c56f1942630ac113b5 WatchSource:0}: Error finding container fc928e6b56792764c5054a50003100d86f227b5af73244c56f1942630ac113b5: Status 404 returned error can't find the container with id fc928e6b56792764c5054a50003100d86f227b5af73244c56f1942630ac113b5 Oct 30 05:38:53.529370 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.529324 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" event={"ID":"096248dd-a845-4351-8b36-03d6178fac68","Type":"ContainerStarted","Data":"fc928e6b56792764c5054a50003100d86f227b5af73244c56f1942630ac113b5"} Oct 30 05:38:53.698340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.698311 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" Oct 30 05:38:53.756093 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.756036 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" podUID="34ed4f94-1796-44b1-9de3-60065c919d4a" containerName="payload-processing" containerID="cri-o://50603cc7d711a8eea11b0e1bad9f9eb929fe6e45d78bce9804f6cd44dfd912ac" gracePeriod=25 Oct 30 05:38:53.774138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.774109 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-xks-ca\") pod \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\" (UID: \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\") " Oct 30 05:38:53.774286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.774161 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-openshift-ca\") pod \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\" (UID: \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\") " Oct 30 05:38:53.774286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.774205 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42c0cb6-4501-45ad-b0a2-5af41a49f204-kube-api-access-dckbh\" (UniqueName: \"kubernetes.io/projected/b42c0cb6-4501-45ad-b0a2-5af41a49f204-kube-api-access-dckbh\") pod \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\" (UID: \"b42c0cb6-4501-45ad-b0a2-5af41a49f204\") " Oct 30 05:38:53.774627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.774602 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-openshift-ca" pod "b42c0cb6-4501-45ad-b0a2-5af41a49f204" (UID: "b42c0cb6-4501-45ad-b0a2-5af41a49f204"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:53.774842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.774822 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:53.776329 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.776301 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42c0cb6-4501-45ad-b0a2-5af41a49f204-kube-api-access-dckbh" pod "b42c0cb6-4501-45ad-b0a2-5af41a49f204" (UID: "b42c0cb6-4501-45ad-b0a2-5af41a49f204"). InnerVolumeSpecName "kube-api-access-dckbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:53.776436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.776375 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-xks-ca" pod "b42c0cb6-4501-45ad-b0a2-5af41a49f204" (UID: "b42c0cb6-4501-45ad-b0a2-5af41a49f204"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:53.793486 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.793449 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" podUID="461cc202-2197-4c68-b5a7-734fb8e27072" containerName="payload-pre-processing" containerID="cri-o://1f9a94e78ff8d0e786adf74d5d0cae559b687bfd6b3acc8c444a7688b9b8dcbb" gracePeriod=25 Oct 30 05:38:53.876837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.876741 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42c0cb6-4501-45ad-b0a2-5af41a49f204-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:53.876837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:53.876787 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dckbh\" (UniqueName: \"kubernetes.io/projected/b42c0cb6-4501-45ad-b0a2-5af41a49f204-kube-api-access-dckbh\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:54.543445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:54.540754 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" Oct 30 05:38:54.543445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:54.540750 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx" event={"ID":"b42c0cb6-4501-45ad-b0a2-5af41a49f204","Type":"ContainerDied","Data":"4d4a77ee9355c9b178c5f26c112321a431c522e962d3d97cb0e73c788b1a9832"} Oct 30 05:38:54.543445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:54.540919 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d4a77ee9355c9b178c5f26c112321a431c522e962d3d97cb0e73c788b1a9832" Oct 30 05:38:54.543445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:54.542871 6587 generic.go:356] "Generic (PLEG): container finished" podID="096248dd-a845-4351-8b36-03d6178fac68" containerID="d16ea14e4a943c3eb5e63620243b581345e681f20fd38a4454a0fa4776a2b5ec" exitCode=0 Oct 30 05:38:54.543445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:54.542909 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" event={"ID":"096248dd-a845-4351-8b36-03d6178fac68","Type":"ContainerDied","Data":"d16ea14e4a943c3eb5e63620243b581345e681f20fd38a4454a0fa4776a2b5ec"} Oct 30 05:38:55.760836 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.760806 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" Oct 30 05:38:55.812264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.812176 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096248dd-a845-4351-8b36-03d6178fac68-kube-api-access-775rk\" (UniqueName: \"kubernetes.io/projected/096248dd-a845-4351-8b36-03d6178fac68-kube-api-access-775rk\") pod \"096248dd-a845-4351-8b36-03d6178fac68\" (UID: \"096248dd-a845-4351-8b36-03d6178fac68\") " Oct 30 05:38:55.812501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.812461 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/096248dd-a845-4351-8b36-03d6178fac68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096248dd-a845-4351-8b36-03d6178fac68-maas-api-xks-ca\") pod \"096248dd-a845-4351-8b36-03d6178fac68\" (UID: \"096248dd-a845-4351-8b36-03d6178fac68\") " Oct 30 05:38:55.812586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.812563 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096248dd-a845-4351-8b36-03d6178fac68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096248dd-a845-4351-8b36-03d6178fac68-maas-api-openshift-ca\") pod \"096248dd-a845-4351-8b36-03d6178fac68\" (UID: \"096248dd-a845-4351-8b36-03d6178fac68\") " Oct 30 05:38:55.815975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.815944 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096248dd-a845-4351-8b36-03d6178fac68-maas-api-openshift-ca" pod "096248dd-a845-4351-8b36-03d6178fac68" (UID: "096248dd-a845-4351-8b36-03d6178fac68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:55.816260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.816231 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096248dd-a845-4351-8b36-03d6178fac68-kube-api-access-775rk" pod "096248dd-a845-4351-8b36-03d6178fac68" (UID: "096248dd-a845-4351-8b36-03d6178fac68"). InnerVolumeSpecName "kube-api-access-775rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:55.824966 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.824932 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096248dd-a845-4351-8b36-03d6178fac68-maas-api-xks-ca" pod "096248dd-a845-4351-8b36-03d6178fac68" (UID: "096248dd-a845-4351-8b36-03d6178fac68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:55.863352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.863320 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7"] Oct 30 05:38:55.863973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.863957 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b42c0cb6-4501-45ad-b0a2-5af41a49f204" containerName="revoke-keys" Oct 30 05:38:55.864022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.863975 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42c0cb6-4501-45ad-b0a2-5af41a49f204" containerName="revoke-keys" Oct 30 05:38:55.864022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.863993 6587 cpu_manager.go:414] "containerMap: removing container" podUID="096248dd-a845-4351-8b36-03d6178fac68" containerName="revoke-keys" Oct 30 05:38:55.864022 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.864000 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096248dd-a845-4351-8b36-03d6178fac68" containerName="revoke-keys" Oct 30 05:38:55.864125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.864107 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="096248dd-a845-4351-8b36-03d6178fac68" containerName="revoke-keys" Oct 30 05:38:55.864161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.864126 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42c0cb6-4501-45ad-b0a2-5af41a49f204" containerName="revoke-keys" Oct 30 05:38:55.868028 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.867999 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" Oct 30 05:38:55.905302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.905268 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7"] Oct 30 05:38:55.918286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.918233 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd8d7\" (UniqueName: \"kubernetes.io/projected/d677d710-645b-40d3-9222-24ca3d7047e5-kube-api-access-cd8d7\") pod \"maas-api-revoke-sub-a0f9e1c2-6rst7\" (UID: \"d677d710-645b-40d3-9222-24ca3d7047e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" Oct 30 05:38:55.918545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.918486 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0f9e1c2-6rst7\" (UID: \"d677d710-645b-40d3-9222-24ca3d7047e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" Oct 30 05:38:55.918545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.918527 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0f9e1c2-6rst7\" (UID: \"d677d710-645b-40d3-9222-24ca3d7047e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" Oct 30 05:38:55.918775 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.918753 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096248dd-a845-4351-8b36-03d6178fac68-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:55.918853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.918780 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-775rk\" (UniqueName: \"kubernetes.io/projected/096248dd-a845-4351-8b36-03d6178fac68-kube-api-access-775rk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:55.918853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.918797 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096248dd-a845-4351-8b36-03d6178fac68-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:55.938450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.925114 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8"] Oct 30 05:38:55.940219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.940186 6587 patch_prober.go:29] interesting pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:38:55.940403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.940238 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:38:55.942208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.941742 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" Oct 30 05:38:55.960007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:55.959941 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8"] Oct 30 05:38:56.022979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.021351 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0f9e1c2-6rst7\" (UID: \"d677d710-645b-40d3-9222-24ca3d7047e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" Oct 30 05:38:56.022979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.021422 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0f9e1c2-6rst7\" (UID: \"d677d710-645b-40d3-9222-24ca3d7047e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" Oct 30 05:38:56.022979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.021502 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dd6j\" (UniqueName: \"kubernetes.io/projected/730e3261-6722-43fd-8509-19129cbe469f-kube-api-access-4dd6j\") pod \"maas-api-revoke-sub-24bb6c74-5bkx8\" (UID: \"730e3261-6722-43fd-8509-19129cbe469f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" Oct 30 05:38:56.022979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.021615 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730e3261-6722-43fd-8509-19129cbe469f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24bb6c74-5bkx8\" (UID: \"730e3261-6722-43fd-8509-19129cbe469f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" Oct 30 05:38:56.022979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.021712 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730e3261-6722-43fd-8509-19129cbe469f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24bb6c74-5bkx8\" (UID: \"730e3261-6722-43fd-8509-19129cbe469f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" Oct 30 05:38:56.022979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.021767 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd8d7\" (UniqueName: \"kubernetes.io/projected/d677d710-645b-40d3-9222-24ca3d7047e5-kube-api-access-cd8d7\") pod \"maas-api-revoke-sub-a0f9e1c2-6rst7\" (UID: \"d677d710-645b-40d3-9222-24ca3d7047e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" Oct 30 05:38:56.022979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.022907 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0f9e1c2-6rst7\" (UID: \"d677d710-645b-40d3-9222-24ca3d7047e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" Oct 30 05:38:56.029897 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:56.029858 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:38:56.030060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.030042 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0f9e1c2-6rst7\" (UID: \"d677d710-645b-40d3-9222-24ca3d7047e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" Oct 30 05:38:56.043471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.043441 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd8d7\" (UniqueName: \"kubernetes.io/projected/d677d710-645b-40d3-9222-24ca3d7047e5-kube-api-access-cd8d7\") pod \"maas-api-revoke-sub-a0f9e1c2-6rst7\" (UID: \"d677d710-645b-40d3-9222-24ca3d7047e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" Oct 30 05:38:56.125427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.125301 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dd6j\" (UniqueName: \"kubernetes.io/projected/730e3261-6722-43fd-8509-19129cbe469f-kube-api-access-4dd6j\") pod \"maas-api-revoke-sub-24bb6c74-5bkx8\" (UID: \"730e3261-6722-43fd-8509-19129cbe469f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" Oct 30 05:38:56.125612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.125441 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730e3261-6722-43fd-8509-19129cbe469f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24bb6c74-5bkx8\" (UID: \"730e3261-6722-43fd-8509-19129cbe469f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" Oct 30 05:38:56.125612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.125533 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730e3261-6722-43fd-8509-19129cbe469f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24bb6c74-5bkx8\" (UID: \"730e3261-6722-43fd-8509-19129cbe469f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" Oct 30 05:38:56.126262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.126230 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730e3261-6722-43fd-8509-19129cbe469f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24bb6c74-5bkx8\" (UID: \"730e3261-6722-43fd-8509-19129cbe469f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" Oct 30 05:38:56.128492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.128467 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730e3261-6722-43fd-8509-19129cbe469f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24bb6c74-5bkx8\" (UID: \"730e3261-6722-43fd-8509-19129cbe469f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" Oct 30 05:38:56.149020 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.148994 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dd6j\" (UniqueName: \"kubernetes.io/projected/730e3261-6722-43fd-8509-19129cbe469f-kube-api-access-4dd6j\") pod \"maas-api-revoke-sub-24bb6c74-5bkx8\" (UID: \"730e3261-6722-43fd-8509-19129cbe469f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" Oct 30 05:38:56.183595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.183565 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" Oct 30 05:38:56.284219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.284184 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" Oct 30 05:38:56.566287 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.566253 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" Oct 30 05:38:56.566498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.566280 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s" event={"ID":"096248dd-a845-4351-8b36-03d6178fac68","Type":"ContainerDied","Data":"fc928e6b56792764c5054a50003100d86f227b5af73244c56f1942630ac113b5"} Oct 30 05:38:56.566498 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.566338 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc928e6b56792764c5054a50003100d86f227b5af73244c56f1942630ac113b5" Oct 30 05:38:56.568331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.568308 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7"] Oct 30 05:38:56.570935 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:38:56.570907 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd677d710_645b_40d3_9222_24ca3d7047e5.slice/crio-326212e74c2530e68ae44fd1c3b40acee38250c8feeb47e82d8581877aa87d81 WatchSource:0}: Error finding container 326212e74c2530e68ae44fd1c3b40acee38250c8feeb47e82d8581877aa87d81: Status 404 returned error can't find the container with id 326212e74c2530e68ae44fd1c3b40acee38250c8feeb47e82d8581877aa87d81 Oct 30 05:38:56.628898 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.628870 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8"] Oct 30 05:38:56.630840 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:38:56.630810 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730e3261_6722_43fd_8509_19129cbe469f.slice/crio-6217179fb34f0f0ff22553299bbba2ffa7ae305b03aebc2447dc01a270810753 WatchSource:0}: Error finding container 6217179fb34f0f0ff22553299bbba2ffa7ae305b03aebc2447dc01a270810753: Status 404 returned error can't find the container with id 6217179fb34f0f0ff22553299bbba2ffa7ae305b03aebc2447dc01a270810753 Oct 30 05:38:56.795195 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.795153 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm"] Oct 30 05:38:56.798386 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:56.798344 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b21507-rkhcm"] Oct 30 05:38:57.431019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.430993 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:38:57.566231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.566204 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-certs\") pod \"f360336e-96e3-41c3-b483-aecf620a959f\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " Oct 30 05:38:57.566896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.566241 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-kube-api-access-qjmgz\" (UniqueName: \"kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-kube-api-access-qjmgz\") pod \"f360336e-96e3-41c3-b483-aecf620a959f\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " Oct 30 05:38:57.566896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.566301 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f360336e-96e3-41c3-b483-aecf620a959f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f360336e-96e3-41c3-b483-aecf620a959f-istio-podinfo\") pod \"f360336e-96e3-41c3-b483-aecf620a959f\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " Oct 30 05:38:57.566896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.566423 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-socket\") pod \"f360336e-96e3-41c3-b483-aecf620a959f\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " Oct 30 05:38:57.566896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.566489 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f360336e-96e3-41c3-b483-aecf620a959f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f360336e-96e3-41c3-b483-aecf620a959f-istiod-ca-cert\") pod \"f360336e-96e3-41c3-b483-aecf620a959f\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " Oct 30 05:38:57.566896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.566515 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-certs" pod "f360336e-96e3-41c3-b483-aecf620a959f" (UID: "f360336e-96e3-41c3-b483-aecf620a959f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:57.566896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.566532 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-istio-token\" (UniqueName: \"kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-istio-token\") pod \"f360336e-96e3-41c3-b483-aecf620a959f\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " Oct 30 05:38:57.566896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.566595 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-credential-socket\") pod \"f360336e-96e3-41c3-b483-aecf620a959f\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " Oct 30 05:38:57.566896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.566678 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-envoy\") pod \"f360336e-96e3-41c3-b483-aecf620a959f\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " Oct 30 05:38:57.566896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.566722 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-data\") pod \"f360336e-96e3-41c3-b483-aecf620a959f\" (UID: \"f360336e-96e3-41c3-b483-aecf620a959f\") " Oct 30 05:38:57.566896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.566874 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-socket" pod "f360336e-96e3-41c3-b483-aecf620a959f" (UID: "f360336e-96e3-41c3-b483-aecf620a959f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:57.567831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.566931 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f360336e-96e3-41c3-b483-aecf620a959f-istiod-ca-cert" pod "f360336e-96e3-41c3-b483-aecf620a959f" (UID: "f360336e-96e3-41c3-b483-aecf620a959f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:57.567831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.567152 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-credential-socket" pod "f360336e-96e3-41c3-b483-aecf620a959f" (UID: "f360336e-96e3-41c3-b483-aecf620a959f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:57.567831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.567172 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-data" pod "f360336e-96e3-41c3-b483-aecf620a959f" (UID: "f360336e-96e3-41c3-b483-aecf620a959f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:57.567831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.567504 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f360336e-96e3-41c3-b483-aecf620a959f-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:57.567831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.567524 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:57.567831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.567538 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:57.567831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.567551 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:57.567831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.567563 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:57.569262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.569135 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f360336e-96e3-41c3-b483-aecf620a959f-istio-podinfo" pod "f360336e-96e3-41c3-b483-aecf620a959f" (UID: "f360336e-96e3-41c3-b483-aecf620a959f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:38:57.569403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.569310 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-kube-api-access-qjmgz" pod "f360336e-96e3-41c3-b483-aecf620a959f" (UID: "f360336e-96e3-41c3-b483-aecf620a959f"). InnerVolumeSpecName "kube-api-access-qjmgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:57.569539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.569514 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-envoy" pod "f360336e-96e3-41c3-b483-aecf620a959f" (UID: "f360336e-96e3-41c3-b483-aecf620a959f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:57.569846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.569827 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-istio-token" pod "f360336e-96e3-41c3-b483-aecf620a959f" (UID: "f360336e-96e3-41c3-b483-aecf620a959f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:57.580978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.580955 6587 generic.go:356] "Generic (PLEG): container finished" podID="d677d710-645b-40d3-9222-24ca3d7047e5" containerID="e1ee558e4e1a95327e3d12f841420de57018dbdc21dce616b0cf3df12c365a69" exitCode=0 Oct 30 05:38:57.582943 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.582916 6587 generic.go:356] "Generic (PLEG): container finished" podID="f360336e-96e3-41c3-b483-aecf620a959f" containerID="647fc8243b3bc774146f26476db37b07471a5420259af7c0066a45e7d503d6e8" exitCode=0 Oct 30 05:38:57.583108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.583029 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" Oct 30 05:38:57.584644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.584620 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43880e9c-de88-47e4-ae55-b1f1ea8bf329" path="/var/lib/kubelet/pods/43880e9c-de88-47e4-ae55-b1f1ea8bf329/volumes" Oct 30 05:38:57.585157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.585135 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" event={"ID":"730e3261-6722-43fd-8509-19129cbe469f","Type":"ContainerStarted","Data":"d3bd2c847f8fd6d76501e7a623f37e303f37c64583986c015c637f3b7c5380e3"} Oct 30 05:38:57.585249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.585178 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" event={"ID":"730e3261-6722-43fd-8509-19129cbe469f","Type":"ContainerStarted","Data":"6217179fb34f0f0ff22553299bbba2ffa7ae305b03aebc2447dc01a270810753"} Oct 30 05:38:57.585249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.585202 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" event={"ID":"d677d710-645b-40d3-9222-24ca3d7047e5","Type":"ContainerDied","Data":"e1ee558e4e1a95327e3d12f841420de57018dbdc21dce616b0cf3df12c365a69"} Oct 30 05:38:57.585249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.585228 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" event={"ID":"d677d710-645b-40d3-9222-24ca3d7047e5","Type":"ContainerStarted","Data":"326212e74c2530e68ae44fd1c3b40acee38250c8feeb47e82d8581877aa87d81"} Oct 30 05:38:57.585460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.585249 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" event={"ID":"f360336e-96e3-41c3-b483-aecf620a959f","Type":"ContainerDied","Data":"647fc8243b3bc774146f26476db37b07471a5420259af7c0066a45e7d503d6e8"} Oct 30 05:38:57.585460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.585289 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w" event={"ID":"f360336e-96e3-41c3-b483-aecf620a959f","Type":"ContainerDied","Data":"928050fd4fd1667822b4fb15f8fde6cde08cd74ad025926c07b88c94c2725a8f"} Oct 30 05:38:57.585460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.585322 6587 scope.go:122] "RemoveContainer" containerID="647fc8243b3bc774146f26476db37b07471a5420259af7c0066a45e7d503d6e8" Oct 30 05:38:57.596709 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.596690 6587 scope.go:122] "RemoveContainer" containerID="fcbc9da06e47375d5ca044c36e00ab5d6c3306744c2444d01175dcd759dea8e9" Oct 30 05:38:57.606931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.606910 6587 scope.go:122] "RemoveContainer" containerID="647fc8243b3bc774146f26476db37b07471a5420259af7c0066a45e7d503d6e8" Oct 30 05:38:57.607224 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:57.607196 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647fc8243b3bc774146f26476db37b07471a5420259af7c0066a45e7d503d6e8\": container with ID starting with 647fc8243b3bc774146f26476db37b07471a5420259af7c0066a45e7d503d6e8 not found: ID does not exist" containerID="647fc8243b3bc774146f26476db37b07471a5420259af7c0066a45e7d503d6e8" Oct 30 05:38:57.607322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.607225 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647fc8243b3bc774146f26476db37b07471a5420259af7c0066a45e7d503d6e8"} err="failed to get container status \"647fc8243b3bc774146f26476db37b07471a5420259af7c0066a45e7d503d6e8\": rpc error: code = NotFound desc = could not find container \"647fc8243b3bc774146f26476db37b07471a5420259af7c0066a45e7d503d6e8\": container with ID starting with 647fc8243b3bc774146f26476db37b07471a5420259af7c0066a45e7d503d6e8 not found: ID does not exist" Oct 30 05:38:57.607322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.607246 6587 scope.go:122] "RemoveContainer" containerID="fcbc9da06e47375d5ca044c36e00ab5d6c3306744c2444d01175dcd759dea8e9" Oct 30 05:38:57.607528 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:57.607506 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbc9da06e47375d5ca044c36e00ab5d6c3306744c2444d01175dcd759dea8e9\": container with ID starting with fcbc9da06e47375d5ca044c36e00ab5d6c3306744c2444d01175dcd759dea8e9 not found: ID does not exist" containerID="fcbc9da06e47375d5ca044c36e00ab5d6c3306744c2444d01175dcd759dea8e9" Oct 30 05:38:57.607578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.607533 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbc9da06e47375d5ca044c36e00ab5d6c3306744c2444d01175dcd759dea8e9"} err="failed to get container status \"fcbc9da06e47375d5ca044c36e00ab5d6c3306744c2444d01175dcd759dea8e9\": rpc error: code = NotFound desc = could not find container \"fcbc9da06e47375d5ca044c36e00ab5d6c3306744c2444d01175dcd759dea8e9\": container with ID starting with fcbc9da06e47375d5ca044c36e00ab5d6c3306744c2444d01175dcd759dea8e9 not found: ID does not exist" Oct 30 05:38:57.630729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.630686 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" podStartSLOduration=2.63067335 podStartE2EDuration="2.63067335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:57.629677362 +0000 UTC m=+3262.157444147" watchObservedRunningTime="2026-10-30 05:38:57.63067335 +0000 UTC m=+3262.158440135" Oct 30 05:38:57.669012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.668987 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f360336e-96e3-41c3-b483-aecf620a959f-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:57.669012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.669011 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:57.669157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.669028 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f360336e-96e3-41c3-b483-aecf620a959f-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:57.669157 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.669040 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjmgz\" (UniqueName: \"kubernetes.io/projected/f360336e-96e3-41c3-b483-aecf620a959f-kube-api-access-qjmgz\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:57.715501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.715470 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w"] Oct 30 05:38:57.721687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:57.721663 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w"] Oct 30 05:38:58.062939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.062876 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" containerID="cri-o://cf2e7e44c211ec48b0c2aba944ded4496eceba945be1c6907788b4eab4aa6a84" gracePeriod=45 Oct 30 05:38:58.368750 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.368722 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:38:58.480927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.480895 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-home\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-home\") pod \"3030b121-7591-4c40-b891-52a6009228c9\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " Oct 30 05:38:58.481112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.480944 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-tmp-dir\") pod \"3030b121-7591-4c40-b891-52a6009228c9\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " Oct 30 05:38:58.481112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.480986 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3030b121-7591-4c40-b891-52a6009228c9-kube-api-access-47zn2\" (UniqueName: \"kubernetes.io/projected/3030b121-7591-4c40-b891-52a6009228c9-kube-api-access-47zn2\") pod \"3030b121-7591-4c40-b891-52a6009228c9\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " Oct 30 05:38:58.481112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.481083 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-model-cache\") pod \"3030b121-7591-4c40-b891-52a6009228c9\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " Oct 30 05:38:58.481305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.481225 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-home" pod "3030b121-7591-4c40-b891-52a6009228c9" (UID: "3030b121-7591-4c40-b891-52a6009228c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:58.481305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.481269 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-dshm\") pod \"3030b121-7591-4c40-b891-52a6009228c9\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " Oct 30 05:38:58.481444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.481318 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3030b121-7591-4c40-b891-52a6009228c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/3030b121-7591-4c40-b891-52a6009228c9-tls-certs\") pod \"3030b121-7591-4c40-b891-52a6009228c9\" (UID: \"3030b121-7591-4c40-b891-52a6009228c9\") " Oct 30 05:38:58.481444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.481343 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-tmp-dir" pod "3030b121-7591-4c40-b891-52a6009228c9" (UID: "3030b121-7591-4c40-b891-52a6009228c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:58.481562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.481470 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-model-cache" pod "3030b121-7591-4c40-b891-52a6009228c9" (UID: "3030b121-7591-4c40-b891-52a6009228c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:58.481895 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.481871 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:58.481985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.481926 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:58.481985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.481943 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:58.483930 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.483898 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3030b121-7591-4c40-b891-52a6009228c9-kube-api-access-47zn2" pod "3030b121-7591-4c40-b891-52a6009228c9" (UID: "3030b121-7591-4c40-b891-52a6009228c9"). InnerVolumeSpecName "kube-api-access-47zn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:58.484067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.484003 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-dshm" pod "3030b121-7591-4c40-b891-52a6009228c9" (UID: "3030b121-7591-4c40-b891-52a6009228c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:58.484194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.484168 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3030b121-7591-4c40-b891-52a6009228c9-tls-certs" pod "3030b121-7591-4c40-b891-52a6009228c9" (UID: "3030b121-7591-4c40-b891-52a6009228c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:58.583180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.583098 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3030b121-7591-4c40-b891-52a6009228c9-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:58.583180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.583127 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3030b121-7591-4c40-b891-52a6009228c9-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:58.583180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.583136 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47zn2\" (UniqueName: \"kubernetes.io/projected/3030b121-7591-4c40-b891-52a6009228c9-kube-api-access-47zn2\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:58.595229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.595202 6587 generic.go:356] "Generic (PLEG): container finished" podID="3030b121-7591-4c40-b891-52a6009228c9" containerID="cf2e7e44c211ec48b0c2aba944ded4496eceba945be1c6907788b4eab4aa6a84" exitCode=0 Oct 30 05:38:58.595414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.595263 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" Oct 30 05:38:58.595414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.595292 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" event={"ID":"3030b121-7591-4c40-b891-52a6009228c9","Type":"ContainerDied","Data":"cf2e7e44c211ec48b0c2aba944ded4496eceba945be1c6907788b4eab4aa6a84"} Oct 30 05:38:58.595414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.595333 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv" event={"ID":"3030b121-7591-4c40-b891-52a6009228c9","Type":"ContainerDied","Data":"a66318250e3ac1f13be2cc419791987a691f6b71f25d7d6fd6e1f18bfc7857d1"} Oct 30 05:38:58.595414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.595386 6587 scope.go:122] "RemoveContainer" containerID="cf2e7e44c211ec48b0c2aba944ded4496eceba945be1c6907788b4eab4aa6a84" Oct 30 05:38:58.597565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.597541 6587 generic.go:356] "Generic (PLEG): container finished" podID="730e3261-6722-43fd-8509-19129cbe469f" containerID="d3bd2c847f8fd6d76501e7a623f37e303f37c64583986c015c637f3b7c5380e3" exitCode=0 Oct 30 05:38:58.597869 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.597579 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" event={"ID":"730e3261-6722-43fd-8509-19129cbe469f","Type":"ContainerDied","Data":"d3bd2c847f8fd6d76501e7a623f37e303f37c64583986c015c637f3b7c5380e3"} Oct 30 05:38:58.607732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.607716 6587 scope.go:122] "RemoveContainer" containerID="cf2e7e44c211ec48b0c2aba944ded4496eceba945be1c6907788b4eab4aa6a84" Oct 30 05:38:58.608010 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:38:58.607989 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2e7e44c211ec48b0c2aba944ded4496eceba945be1c6907788b4eab4aa6a84\": container with ID starting with cf2e7e44c211ec48b0c2aba944ded4496eceba945be1c6907788b4eab4aa6a84 not found: ID does not exist" containerID="cf2e7e44c211ec48b0c2aba944ded4496eceba945be1c6907788b4eab4aa6a84" Oct 30 05:38:58.608106 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.608020 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2e7e44c211ec48b0c2aba944ded4496eceba945be1c6907788b4eab4aa6a84"} err="failed to get container status \"cf2e7e44c211ec48b0c2aba944ded4496eceba945be1c6907788b4eab4aa6a84\": rpc error: code = NotFound desc = could not find container \"cf2e7e44c211ec48b0c2aba944ded4496eceba945be1c6907788b4eab4aa6a84\": container with ID starting with cf2e7e44c211ec48b0c2aba944ded4496eceba945be1c6907788b4eab4aa6a84 not found: ID does not exist" Oct 30 05:38:58.693705 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.693672 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv"] Oct 30 05:38:58.702727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.702695 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-397179/distinct-2-w2-397179-kserve-7cc95f4984-qtszv"] Oct 30 05:38:58.781981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.781948 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" Oct 30 05:38:58.888599 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.888501 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-openshift-ca\") pod \"d677d710-645b-40d3-9222-24ca3d7047e5\" (UID: \"d677d710-645b-40d3-9222-24ca3d7047e5\") " Oct 30 05:38:58.888788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.888639 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-xks-ca\") pod \"d677d710-645b-40d3-9222-24ca3d7047e5\" (UID: \"d677d710-645b-40d3-9222-24ca3d7047e5\") " Oct 30 05:38:58.888907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.888887 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d677d710-645b-40d3-9222-24ca3d7047e5-kube-api-access-cd8d7\" (UniqueName: \"kubernetes.io/projected/d677d710-645b-40d3-9222-24ca3d7047e5-kube-api-access-cd8d7\") pod \"d677d710-645b-40d3-9222-24ca3d7047e5\" (UID: \"d677d710-645b-40d3-9222-24ca3d7047e5\") " Oct 30 05:38:58.888979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.888952 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-openshift-ca" pod "d677d710-645b-40d3-9222-24ca3d7047e5" (UID: "d677d710-645b-40d3-9222-24ca3d7047e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:58.889328 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.889305 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:58.890984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.890957 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-xks-ca" pod "d677d710-645b-40d3-9222-24ca3d7047e5" (UID: "d677d710-645b-40d3-9222-24ca3d7047e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:58.890984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.890955 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d677d710-645b-40d3-9222-24ca3d7047e5-kube-api-access-cd8d7" pod "d677d710-645b-40d3-9222-24ca3d7047e5" (UID: "d677d710-645b-40d3-9222-24ca3d7047e5"). InnerVolumeSpecName "kube-api-access-cd8d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:58.990352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.990320 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd8d7\" (UniqueName: \"kubernetes.io/projected/d677d710-645b-40d3-9222-24ca3d7047e5-kube-api-access-cd8d7\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:58.990352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:58.990348 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d677d710-645b-40d3-9222-24ca3d7047e5-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:59.581860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:59.581828 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3030b121-7591-4c40-b891-52a6009228c9" path="/var/lib/kubelet/pods/3030b121-7591-4c40-b891-52a6009228c9/volumes" Oct 30 05:38:59.582390 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:59.582372 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f360336e-96e3-41c3-b483-aecf620a959f" path="/var/lib/kubelet/pods/f360336e-96e3-41c3-b483-aecf620a959f/volumes" Oct 30 05:38:59.606720 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:59.606700 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" Oct 30 05:38:59.606849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:59.606700 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7" event={"ID":"d677d710-645b-40d3-9222-24ca3d7047e5","Type":"ContainerDied","Data":"326212e74c2530e68ae44fd1c3b40acee38250c8feeb47e82d8581877aa87d81"} Oct 30 05:38:59.606907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:59.606853 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="326212e74c2530e68ae44fd1c3b40acee38250c8feeb47e82d8581877aa87d81" Oct 30 05:38:59.783902 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:59.783875 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" Oct 30 05:38:59.900150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:59.900070 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730e3261-6722-43fd-8509-19129cbe469f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730e3261-6722-43fd-8509-19129cbe469f-maas-api-openshift-ca\") pod \"730e3261-6722-43fd-8509-19129cbe469f\" (UID: \"730e3261-6722-43fd-8509-19129cbe469f\") " Oct 30 05:38:59.900150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:59.900130 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730e3261-6722-43fd-8509-19129cbe469f-kube-api-access-4dd6j\" (UniqueName: \"kubernetes.io/projected/730e3261-6722-43fd-8509-19129cbe469f-kube-api-access-4dd6j\") pod \"730e3261-6722-43fd-8509-19129cbe469f\" (UID: \"730e3261-6722-43fd-8509-19129cbe469f\") " Oct 30 05:38:59.900381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:59.900172 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/730e3261-6722-43fd-8509-19129cbe469f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730e3261-6722-43fd-8509-19129cbe469f-maas-api-xks-ca\") pod \"730e3261-6722-43fd-8509-19129cbe469f\" (UID: \"730e3261-6722-43fd-8509-19129cbe469f\") " Oct 30 05:38:59.900558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:59.900528 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730e3261-6722-43fd-8509-19129cbe469f-maas-api-openshift-ca" pod "730e3261-6722-43fd-8509-19129cbe469f" (UID: "730e3261-6722-43fd-8509-19129cbe469f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:59.900856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:59.900835 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730e3261-6722-43fd-8509-19129cbe469f-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:38:59.902421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:59.902396 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730e3261-6722-43fd-8509-19129cbe469f-kube-api-access-4dd6j" pod "730e3261-6722-43fd-8509-19129cbe469f" (UID: "730e3261-6722-43fd-8509-19129cbe469f"). InnerVolumeSpecName "kube-api-access-4dd6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:59.902421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:38:59.902410 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730e3261-6722-43fd-8509-19129cbe469f-maas-api-xks-ca" pod "730e3261-6722-43fd-8509-19129cbe469f" (UID: "730e3261-6722-43fd-8509-19129cbe469f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:39:00.001557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:00.001523 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dd6j\" (UniqueName: \"kubernetes.io/projected/730e3261-6722-43fd-8509-19129cbe469f-kube-api-access-4dd6j\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:00.001557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:00.001555 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730e3261-6722-43fd-8509-19129cbe469f-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:00.047536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:00.047502 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6"] Oct 30 05:39:00.056797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:00.056764 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ba2b02-2qqs6"] Oct 30 05:39:00.616892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:00.616849 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" event={"ID":"730e3261-6722-43fd-8509-19129cbe469f","Type":"ContainerDied","Data":"6217179fb34f0f0ff22553299bbba2ffa7ae305b03aebc2447dc01a270810753"} Oct 30 05:39:00.616892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:00.616880 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8" Oct 30 05:39:00.617484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:00.616905 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6217179fb34f0f0ff22553299bbba2ffa7ae305b03aebc2447dc01a270810753" Oct 30 05:39:01.377964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:01.377933 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q"] Oct 30 05:39:01.433600 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:01.433570 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz"] Oct 30 05:39:01.585490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:01.585446 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e33da2d-1ccb-42af-8326-3762acfb079e" path="/var/lib/kubelet/pods/3e33da2d-1ccb-42af-8326-3762acfb079e/volumes" Oct 30 05:39:03.667415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:03.667377 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" podUID="461cc202-2197-4c68-b5a7-734fb8e27072" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:03.667861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:03.667472 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" podUID="34ed4f94-1796-44b1-9de3-60065c919d4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:04.416301 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:04.416226 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d"] Oct 30 05:39:04.416616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:04.416517 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" podUID="e32ca7a5-ef18-422d-aa8b-cb90ddf88737" containerName="istio-proxy" containerID="cri-o://495044dd984b91a950fc3e87aa35449d381447b0c2a559085b54a7d4ed5f7ea2" gracePeriod=30 Oct 30 05:39:06.378261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:06.378214 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" podUID="d37da4e6-b588-4988-85e8-4658f44dc051" containerName="payload-processing" containerID="cri-o://735cb2773de42be84c7d56e42786f891ce7985bf95f0951a107fa6e33b8e4271" gracePeriod=25 Oct 30 05:39:06.434745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:06.434701 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" podUID="d782c841-b8c5-448c-a60e-2f097ed23724" containerName="payload-pre-processing" containerID="cri-o://1eca442a324bafdbfbaf40e45c4ed6e393618b85b8e78c2d10073cd314722707" gracePeriod=25 Oct 30 05:39:07.923230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:07.923189 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" podUID="d782c841-b8c5-448c-a60e-2f097ed23724" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:07.923676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:07.923265 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" podUID="d37da4e6-b588-4988-85e8-4658f44dc051" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:09.691822 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.691790 6587 generic.go:356] "Generic (PLEG): container finished" podID="e32ca7a5-ef18-422d-aa8b-cb90ddf88737" containerID="495044dd984b91a950fc3e87aa35449d381447b0c2a559085b54a7d4ed5f7ea2" exitCode=0 Oct 30 05:39:09.692184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.691865 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" event={"ID":"e32ca7a5-ef18-422d-aa8b-cb90ddf88737","Type":"ContainerDied","Data":"495044dd984b91a950fc3e87aa35449d381447b0c2a559085b54a7d4ed5f7ea2"} Oct 30 05:39:09.692184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.691919 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" event={"ID":"e32ca7a5-ef18-422d-aa8b-cb90ddf88737","Type":"ContainerDied","Data":"f575598cdeb72f8f9ede5b9a30efc37f9ccc05baf83c4eb6fb9e8bfb45215e7d"} Oct 30 05:39:09.692184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.691936 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f575598cdeb72f8f9ede5b9a30efc37f9ccc05baf83c4eb6fb9e8bfb45215e7d" Oct 30 05:39:09.699305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.699252 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:39:09.845413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.845382 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-kube-api-access-s6wzz\" (UniqueName: \"kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-kube-api-access-s6wzz\") pod \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " Oct 30 05:39:09.845598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.845522 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-socket\") pod \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " Oct 30 05:39:09.845598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.845556 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-credential-socket\") pod \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " Oct 30 05:39:09.845598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.845584 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-data\") pod \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " Oct 30 05:39:09.846400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.845616 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-podinfo\") pod \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " Oct 30 05:39:09.846400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.845701 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istiod-ca-cert\") pod \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " Oct 30 05:39:09.846400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.845773 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-token\" (UniqueName: \"kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-token\") pod \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " Oct 30 05:39:09.846400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.845815 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-certs\") pod \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " Oct 30 05:39:09.846400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.845855 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-envoy\") pod \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\" (UID: \"e32ca7a5-ef18-422d-aa8b-cb90ddf88737\") " Oct 30 05:39:09.846400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.845951 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-data" pod "e32ca7a5-ef18-422d-aa8b-cb90ddf88737" (UID: "e32ca7a5-ef18-422d-aa8b-cb90ddf88737"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:09.846400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.845964 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-socket" pod "e32ca7a5-ef18-422d-aa8b-cb90ddf88737" (UID: "e32ca7a5-ef18-422d-aa8b-cb90ddf88737"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:09.846400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.845979 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-credential-socket" pod "e32ca7a5-ef18-422d-aa8b-cb90ddf88737" (UID: "e32ca7a5-ef18-422d-aa8b-cb90ddf88737"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:09.846400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.846227 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-certs" pod "e32ca7a5-ef18-422d-aa8b-cb90ddf88737" (UID: "e32ca7a5-ef18-422d-aa8b-cb90ddf88737"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:09.846400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.846304 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istiod-ca-cert" pod "e32ca7a5-ef18-422d-aa8b-cb90ddf88737" (UID: "e32ca7a5-ef18-422d-aa8b-cb90ddf88737"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:09.847029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.846487 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:09.847029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.846511 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:09.847029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.846528 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:09.847029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.846543 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:09.847029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.846556 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:09.848082 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.848056 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-kube-api-access-s6wzz" pod "e32ca7a5-ef18-422d-aa8b-cb90ddf88737" (UID: "e32ca7a5-ef18-422d-aa8b-cb90ddf88737"). InnerVolumeSpecName "kube-api-access-s6wzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:09.848319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.848294 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-token" pod "e32ca7a5-ef18-422d-aa8b-cb90ddf88737" (UID: "e32ca7a5-ef18-422d-aa8b-cb90ddf88737"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:09.848526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.848507 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-envoy" pod "e32ca7a5-ef18-422d-aa8b-cb90ddf88737" (UID: "e32ca7a5-ef18-422d-aa8b-cb90ddf88737"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:09.848760 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.848736 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-podinfo" pod "e32ca7a5-ef18-422d-aa8b-cb90ddf88737" (UID: "e32ca7a5-ef18-422d-aa8b-cb90ddf88737"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:39:09.947423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.947347 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:09.947423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.947418 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:09.947627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.947434 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:09.947627 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:09.947443 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6wzz\" (UniqueName: \"kubernetes.io/projected/e32ca7a5-ef18-422d-aa8b-cb90ddf88737-kube-api-access-s6wzz\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:10.698936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:10.698898 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d" Oct 30 05:39:10.775502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:10.775466 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d"] Oct 30 05:39:10.788593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:10.788564 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d"] Oct 30 05:39:10.946067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:10.946034 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s"] Oct 30 05:39:10.949212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:10.949149 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a62fee-r9x9s"] Oct 30 05:39:11.470592 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:11.470557 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78"] Oct 30 05:39:11.474707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:11.474681 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-397179-6xs78"] Oct 30 05:39:11.583025 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:11.582994 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096248dd-a845-4351-8b36-03d6178fac68" path="/var/lib/kubelet/pods/096248dd-a845-4351-8b36-03d6178fac68/volumes" Oct 30 05:39:11.583471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:11.583456 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29fbeaee-26a9-400f-a520-2c55782d2fc4" path="/var/lib/kubelet/pods/29fbeaee-26a9-400f-a520-2c55782d2fc4/volumes" Oct 30 05:39:11.583898 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:11.583885 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32ca7a5-ef18-422d-aa8b-cb90ddf88737" path="/var/lib/kubelet/pods/e32ca7a5-ef18-422d-aa8b-cb90ddf88737/volumes" Oct 30 05:39:11.889524 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:11.889483 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx"] Oct 30 05:39:11.892263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:11.892231 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34722e-hsmdx"] Oct 30 05:39:13.031821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:13.031778 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b"] Oct 30 05:39:13.043309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:13.043278 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b1538d-fd76b"] Oct 30 05:39:13.582829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:13.582800 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c6bfca0-77af-43f1-97be-bef4862c4c23" path="/var/lib/kubelet/pods/8c6bfca0-77af-43f1-97be-bef4862c4c23/volumes" Oct 30 05:39:13.583258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:13.583244 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42c0cb6-4501-45ad-b0a2-5af41a49f204" path="/var/lib/kubelet/pods/b42c0cb6-4501-45ad-b0a2-5af41a49f204/volumes" Oct 30 05:39:13.666936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:13.666904 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" podUID="461cc202-2197-4c68-b5a7-734fb8e27072" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:13.667085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:13.666904 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" podUID="34ed4f94-1796-44b1-9de3-60065c919d4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:14.044074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.043911 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj"] Oct 30 05:39:14.058669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.058626 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0486f7c4-dmhbj"] Oct 30 05:39:14.151396 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.151369 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" Oct 30 05:39:14.171107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.171080 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz"] Oct 30 05:39:14.171274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.171255 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" podUID="10ddbb1e-572b-46df-a64e-2b8e966fe680" containerName="istio-proxy" containerID="cri-o://0f73de0764fda050572c150e3ca627d6763120b900bb48e77832f43be38ec0f4" gracePeriod=30 Oct 30 05:39:14.189329 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.189308 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" Oct 30 05:39:14.232299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.232260 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ed4f94-1796-44b1-9de3-60065c919d4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ed4f94-1796-44b1-9de3-60065c919d4a-plugins-config-volume\") pod \"34ed4f94-1796-44b1-9de3-60065c919d4a\" (UID: \"34ed4f94-1796-44b1-9de3-60065c919d4a\") " Oct 30 05:39:14.232487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.232318 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ed4f94-1796-44b1-9de3-60065c919d4a-kube-api-access-c6dm2\" (UniqueName: \"kubernetes.io/projected/34ed4f94-1796-44b1-9de3-60065c919d4a-kube-api-access-c6dm2\") pod \"34ed4f94-1796-44b1-9de3-60065c919d4a\" (UID: \"34ed4f94-1796-44b1-9de3-60065c919d4a\") " Oct 30 05:39:14.232487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.232388 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461cc202-2197-4c68-b5a7-734fb8e27072-kube-api-access-j4lxq\" (UniqueName: \"kubernetes.io/projected/461cc202-2197-4c68-b5a7-734fb8e27072-kube-api-access-j4lxq\") pod \"461cc202-2197-4c68-b5a7-734fb8e27072\" (UID: \"461cc202-2197-4c68-b5a7-734fb8e27072\") " Oct 30 05:39:14.232583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.232547 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/461cc202-2197-4c68-b5a7-734fb8e27072-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461cc202-2197-4c68-b5a7-734fb8e27072-plugins-config-volume\") pod \"461cc202-2197-4c68-b5a7-734fb8e27072\" (UID: \"461cc202-2197-4c68-b5a7-734fb8e27072\") " Oct 30 05:39:14.233464 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.233434 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461cc202-2197-4c68-b5a7-734fb8e27072-plugins-config-volume" pod "461cc202-2197-4c68-b5a7-734fb8e27072" (UID: "461cc202-2197-4c68-b5a7-734fb8e27072"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:14.234174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.234146 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ed4f94-1796-44b1-9de3-60065c919d4a-plugins-config-volume" pod "34ed4f94-1796-44b1-9de3-60065c919d4a" (UID: "34ed4f94-1796-44b1-9de3-60065c919d4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:14.236229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.236201 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ed4f94-1796-44b1-9de3-60065c919d4a-kube-api-access-c6dm2" pod "34ed4f94-1796-44b1-9de3-60065c919d4a" (UID: "34ed4f94-1796-44b1-9de3-60065c919d4a"). InnerVolumeSpecName "kube-api-access-c6dm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:14.236584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.236551 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461cc202-2197-4c68-b5a7-734fb8e27072-kube-api-access-j4lxq" pod "461cc202-2197-4c68-b5a7-734fb8e27072" (UID: "461cc202-2197-4c68-b5a7-734fb8e27072"). InnerVolumeSpecName "kube-api-access-j4lxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:14.334107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.334074 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461cc202-2197-4c68-b5a7-734fb8e27072-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:14.334107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.334105 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ed4f94-1796-44b1-9de3-60065c919d4a-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:14.334107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.334115 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6dm2\" (UniqueName: \"kubernetes.io/projected/34ed4f94-1796-44b1-9de3-60065c919d4a-kube-api-access-c6dm2\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:14.334366 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.334124 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4lxq\" (UniqueName: \"kubernetes.io/projected/461cc202-2197-4c68-b5a7-734fb8e27072-kube-api-access-j4lxq\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:14.736799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.736762 6587 generic.go:356] "Generic (PLEG): container finished" podID="34ed4f94-1796-44b1-9de3-60065c919d4a" containerID="50603cc7d711a8eea11b0e1bad9f9eb929fe6e45d78bce9804f6cd44dfd912ac" exitCode=0 Oct 30 05:39:14.737013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.736829 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" Oct 30 05:39:14.737013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.736851 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" event={"ID":"34ed4f94-1796-44b1-9de3-60065c919d4a","Type":"ContainerDied","Data":"50603cc7d711a8eea11b0e1bad9f9eb929fe6e45d78bce9804f6cd44dfd912ac"} Oct 30 05:39:14.737013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.736919 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4" event={"ID":"34ed4f94-1796-44b1-9de3-60065c919d4a","Type":"ContainerDied","Data":"e1b36643785a3faea00c6ebe38976f0f661a6c161b69ddae56b5951b9a997b59"} Oct 30 05:39:14.737013 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.736948 6587 scope.go:122] "RemoveContainer" containerID="50603cc7d711a8eea11b0e1bad9f9eb929fe6e45d78bce9804f6cd44dfd912ac" Oct 30 05:39:14.740988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.740238 6587 generic.go:356] "Generic (PLEG): container finished" podID="461cc202-2197-4c68-b5a7-734fb8e27072" containerID="1f9a94e78ff8d0e786adf74d5d0cae559b687bfd6b3acc8c444a7688b9b8dcbb" exitCode=0 Oct 30 05:39:14.740988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.740309 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" Oct 30 05:39:14.740988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.740388 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" event={"ID":"461cc202-2197-4c68-b5a7-734fb8e27072","Type":"ContainerDied","Data":"1f9a94e78ff8d0e786adf74d5d0cae559b687bfd6b3acc8c444a7688b9b8dcbb"} Oct 30 05:39:14.740988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.740477 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72" event={"ID":"461cc202-2197-4c68-b5a7-734fb8e27072","Type":"ContainerDied","Data":"93195b9f484da9f66b32d5daa12119089cc479c8d45021607edbea6d1a14b9fc"} Oct 30 05:39:14.752607 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.752589 6587 scope.go:122] "RemoveContainer" containerID="50603cc7d711a8eea11b0e1bad9f9eb929fe6e45d78bce9804f6cd44dfd912ac" Oct 30 05:39:14.752891 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:14.752868 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50603cc7d711a8eea11b0e1bad9f9eb929fe6e45d78bce9804f6cd44dfd912ac\": container with ID starting with 50603cc7d711a8eea11b0e1bad9f9eb929fe6e45d78bce9804f6cd44dfd912ac not found: ID does not exist" containerID="50603cc7d711a8eea11b0e1bad9f9eb929fe6e45d78bce9804f6cd44dfd912ac" Oct 30 05:39:14.752973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.752901 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50603cc7d711a8eea11b0e1bad9f9eb929fe6e45d78bce9804f6cd44dfd912ac"} err="failed to get container status \"50603cc7d711a8eea11b0e1bad9f9eb929fe6e45d78bce9804f6cd44dfd912ac\": rpc error: code = NotFound desc = could not find container \"50603cc7d711a8eea11b0e1bad9f9eb929fe6e45d78bce9804f6cd44dfd912ac\": container with ID starting with 50603cc7d711a8eea11b0e1bad9f9eb929fe6e45d78bce9804f6cd44dfd912ac not found: ID does not exist" Oct 30 05:39:14.752973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.752921 6587 scope.go:122] "RemoveContainer" containerID="1f9a94e78ff8d0e786adf74d5d0cae559b687bfd6b3acc8c444a7688b9b8dcbb" Oct 30 05:39:14.762640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.762625 6587 scope.go:122] "RemoveContainer" containerID="1f9a94e78ff8d0e786adf74d5d0cae559b687bfd6b3acc8c444a7688b9b8dcbb" Oct 30 05:39:14.762898 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:14.762879 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9a94e78ff8d0e786adf74d5d0cae559b687bfd6b3acc8c444a7688b9b8dcbb\": container with ID starting with 1f9a94e78ff8d0e786adf74d5d0cae559b687bfd6b3acc8c444a7688b9b8dcbb not found: ID does not exist" containerID="1f9a94e78ff8d0e786adf74d5d0cae559b687bfd6b3acc8c444a7688b9b8dcbb" Oct 30 05:39:14.762965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.762904 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9a94e78ff8d0e786adf74d5d0cae559b687bfd6b3acc8c444a7688b9b8dcbb"} err="failed to get container status \"1f9a94e78ff8d0e786adf74d5d0cae559b687bfd6b3acc8c444a7688b9b8dcbb\": rpc error: code = NotFound desc = could not find container \"1f9a94e78ff8d0e786adf74d5d0cae559b687bfd6b3acc8c444a7688b9b8dcbb\": container with ID starting with 1f9a94e78ff8d0e786adf74d5d0cae559b687bfd6b3acc8c444a7688b9b8dcbb not found: ID does not exist" Oct 30 05:39:14.790542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.790515 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4"] Oct 30 05:39:14.798531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.798508 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4"] Oct 30 05:39:14.848274 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.848243 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72"] Oct 30 05:39:14.853875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:14.853848 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72"] Oct 30 05:39:15.582217 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:15.582183 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ed4f94-1796-44b1-9de3-60065c919d4a" path="/var/lib/kubelet/pods/34ed4f94-1796-44b1-9de3-60065c919d4a/volumes" Oct 30 05:39:15.582688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:15.582633 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="461cc202-2197-4c68-b5a7-734fb8e27072" path="/var/lib/kubelet/pods/461cc202-2197-4c68-b5a7-734fb8e27072/volumes" Oct 30 05:39:15.583037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:15.583023 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a84f548e-8a0f-4c30-90ba-b486f657e0b4" path="/var/lib/kubelet/pods/a84f548e-8a0f-4c30-90ba-b486f657e0b4/volumes" Oct 30 05:39:17.923797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:17.923748 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" podUID="d782c841-b8c5-448c-a60e-2f097ed23724" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:17.923797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:17.923796 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" podUID="d37da4e6-b588-4988-85e8-4658f44dc051" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:19.227220 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227187 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv"] Oct 30 05:39:19.227780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227763 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" Oct 30 05:39:19.227780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227779 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227803 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e32ca7a5-ef18-422d-aa8b-cb90ddf88737" containerName="istio-proxy" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227809 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32ca7a5-ef18-422d-aa8b-cb90ddf88737" containerName="istio-proxy" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227821 6587 cpu_manager.go:414] "containerMap: removing container" podUID="34ed4f94-1796-44b1-9de3-60065c919d4a" containerName="payload-processing" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227826 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ed4f94-1796-44b1-9de3-60065c919d4a" containerName="payload-processing" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227839 6587 cpu_manager.go:414] "containerMap: removing container" podUID="461cc202-2197-4c68-b5a7-734fb8e27072" containerName="payload-pre-processing" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227844 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461cc202-2197-4c68-b5a7-734fb8e27072" containerName="payload-pre-processing" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227853 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227859 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227872 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227876 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227884 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d677d710-645b-40d3-9222-24ca3d7047e5" containerName="revoke-keys" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227888 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d677d710-645b-40d3-9222-24ca3d7047e5" containerName="revoke-keys" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227897 6587 cpu_manager.go:414] "containerMap: removing container" podUID="730e3261-6722-43fd-8509-19129cbe469f" containerName="revoke-keys" Oct 30 05:39:19.227912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227902 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730e3261-6722-43fd-8509-19129cbe469f" containerName="revoke-keys" Oct 30 05:39:19.228744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227977 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3030b121-7591-4c40-b891-52a6009228c9" containerName="main" Oct 30 05:39:19.228744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227988 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32ca7a5-ef18-422d-aa8b-cb90ddf88737" containerName="istio-proxy" Oct 30 05:39:19.228744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.227994 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ed4f94-1796-44b1-9de3-60065c919d4a" containerName="payload-processing" Oct 30 05:39:19.228744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.228001 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="461cc202-2197-4c68-b5a7-734fb8e27072" containerName="payload-pre-processing" Oct 30 05:39:19.228744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.228008 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" Oct 30 05:39:19.228744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.228014 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f360336e-96e3-41c3-b483-aecf620a959f" containerName="istio-proxy" Oct 30 05:39:19.228744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.228021 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d677d710-645b-40d3-9222-24ca3d7047e5" containerName="revoke-keys" Oct 30 05:39:19.228744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.228026 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="730e3261-6722-43fd-8509-19129cbe469f" containerName="revoke-keys" Oct 30 05:39:19.231447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.231425 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" Oct 30 05:39:19.251090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.251054 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:39:19.263234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.263205 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:39:19.263426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.263301 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:39:19.263533 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.263437 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:39:19.276438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.276411 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv"] Oct 30 05:39:19.307336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.307308 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv\" (UID: \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" Oct 30 05:39:19.307488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.307446 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv\" (UID: \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" Oct 30 05:39:19.307574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.307534 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbfb9\" (UniqueName: \"kubernetes.io/projected/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-kube-api-access-vbfb9\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv\" (UID: \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" Oct 30 05:39:19.409050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.409007 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbfb9\" (UniqueName: \"kubernetes.io/projected/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-kube-api-access-vbfb9\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv\" (UID: \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" Oct 30 05:39:19.409246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.409192 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv\" (UID: \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" Oct 30 05:39:19.409341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.409273 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv\" (UID: \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" Oct 30 05:39:19.409911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.409888 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv\" (UID: \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" Oct 30 05:39:19.411658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.411634 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv\" (UID: \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" Oct 30 05:39:19.436505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.436482 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbfb9\" (UniqueName: \"kubernetes.io/projected/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-kube-api-access-vbfb9\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv\" (UID: \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" Oct 30 05:39:19.469740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.469683 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:39:19.591891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.591860 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" Oct 30 05:39:19.611496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.611459 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-certs\") pod \"10ddbb1e-572b-46df-a64e-2b8e966fe680\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " Oct 30 05:39:19.611644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.611601 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-credential-socket\") pod \"10ddbb1e-572b-46df-a64e-2b8e966fe680\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " Oct 30 05:39:19.611644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.611636 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-envoy\") pod \"10ddbb1e-572b-46df-a64e-2b8e966fe680\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " Oct 30 05:39:19.611774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.611684 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-podinfo\") pod \"10ddbb1e-572b-46df-a64e-2b8e966fe680\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " Oct 30 05:39:19.611774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.611720 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-token\" (UniqueName: \"kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-token\") pod \"10ddbb1e-572b-46df-a64e-2b8e966fe680\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " Oct 30 05:39:19.611891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.611789 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10ddbb1e-572b-46df-a64e-2b8e966fe680-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ddbb1e-572b-46df-a64e-2b8e966fe680-istiod-ca-cert\") pod \"10ddbb1e-572b-46df-a64e-2b8e966fe680\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " Oct 30 05:39:19.611891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.611832 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-kube-api-access-7bttt\" (UniqueName: \"kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-kube-api-access-7bttt\") pod \"10ddbb1e-572b-46df-a64e-2b8e966fe680\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " Oct 30 05:39:19.611891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.611849 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-certs" pod "10ddbb1e-572b-46df-a64e-2b8e966fe680" (UID: "10ddbb1e-572b-46df-a64e-2b8e966fe680"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:19.612071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.611927 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-socket\") pod \"10ddbb1e-572b-46df-a64e-2b8e966fe680\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " Oct 30 05:39:19.612071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.611984 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-data\") pod \"10ddbb1e-572b-46df-a64e-2b8e966fe680\" (UID: \"10ddbb1e-572b-46df-a64e-2b8e966fe680\") " Oct 30 05:39:19.612557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.612533 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:19.612687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.612570 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-credential-socket" pod "10ddbb1e-572b-46df-a64e-2b8e966fe680" (UID: "10ddbb1e-572b-46df-a64e-2b8e966fe680"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:19.612761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.612688 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ddbb1e-572b-46df-a64e-2b8e966fe680-istiod-ca-cert" pod "10ddbb1e-572b-46df-a64e-2b8e966fe680" (UID: "10ddbb1e-572b-46df-a64e-2b8e966fe680"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:19.612820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.612774 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-data" pod "10ddbb1e-572b-46df-a64e-2b8e966fe680" (UID: "10ddbb1e-572b-46df-a64e-2b8e966fe680"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:19.612820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.612787 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-socket" pod "10ddbb1e-572b-46df-a64e-2b8e966fe680" (UID: "10ddbb1e-572b-46df-a64e-2b8e966fe680"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:19.614811 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.614788 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-token" pod "10ddbb1e-572b-46df-a64e-2b8e966fe680" (UID: "10ddbb1e-572b-46df-a64e-2b8e966fe680"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:19.615146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.615120 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-kube-api-access-7bttt" pod "10ddbb1e-572b-46df-a64e-2b8e966fe680" (UID: "10ddbb1e-572b-46df-a64e-2b8e966fe680"). InnerVolumeSpecName "kube-api-access-7bttt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:19.615259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.615229 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-podinfo" pod "10ddbb1e-572b-46df-a64e-2b8e966fe680" (UID: "10ddbb1e-572b-46df-a64e-2b8e966fe680"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:39:19.615326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.615231 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-envoy" pod "10ddbb1e-572b-46df-a64e-2b8e966fe680" (UID: "10ddbb1e-572b-46df-a64e-2b8e966fe680"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:19.716475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.716301 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:19.716475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.716343 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:19.716475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.716376 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10ddbb1e-572b-46df-a64e-2b8e966fe680-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:19.716475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.716390 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bttt\" (UniqueName: \"kubernetes.io/projected/10ddbb1e-572b-46df-a64e-2b8e966fe680-kube-api-access-7bttt\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:19.716475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.716407 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:19.716475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.716419 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:19.716475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.716433 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:19.716475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.716446 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10ddbb1e-572b-46df-a64e-2b8e966fe680-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:19.780150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.780064 6587 generic.go:356] "Generic (PLEG): container finished" podID="10ddbb1e-572b-46df-a64e-2b8e966fe680" containerID="0f73de0764fda050572c150e3ca627d6763120b900bb48e77832f43be38ec0f4" exitCode=0 Oct 30 05:39:19.780150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.780126 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" Oct 30 05:39:19.780150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.780136 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" event={"ID":"10ddbb1e-572b-46df-a64e-2b8e966fe680","Type":"ContainerDied","Data":"0f73de0764fda050572c150e3ca627d6763120b900bb48e77832f43be38ec0f4"} Oct 30 05:39:19.780409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.780190 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz" event={"ID":"10ddbb1e-572b-46df-a64e-2b8e966fe680","Type":"ContainerDied","Data":"a2575ae4d7ee5c240801a9191826c59d90116c75889f2fa51d4ebca5980f47a9"} Oct 30 05:39:19.780409 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.780213 6587 scope.go:122] "RemoveContainer" containerID="0f73de0764fda050572c150e3ca627d6763120b900bb48e77832f43be38ec0f4" Oct 30 05:39:19.792932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.792911 6587 scope.go:122] "RemoveContainer" containerID="0f73de0764fda050572c150e3ca627d6763120b900bb48e77832f43be38ec0f4" Oct 30 05:39:19.793206 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:19.793186 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f73de0764fda050572c150e3ca627d6763120b900bb48e77832f43be38ec0f4\": container with ID starting with 0f73de0764fda050572c150e3ca627d6763120b900bb48e77832f43be38ec0f4 not found: ID does not exist" containerID="0f73de0764fda050572c150e3ca627d6763120b900bb48e77832f43be38ec0f4" Oct 30 05:39:19.793292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.793213 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f73de0764fda050572c150e3ca627d6763120b900bb48e77832f43be38ec0f4"} err="failed to get container status \"0f73de0764fda050572c150e3ca627d6763120b900bb48e77832f43be38ec0f4\": rpc error: code = NotFound desc = could not find container \"0f73de0764fda050572c150e3ca627d6763120b900bb48e77832f43be38ec0f4\": container with ID starting with 0f73de0764fda050572c150e3ca627d6763120b900bb48e77832f43be38ec0f4 not found: ID does not exist" Oct 30 05:39:19.832695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.832663 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz"] Oct 30 05:39:19.838247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.838182 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz"] Oct 30 05:39:19.902318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:19.902296 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv"] Oct 30 05:39:19.905760 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:39:19.905731 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a358e5_9d9c_4af0_b9e6_0e22ba411407.slice/crio-71f5e23a96177ed8f4278176a50494576855ba8e7e81feebc50e8004f3d05e60 WatchSource:0}: Error finding container 71f5e23a96177ed8f4278176a50494576855ba8e7e81feebc50e8004f3d05e60: Status 404 returned error can't find the container with id 71f5e23a96177ed8f4278176a50494576855ba8e7e81feebc50e8004f3d05e60 Oct 30 05:39:20.474146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:20.474057 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v"] Oct 30 05:39:20.474985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:20.474927 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" podUID="405736d9-0831-4381-9707-a3392b6da104" containerName="istio-proxy" containerID="cri-o://1a23fbc2702a8976a4937d6236e8990f7078ac09e4c41d81a554ceca33b8c8ac" gracePeriod=30 Oct 30 05:39:20.790170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:20.790082 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" event={"ID":"a5a358e5-9d9c-4af0-b9e6-0e22ba411407","Type":"ContainerStarted","Data":"6f8c54d932475db788d8c67f91ad904fd9f3502a8ac0c9de1ec6d1dedb43f2e3"} Oct 30 05:39:20.790170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:20.790143 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" event={"ID":"a5a358e5-9d9c-4af0-b9e6-0e22ba411407","Type":"ContainerStarted","Data":"71f5e23a96177ed8f4278176a50494576855ba8e7e81feebc50e8004f3d05e60"} Oct 30 05:39:20.839565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:20.839498 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" podStartSLOduration=1.8394831379999999 podStartE2EDuration="1.839483138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:39:20.839011402 +0000 UTC m=+3285.366778189" watchObservedRunningTime="2026-10-30 05:39:20.839483138 +0000 UTC m=+3285.367250092" Oct 30 05:39:20.994823 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:20.994782 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m"] Oct 30 05:39:20.995441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:20.995420 6587 cpu_manager.go:414] "containerMap: removing container" podUID="10ddbb1e-572b-46df-a64e-2b8e966fe680" containerName="istio-proxy" Oct 30 05:39:20.995559 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:20.995441 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10ddbb1e-572b-46df-a64e-2b8e966fe680" containerName="istio-proxy" Oct 30 05:39:20.995659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:20.995644 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="10ddbb1e-572b-46df-a64e-2b8e966fe680" containerName="istio-proxy" Oct 30 05:39:20.999138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:20.999115 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" Oct 30 05:39:21.020149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.020118 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m"] Oct 30 05:39:21.097669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.097641 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzj57\" (UniqueName: \"kubernetes.io/projected/402b7f6b-0299-436a-ad22-7e43432a074a-kube-api-access-wzj57\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m\" (UID: \"402b7f6b-0299-436a-ad22-7e43432a074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" Oct 30 05:39:21.097825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.097793 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m\" (UID: \"402b7f6b-0299-436a-ad22-7e43432a074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" Oct 30 05:39:21.097874 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.097849 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m\" (UID: \"402b7f6b-0299-436a-ad22-7e43432a074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" Oct 30 05:39:21.198767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.198735 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m\" (UID: \"402b7f6b-0299-436a-ad22-7e43432a074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" Oct 30 05:39:21.198973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.198796 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzj57\" (UniqueName: \"kubernetes.io/projected/402b7f6b-0299-436a-ad22-7e43432a074a-kube-api-access-wzj57\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m\" (UID: \"402b7f6b-0299-436a-ad22-7e43432a074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" Oct 30 05:39:21.198973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.198910 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m\" (UID: \"402b7f6b-0299-436a-ad22-7e43432a074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" Oct 30 05:39:21.199570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.199548 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m\" (UID: \"402b7f6b-0299-436a-ad22-7e43432a074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" Oct 30 05:39:21.201230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.201201 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m\" (UID: \"402b7f6b-0299-436a-ad22-7e43432a074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" Oct 30 05:39:21.219602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.219577 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzj57\" (UniqueName: \"kubernetes.io/projected/402b7f6b-0299-436a-ad22-7e43432a074a-kube-api-access-wzj57\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m\" (UID: \"402b7f6b-0299-436a-ad22-7e43432a074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" Oct 30 05:39:21.340029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.339993 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" Oct 30 05:39:21.581780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.581744 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10ddbb1e-572b-46df-a64e-2b8e966fe680" path="/var/lib/kubelet/pods/10ddbb1e-572b-46df-a64e-2b8e966fe680/volumes" Oct 30 05:39:21.647863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.647632 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m"] Oct 30 05:39:21.651292 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:39:21.651264 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402b7f6b_0299_436a_ad22_7e43432a074a.slice/crio-513125f2d7b7dfde6ceace095a59bad3e3599b5b291aee08e7a4404c034c057e WatchSource:0}: Error finding container 513125f2d7b7dfde6ceace095a59bad3e3599b5b291aee08e7a4404c034c057e: Status 404 returned error can't find the container with id 513125f2d7b7dfde6ceace095a59bad3e3599b5b291aee08e7a4404c034c057e Oct 30 05:39:21.801032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.800997 6587 generic.go:356] "Generic (PLEG): container finished" podID="a5a358e5-9d9c-4af0-b9e6-0e22ba411407" containerID="6f8c54d932475db788d8c67f91ad904fd9f3502a8ac0c9de1ec6d1dedb43f2e3" exitCode=0 Oct 30 05:39:21.801208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.801083 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" event={"ID":"a5a358e5-9d9c-4af0-b9e6-0e22ba411407","Type":"ContainerDied","Data":"6f8c54d932475db788d8c67f91ad904fd9f3502a8ac0c9de1ec6d1dedb43f2e3"} Oct 30 05:39:21.802666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:21.802644 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" event={"ID":"402b7f6b-0299-436a-ad22-7e43432a074a","Type":"ContainerStarted","Data":"513125f2d7b7dfde6ceace095a59bad3e3599b5b291aee08e7a4404c034c057e"} Oct 30 05:39:22.811348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:22.811310 6587 generic.go:356] "Generic (PLEG): container finished" podID="402b7f6b-0299-436a-ad22-7e43432a074a" containerID="d6db06b8c30720d08fc49ad8eaa1f3869b896bd349864ea5080a312c65863034" exitCode=0 Oct 30 05:39:22.811850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:22.811399 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" event={"ID":"402b7f6b-0299-436a-ad22-7e43432a074a","Type":"ContainerDied","Data":"d6db06b8c30720d08fc49ad8eaa1f3869b896bd349864ea5080a312c65863034"} Oct 30 05:39:22.985973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:22.985947 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" Oct 30 05:39:23.024445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:23.024418 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-kube-api-access-vbfb9\" (UniqueName: \"kubernetes.io/projected/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-kube-api-access-vbfb9\") pod \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\" (UID: \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\") " Oct 30 05:39:23.024626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:23.024507 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-openshift-ca\") pod \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\" (UID: \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\") " Oct 30 05:39:23.024626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:23.024608 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-xks-ca\") pod \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\" (UID: \"a5a358e5-9d9c-4af0-b9e6-0e22ba411407\") " Oct 30 05:39:23.025165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:23.025126 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-openshift-ca" pod "a5a358e5-9d9c-4af0-b9e6-0e22ba411407" (UID: "a5a358e5-9d9c-4af0-b9e6-0e22ba411407"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:23.026877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:23.026846 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-kube-api-access-vbfb9" pod "a5a358e5-9d9c-4af0-b9e6-0e22ba411407" (UID: "a5a358e5-9d9c-4af0-b9e6-0e22ba411407"). InnerVolumeSpecName "kube-api-access-vbfb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:23.027031 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:23.027006 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-xks-ca" pod "a5a358e5-9d9c-4af0-b9e6-0e22ba411407" (UID: "a5a358e5-9d9c-4af0-b9e6-0e22ba411407"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:39:23.126587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:23.126551 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:23.126587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:23.126587 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbfb9\" (UniqueName: \"kubernetes.io/projected/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-kube-api-access-vbfb9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:23.126793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:23.126605 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a358e5-9d9c-4af0-b9e6-0e22ba411407-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:23.821399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:23.821352 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" Oct 30 05:39:23.821829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:23.821396 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv" event={"ID":"a5a358e5-9d9c-4af0-b9e6-0e22ba411407","Type":"ContainerDied","Data":"71f5e23a96177ed8f4278176a50494576855ba8e7e81feebc50e8004f3d05e60"} Oct 30 05:39:23.821829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:23.821464 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f5e23a96177ed8f4278176a50494576855ba8e7e81feebc50e8004f3d05e60" Oct 30 05:39:23.988522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:23.988500 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" Oct 30 05:39:24.037716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:24.037683 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-xks-ca\") pod \"402b7f6b-0299-436a-ad22-7e43432a074a\" (UID: \"402b7f6b-0299-436a-ad22-7e43432a074a\") " Oct 30 05:39:24.038003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:24.037977 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402b7f6b-0299-436a-ad22-7e43432a074a-kube-api-access-wzj57\" (UniqueName: \"kubernetes.io/projected/402b7f6b-0299-436a-ad22-7e43432a074a-kube-api-access-wzj57\") pod \"402b7f6b-0299-436a-ad22-7e43432a074a\" (UID: \"402b7f6b-0299-436a-ad22-7e43432a074a\") " Oct 30 05:39:24.038138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:24.038052 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-openshift-ca\") pod \"402b7f6b-0299-436a-ad22-7e43432a074a\" (UID: \"402b7f6b-0299-436a-ad22-7e43432a074a\") " Oct 30 05:39:24.038847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:24.038817 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-openshift-ca" pod "402b7f6b-0299-436a-ad22-7e43432a074a" (UID: "402b7f6b-0299-436a-ad22-7e43432a074a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:24.041034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:24.041008 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402b7f6b-0299-436a-ad22-7e43432a074a-kube-api-access-wzj57" pod "402b7f6b-0299-436a-ad22-7e43432a074a" (UID: "402b7f6b-0299-436a-ad22-7e43432a074a"). InnerVolumeSpecName "kube-api-access-wzj57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:24.041631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:24.041604 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-xks-ca" pod "402b7f6b-0299-436a-ad22-7e43432a074a" (UID: "402b7f6b-0299-436a-ad22-7e43432a074a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:39:24.140335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:24.140250 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:24.140335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:24.140295 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzj57\" (UniqueName: \"kubernetes.io/projected/402b7f6b-0299-436a-ad22-7e43432a074a-kube-api-access-wzj57\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:24.140335 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:24.140312 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/402b7f6b-0299-436a-ad22-7e43432a074a-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:24.830605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:24.830575 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" Oct 30 05:39:24.830988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:24.830605 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m" event={"ID":"402b7f6b-0299-436a-ad22-7e43432a074a","Type":"ContainerDied","Data":"513125f2d7b7dfde6ceace095a59bad3e3599b5b291aee08e7a4404c034c057e"} Oct 30 05:39:24.830988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:24.830656 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="513125f2d7b7dfde6ceace095a59bad3e3599b5b291aee08e7a4404c034c057e" Oct 30 05:39:25.801669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.801643 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:39:25.841318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.841281 6587 generic.go:356] "Generic (PLEG): container finished" podID="405736d9-0831-4381-9707-a3392b6da104" containerID="1a23fbc2702a8976a4937d6236e8990f7078ac09e4c41d81a554ceca33b8c8ac" exitCode=0 Oct 30 05:39:25.841776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.841321 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" Oct 30 05:39:25.841776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.841377 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" event={"ID":"405736d9-0831-4381-9707-a3392b6da104","Type":"ContainerDied","Data":"1a23fbc2702a8976a4937d6236e8990f7078ac09e4c41d81a554ceca33b8c8ac"} Oct 30 05:39:25.841776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.841445 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v" event={"ID":"405736d9-0831-4381-9707-a3392b6da104","Type":"ContainerDied","Data":"f54de6f7081f05d3c36cd7d656400f2eeec26d7c102548592444a661b8332186"} Oct 30 05:39:25.841776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.841480 6587 scope.go:122] "RemoveContainer" containerID="1a23fbc2702a8976a4937d6236e8990f7078ac09e4c41d81a554ceca33b8c8ac" Oct 30 05:39:25.857988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.857960 6587 scope.go:122] "RemoveContainer" containerID="99c1ab4fedcc6bb6f05ed8d2ba2610dc30192efaf7147393f45bef1634a1488f" Oct 30 05:39:25.861880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.861855 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-credential-socket\") pod \"405736d9-0831-4381-9707-a3392b6da104\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " Oct 30 05:39:25.861998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.861921 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-istio-token\" (UniqueName: \"kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-istio-token\") pod \"405736d9-0831-4381-9707-a3392b6da104\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " Oct 30 05:39:25.861998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.861959 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-socket\") pod \"405736d9-0831-4381-9707-a3392b6da104\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " Oct 30 05:39:25.862100 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.861996 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-envoy\") pod \"405736d9-0831-4381-9707-a3392b6da104\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " Oct 30 05:39:25.862100 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.862029 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405736d9-0831-4381-9707-a3392b6da104-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405736d9-0831-4381-9707-a3392b6da104-istiod-ca-cert\") pod \"405736d9-0831-4381-9707-a3392b6da104\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " Oct 30 05:39:25.862100 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.862076 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-certs\") pod \"405736d9-0831-4381-9707-a3392b6da104\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " Oct 30 05:39:25.862209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.862123 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-credential-socket" pod "405736d9-0831-4381-9707-a3392b6da104" (UID: "405736d9-0831-4381-9707-a3392b6da104"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.862209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.862186 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-data\") pod \"405736d9-0831-4381-9707-a3392b6da104\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " Oct 30 05:39:25.862294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.862228 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-kube-api-access-pvdm9\" (UniqueName: \"kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-kube-api-access-pvdm9\") pod \"405736d9-0831-4381-9707-a3392b6da104\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " Oct 30 05:39:25.862333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.862305 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/405736d9-0831-4381-9707-a3392b6da104-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405736d9-0831-4381-9707-a3392b6da104-istio-podinfo\") pod \"405736d9-0831-4381-9707-a3392b6da104\" (UID: \"405736d9-0831-4381-9707-a3392b6da104\") " Oct 30 05:39:25.862333 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.862321 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-socket" pod "405736d9-0831-4381-9707-a3392b6da104" (UID: "405736d9-0831-4381-9707-a3392b6da104"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.862695 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.862674 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-certs" pod "405736d9-0831-4381-9707-a3392b6da104" (UID: "405736d9-0831-4381-9707-a3392b6da104"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.862972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.862952 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405736d9-0831-4381-9707-a3392b6da104-istiod-ca-cert" pod "405736d9-0831-4381-9707-a3392b6da104" (UID: "405736d9-0831-4381-9707-a3392b6da104"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:25.864085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.863166 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-data" pod "405736d9-0831-4381-9707-a3392b6da104" (UID: "405736d9-0831-4381-9707-a3392b6da104"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.864085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.863337 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.864085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.863451 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.864085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.863470 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.864085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.863485 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405736d9-0831-4381-9707-a3392b6da104-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.864085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.863499 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.865634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.865610 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-envoy" pod "405736d9-0831-4381-9707-a3392b6da104" (UID: "405736d9-0831-4381-9707-a3392b6da104"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:25.865722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.865671 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-istio-token" pod "405736d9-0831-4381-9707-a3392b6da104" (UID: "405736d9-0831-4381-9707-a3392b6da104"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:25.865764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.865751 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/405736d9-0831-4381-9707-a3392b6da104-istio-podinfo" pod "405736d9-0831-4381-9707-a3392b6da104" (UID: "405736d9-0831-4381-9707-a3392b6da104"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:39:25.867462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.867440 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-kube-api-access-pvdm9" pod "405736d9-0831-4381-9707-a3392b6da104" (UID: "405736d9-0831-4381-9707-a3392b6da104"). InnerVolumeSpecName "kube-api-access-pvdm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:25.895077 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.895048 6587 scope.go:122] "RemoveContainer" containerID="1a23fbc2702a8976a4937d6236e8990f7078ac09e4c41d81a554ceca33b8c8ac" Oct 30 05:39:25.895513 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:25.895482 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a23fbc2702a8976a4937d6236e8990f7078ac09e4c41d81a554ceca33b8c8ac\": container with ID starting with 1a23fbc2702a8976a4937d6236e8990f7078ac09e4c41d81a554ceca33b8c8ac not found: ID does not exist" containerID="1a23fbc2702a8976a4937d6236e8990f7078ac09e4c41d81a554ceca33b8c8ac" Oct 30 05:39:25.895605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.895527 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a23fbc2702a8976a4937d6236e8990f7078ac09e4c41d81a554ceca33b8c8ac"} err="failed to get container status \"1a23fbc2702a8976a4937d6236e8990f7078ac09e4c41d81a554ceca33b8c8ac\": rpc error: code = NotFound desc = could not find container \"1a23fbc2702a8976a4937d6236e8990f7078ac09e4c41d81a554ceca33b8c8ac\": container with ID starting with 1a23fbc2702a8976a4937d6236e8990f7078ac09e4c41d81a554ceca33b8c8ac not found: ID does not exist" Oct 30 05:39:25.895605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.895555 6587 scope.go:122] "RemoveContainer" containerID="99c1ab4fedcc6bb6f05ed8d2ba2610dc30192efaf7147393f45bef1634a1488f" Oct 30 05:39:25.895980 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:25.895948 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c1ab4fedcc6bb6f05ed8d2ba2610dc30192efaf7147393f45bef1634a1488f\": container with ID starting with 99c1ab4fedcc6bb6f05ed8d2ba2610dc30192efaf7147393f45bef1634a1488f not found: ID does not exist" containerID="99c1ab4fedcc6bb6f05ed8d2ba2610dc30192efaf7147393f45bef1634a1488f" Oct 30 05:39:25.896073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.895991 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c1ab4fedcc6bb6f05ed8d2ba2610dc30192efaf7147393f45bef1634a1488f"} err="failed to get container status \"99c1ab4fedcc6bb6f05ed8d2ba2610dc30192efaf7147393f45bef1634a1488f\": rpc error: code = NotFound desc = could not find container \"99c1ab4fedcc6bb6f05ed8d2ba2610dc30192efaf7147393f45bef1634a1488f\": container with ID starting with 99c1ab4fedcc6bb6f05ed8d2ba2610dc30192efaf7147393f45bef1634a1488f not found: ID does not exist" Oct 30 05:39:25.964727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.964698 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.964727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.964724 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405736d9-0831-4381-9707-a3392b6da104-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.964950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.964737 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvdm9\" (UniqueName: \"kubernetes.io/projected/405736d9-0831-4381-9707-a3392b6da104-kube-api-access-pvdm9\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.964950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:25.964750 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405736d9-0831-4381-9707-a3392b6da104-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:25.985642 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:25.985609 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:39:26.191567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.191536 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v"] Oct 30 05:39:26.195972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.195944 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v"] Oct 30 05:39:26.698614 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.698548 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" Oct 30 05:39:26.759126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.759093 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" Oct 30 05:39:26.776503 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.776472 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37da4e6-b588-4988-85e8-4658f44dc051-kube-api-access-mt9sv\" (UniqueName: \"kubernetes.io/projected/d37da4e6-b588-4988-85e8-4658f44dc051-kube-api-access-mt9sv\") pod \"d37da4e6-b588-4988-85e8-4658f44dc051\" (UID: \"d37da4e6-b588-4988-85e8-4658f44dc051\") " Oct 30 05:39:26.776683 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.776517 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37da4e6-b588-4988-85e8-4658f44dc051-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37da4e6-b588-4988-85e8-4658f44dc051-plugins-config-volume\") pod \"d37da4e6-b588-4988-85e8-4658f44dc051\" (UID: \"d37da4e6-b588-4988-85e8-4658f44dc051\") " Oct 30 05:39:26.776947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.776924 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37da4e6-b588-4988-85e8-4658f44dc051-plugins-config-volume" pod "d37da4e6-b588-4988-85e8-4658f44dc051" (UID: "d37da4e6-b588-4988-85e8-4658f44dc051"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:26.777073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.777057 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37da4e6-b588-4988-85e8-4658f44dc051-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:26.778781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.778759 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37da4e6-b588-4988-85e8-4658f44dc051-kube-api-access-mt9sv" pod "d37da4e6-b588-4988-85e8-4658f44dc051" (UID: "d37da4e6-b588-4988-85e8-4658f44dc051"). InnerVolumeSpecName "kube-api-access-mt9sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:26.850949 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.850914 6587 generic.go:356] "Generic (PLEG): container finished" podID="d37da4e6-b588-4988-85e8-4658f44dc051" containerID="735cb2773de42be84c7d56e42786f891ce7985bf95f0951a107fa6e33b8e4271" exitCode=0 Oct 30 05:39:26.851455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.850968 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" Oct 30 05:39:26.851455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.850993 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" event={"ID":"d37da4e6-b588-4988-85e8-4658f44dc051","Type":"ContainerDied","Data":"735cb2773de42be84c7d56e42786f891ce7985bf95f0951a107fa6e33b8e4271"} Oct 30 05:39:26.851455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.851050 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q" event={"ID":"d37da4e6-b588-4988-85e8-4658f44dc051","Type":"ContainerDied","Data":"d789bf825acdb2f355bcd985dd027f73810ff9fc12f5a25a2150729af4493e75"} Oct 30 05:39:26.851455 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.851078 6587 scope.go:122] "RemoveContainer" containerID="735cb2773de42be84c7d56e42786f891ce7985bf95f0951a107fa6e33b8e4271" Oct 30 05:39:26.852654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.852626 6587 generic.go:356] "Generic (PLEG): container finished" podID="d782c841-b8c5-448c-a60e-2f097ed23724" containerID="1eca442a324bafdbfbaf40e45c4ed6e393618b85b8e78c2d10073cd314722707" exitCode=0 Oct 30 05:39:26.852820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.852681 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" Oct 30 05:39:26.852820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.852725 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" event={"ID":"d782c841-b8c5-448c-a60e-2f097ed23724","Type":"ContainerDied","Data":"1eca442a324bafdbfbaf40e45c4ed6e393618b85b8e78c2d10073cd314722707"} Oct 30 05:39:26.852820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.852776 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz" event={"ID":"d782c841-b8c5-448c-a60e-2f097ed23724","Type":"ContainerDied","Data":"fe759dd95936bce3c05872f793e64be7ca6b5ca3340abf0be9bc1468737b2c7b"} Oct 30 05:39:26.863255 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.863235 6587 scope.go:122] "RemoveContainer" containerID="735cb2773de42be84c7d56e42786f891ce7985bf95f0951a107fa6e33b8e4271" Oct 30 05:39:26.863547 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:26.863520 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735cb2773de42be84c7d56e42786f891ce7985bf95f0951a107fa6e33b8e4271\": container with ID starting with 735cb2773de42be84c7d56e42786f891ce7985bf95f0951a107fa6e33b8e4271 not found: ID does not exist" containerID="735cb2773de42be84c7d56e42786f891ce7985bf95f0951a107fa6e33b8e4271" Oct 30 05:39:26.863632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.863557 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735cb2773de42be84c7d56e42786f891ce7985bf95f0951a107fa6e33b8e4271"} err="failed to get container status \"735cb2773de42be84c7d56e42786f891ce7985bf95f0951a107fa6e33b8e4271\": rpc error: code = NotFound desc = could not find container \"735cb2773de42be84c7d56e42786f891ce7985bf95f0951a107fa6e33b8e4271\": container with ID starting with 735cb2773de42be84c7d56e42786f891ce7985bf95f0951a107fa6e33b8e4271 not found: ID does not exist" Oct 30 05:39:26.863632 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.863575 6587 scope.go:122] "RemoveContainer" containerID="1eca442a324bafdbfbaf40e45c4ed6e393618b85b8e78c2d10073cd314722707" Oct 30 05:39:26.874149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.874131 6587 scope.go:122] "RemoveContainer" containerID="1eca442a324bafdbfbaf40e45c4ed6e393618b85b8e78c2d10073cd314722707" Oct 30 05:39:26.874447 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:26.874429 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eca442a324bafdbfbaf40e45c4ed6e393618b85b8e78c2d10073cd314722707\": container with ID starting with 1eca442a324bafdbfbaf40e45c4ed6e393618b85b8e78c2d10073cd314722707 not found: ID does not exist" containerID="1eca442a324bafdbfbaf40e45c4ed6e393618b85b8e78c2d10073cd314722707" Oct 30 05:39:26.874522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.874453 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eca442a324bafdbfbaf40e45c4ed6e393618b85b8e78c2d10073cd314722707"} err="failed to get container status \"1eca442a324bafdbfbaf40e45c4ed6e393618b85b8e78c2d10073cd314722707\": rpc error: code = NotFound desc = could not find container \"1eca442a324bafdbfbaf40e45c4ed6e393618b85b8e78c2d10073cd314722707\": container with ID starting with 1eca442a324bafdbfbaf40e45c4ed6e393618b85b8e78c2d10073cd314722707 not found: ID does not exist" Oct 30 05:39:26.877856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.877836 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d782c841-b8c5-448c-a60e-2f097ed23724-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d782c841-b8c5-448c-a60e-2f097ed23724-plugins-config-volume\") pod \"d782c841-b8c5-448c-a60e-2f097ed23724\" (UID: \"d782c841-b8c5-448c-a60e-2f097ed23724\") " Oct 30 05:39:26.877972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.877957 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d782c841-b8c5-448c-a60e-2f097ed23724-kube-api-access-hbgmc\" (UniqueName: \"kubernetes.io/projected/d782c841-b8c5-448c-a60e-2f097ed23724-kube-api-access-hbgmc\") pod \"d782c841-b8c5-448c-a60e-2f097ed23724\" (UID: \"d782c841-b8c5-448c-a60e-2f097ed23724\") " Oct 30 05:39:26.878182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.878162 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d782c841-b8c5-448c-a60e-2f097ed23724-plugins-config-volume" pod "d782c841-b8c5-448c-a60e-2f097ed23724" (UID: "d782c841-b8c5-448c-a60e-2f097ed23724"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:26.878450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.878434 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt9sv\" (UniqueName: \"kubernetes.io/projected/d37da4e6-b588-4988-85e8-4658f44dc051-kube-api-access-mt9sv\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:26.878527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.878453 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d782c841-b8c5-448c-a60e-2f097ed23724-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:26.879894 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.879875 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d782c841-b8c5-448c-a60e-2f097ed23724-kube-api-access-hbgmc" pod "d782c841-b8c5-448c-a60e-2f097ed23724" (UID: "d782c841-b8c5-448c-a60e-2f097ed23724"). InnerVolumeSpecName "kube-api-access-hbgmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:26.921038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.920998 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q"] Oct 30 05:39:26.927740 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.927711 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q"] Oct 30 05:39:26.979380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:26.979286 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbgmc\" (UniqueName: \"kubernetes.io/projected/d782c841-b8c5-448c-a60e-2f097ed23724-kube-api-access-hbgmc\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:27.203725 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:27.203685 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz"] Oct 30 05:39:27.208037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:27.208008 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz"] Oct 30 05:39:27.581938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:27.581893 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405736d9-0831-4381-9707-a3392b6da104" path="/var/lib/kubelet/pods/405736d9-0831-4381-9707-a3392b6da104/volumes" Oct 30 05:39:27.582473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:27.582450 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37da4e6-b588-4988-85e8-4658f44dc051" path="/var/lib/kubelet/pods/d37da4e6-b588-4988-85e8-4658f44dc051/volumes" Oct 30 05:39:27.582941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:27.582913 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d782c841-b8c5-448c-a60e-2f097ed23724" path="/var/lib/kubelet/pods/d782c841-b8c5-448c-a60e-2f097ed23724/volumes" Oct 30 05:39:29.751757 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:29.751728 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd"] Oct 30 05:39:29.783161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:29.783134 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz"] Oct 30 05:39:31.521404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:31.521367 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv"] Oct 30 05:39:34.752542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:34.752339 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" podUID="3d330d3a-1853-4bc5-92a6-d7b13bcf2466" containerName="payload-processing" containerID="cri-o://c5ce09fcf677c8c7754a797f2504397e08430bdd9aac783d4e6941e006dcface" gracePeriod=25 Oct 30 05:39:34.784441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:34.784397 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" podUID="3c5ff08b-f239-4e07-863a-3653c81a6708" containerName="payload-pre-processing" containerID="cri-o://3b5a472e2c0750c8b11ad491bf2ca7da9e9f3bd5c5214959f3f5f5e67dd4f931" gracePeriod=25 Oct 30 05:39:34.888232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:34.888195 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" podUID="3d330d3a-1853-4bc5-92a6-d7b13bcf2466" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:34.888454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:34.888208 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" podUID="3c5ff08b-f239-4e07-863a-3653c81a6708" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:35.105964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:35.105929 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m"] Oct 30 05:39:35.117131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:35.117100 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0dba31-r858m"] Oct 30 05:39:35.582185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:35.582153 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="402b7f6b-0299-436a-ad22-7e43432a074a" path="/var/lib/kubelet/pods/402b7f6b-0299-436a-ad22-7e43432a074a/volumes" Oct 30 05:39:36.337190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:36.337156 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:39:36.337733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:36.337300 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:39:36.522030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:36.521951 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" podUID="38da181c-3cfa-48da-b727-08064c3d57fe" containerName="payload-processing" containerID="cri-o://b6bd2e82274909902811212bc020634f12a2f9fe6574ad117156183873129f7e" gracePeriod=25 Oct 30 05:39:38.707658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:38.707616 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" podUID="38da181c-3cfa-48da-b727-08064c3d57fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:39.145916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:39.145880 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv"] Oct 30 05:39:39.159196 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:39.159166 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-35a94e-zgxpv"] Oct 30 05:39:39.580485 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:39.580448 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a358e5-9d9c-4af0-b9e6-0e22ba411407" path="/var/lib/kubelet/pods/a5a358e5-9d9c-4af0-b9e6-0e22ba411407/volumes" Oct 30 05:39:42.765499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:42.765461 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn"] Oct 30 05:39:42.765989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:42.765788 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" containerID="cri-o://f977a47e488d84b54b06f1ad51501c5fe4252a314e0f3f720660defa7389b114" gracePeriod=30 Oct 30 05:39:43.449062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449028 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9"] Oct 30 05:39:43.449626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449602 6587 cpu_manager.go:414] "containerMap: removing container" podUID="405736d9-0831-4381-9707-a3392b6da104" containerName="istio-proxy" Oct 30 05:39:43.449626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449621 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405736d9-0831-4381-9707-a3392b6da104" containerName="istio-proxy" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449633 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d782c841-b8c5-448c-a60e-2f097ed23724" containerName="payload-pre-processing" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449639 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d782c841-b8c5-448c-a60e-2f097ed23724" containerName="payload-pre-processing" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449655 6587 cpu_manager.go:414] "containerMap: removing container" podUID="405736d9-0831-4381-9707-a3392b6da104" containerName="istio-proxy" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449660 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405736d9-0831-4381-9707-a3392b6da104" containerName="istio-proxy" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449671 6587 cpu_manager.go:414] "containerMap: removing container" podUID="a5a358e5-9d9c-4af0-b9e6-0e22ba411407" containerName="revoke-keys" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449678 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a358e5-9d9c-4af0-b9e6-0e22ba411407" containerName="revoke-keys" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449695 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d37da4e6-b588-4988-85e8-4658f44dc051" containerName="payload-processing" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449700 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37da4e6-b588-4988-85e8-4658f44dc051" containerName="payload-processing" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449709 6587 cpu_manager.go:414] "containerMap: removing container" podUID="402b7f6b-0299-436a-ad22-7e43432a074a" containerName="revoke-keys" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449714 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402b7f6b-0299-436a-ad22-7e43432a074a" containerName="revoke-keys" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449805 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37da4e6-b588-4988-85e8-4658f44dc051" containerName="payload-processing" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449816 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="402b7f6b-0299-436a-ad22-7e43432a074a" containerName="revoke-keys" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449834 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="405736d9-0831-4381-9707-a3392b6da104" containerName="istio-proxy" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449842 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d782c841-b8c5-448c-a60e-2f097ed23724" containerName="payload-pre-processing" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449850 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="405736d9-0831-4381-9707-a3392b6da104" containerName="istio-proxy" Oct 30 05:39:43.449864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.449856 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a358e5-9d9c-4af0-b9e6-0e22ba411407" containerName="revoke-keys" Oct 30 05:39:43.453545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.453525 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" Oct 30 05:39:43.473109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.473080 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9"] Oct 30 05:39:43.480910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.480886 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:39:43.481032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.480970 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:39:43.501997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.501955 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:39:43.502732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.502697 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:39:43.504657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.504627 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvl6r\" (UniqueName: \"kubernetes.io/projected/6904fb31-6918-4dd1-bb40-d683a6bde3a3-kube-api-access-pvl6r\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9\" (UID: \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" Oct 30 05:39:43.504788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.504692 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9\" (UID: \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" Oct 30 05:39:43.504891 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.504869 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9\" (UID: \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" Oct 30 05:39:43.605818 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.605787 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvl6r\" (UniqueName: \"kubernetes.io/projected/6904fb31-6918-4dd1-bb40-d683a6bde3a3-kube-api-access-pvl6r\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9\" (UID: \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" Oct 30 05:39:43.606012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.605839 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9\" (UID: \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" Oct 30 05:39:43.606012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.605884 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9\" (UID: \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" Oct 30 05:39:43.606624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.606604 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9\" (UID: \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" Oct 30 05:39:43.608303 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.608282 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9\" (UID: \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" Oct 30 05:39:43.627795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.627769 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvl6r\" (UniqueName: \"kubernetes.io/projected/6904fb31-6918-4dd1-bb40-d683a6bde3a3-kube-api-access-pvl6r\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9\" (UID: \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" Oct 30 05:39:43.774932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:43.774856 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" Oct 30 05:39:44.118669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:44.118637 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9"] Oct 30 05:39:44.314153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:44.314122 6587 scope.go:122] "RemoveContainer" containerID="2ee235afc90c614026b92f2464e128ecd2e053d7d3dfe4e43e513eb99148fc76" Oct 30 05:39:44.341642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:44.341616 6587 scope.go:122] "RemoveContainer" containerID="b809d44cbe25300f3b6546d43edbfad478e9223a051ea0d65ad4f8134aacc439" Oct 30 05:39:44.351643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:44.351623 6587 scope.go:122] "RemoveContainer" containerID="0f6ff9dada6f55fa0bf3f60769f80c0e8d219111303984ca878de3188f8bebaf" Oct 30 05:39:44.363122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:44.363100 6587 scope.go:122] "RemoveContainer" containerID="6a6b29c328554169a39e679970bf56bedc577a1549c787b54f8991f6b712f03b" Oct 30 05:39:44.374417 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:44.374395 6587 scope.go:122] "RemoveContainer" containerID="495044dd984b91a950fc3e87aa35449d381447b0c2a559085b54a7d4ed5f7ea2" Oct 30 05:39:44.390379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:44.386635 6587 scope.go:122] "RemoveContainer" containerID="585389339da7e137c4eceac0928f52d5422622baae1aaec679eb2dc78b8e6b88" Oct 30 05:39:44.888270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:44.888224 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" podUID="3c5ff08b-f239-4e07-863a-3653c81a6708" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:44.888756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:44.888295 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" podUID="3d330d3a-1853-4bc5-92a6-d7b13bcf2466" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:45.021962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:45.021926 6587 generic.go:356] "Generic (PLEG): container finished" podID="6904fb31-6918-4dd1-bb40-d683a6bde3a3" containerID="8d5eb35b07dd0a554b9dea768ea418a2335c9a5ac69c76e89256ef41ee0d1f07" exitCode=0 Oct 30 05:39:45.022123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:45.022012 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" event={"ID":"6904fb31-6918-4dd1-bb40-d683a6bde3a3","Type":"ContainerDied","Data":"8d5eb35b07dd0a554b9dea768ea418a2335c9a5ac69c76e89256ef41ee0d1f07"} Oct 30 05:39:45.022123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:45.022068 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" event={"ID":"6904fb31-6918-4dd1-bb40-d683a6bde3a3","Type":"ContainerStarted","Data":"8bb35b9d02361261b501383263e3ee61223c6b7692a70ff2a97ed2241d15adb7"} Oct 30 05:39:46.184189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.184157 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" Oct 30 05:39:46.240501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.240468 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-openshift-ca\") pod \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\" (UID: \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\") " Oct 30 05:39:46.240687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.240667 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-xks-ca\") pod \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\" (UID: \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\") " Oct 30 05:39:46.240763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.240747 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6904fb31-6918-4dd1-bb40-d683a6bde3a3-kube-api-access-pvl6r\" (UniqueName: \"kubernetes.io/projected/6904fb31-6918-4dd1-bb40-d683a6bde3a3-kube-api-access-pvl6r\") pod \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\" (UID: \"6904fb31-6918-4dd1-bb40-d683a6bde3a3\") " Oct 30 05:39:46.240876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.240848 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-openshift-ca" pod "6904fb31-6918-4dd1-bb40-d683a6bde3a3" (UID: "6904fb31-6918-4dd1-bb40-d683a6bde3a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:46.241230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.241204 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:46.242819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.242789 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6904fb31-6918-4dd1-bb40-d683a6bde3a3-kube-api-access-pvl6r" pod "6904fb31-6918-4dd1-bb40-d683a6bde3a3" (UID: "6904fb31-6918-4dd1-bb40-d683a6bde3a3"). InnerVolumeSpecName "kube-api-access-pvl6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:46.242925 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.242839 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-xks-ca" pod "6904fb31-6918-4dd1-bb40-d683a6bde3a3" (UID: "6904fb31-6918-4dd1-bb40-d683a6bde3a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:39:46.341991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.341957 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6904fb31-6918-4dd1-bb40-d683a6bde3a3-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:46.341991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.341989 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvl6r\" (UniqueName: \"kubernetes.io/projected/6904fb31-6918-4dd1-bb40-d683a6bde3a3-kube-api-access-pvl6r\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:46.570819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.570240 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh"] Oct 30 05:39:46.570819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.570510 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" podUID="8994c5b4-d21b-4b3f-864a-68fa175ab737" containerName="istio-proxy" containerID="cri-o://24bddc0bd1383638807f8a334f2459e789ec8f708e9e706c2ab76aaf9eab0d7d" gracePeriod=30 Oct 30 05:39:46.597636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.597549 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw"] Oct 30 05:39:46.597909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:46.597883 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" containerID="cri-o://62ed9e0c8d5903e83e05f3186145a82721261c2c668309cdb18537a48a403536" gracePeriod=30 Oct 30 05:39:47.036146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.036058 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" Oct 30 05:39:47.036146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.036094 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9" event={"ID":"6904fb31-6918-4dd1-bb40-d683a6bde3a3","Type":"ContainerDied","Data":"8bb35b9d02361261b501383263e3ee61223c6b7692a70ff2a97ed2241d15adb7"} Oct 30 05:39:47.036146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.036139 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bb35b9d02361261b501383263e3ee61223c6b7692a70ff2a97ed2241d15adb7" Oct 30 05:39:47.117791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.117757 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m"] Oct 30 05:39:47.118312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.118298 6587 cpu_manager.go:414] "containerMap: removing container" podUID="6904fb31-6918-4dd1-bb40-d683a6bde3a3" containerName="revoke-keys" Oct 30 05:39:47.118312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.118312 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6904fb31-6918-4dd1-bb40-d683a6bde3a3" containerName="revoke-keys" Oct 30 05:39:47.118449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.118438 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="6904fb31-6918-4dd1-bb40-d683a6bde3a3" containerName="revoke-keys" Oct 30 05:39:47.122042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.122020 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" Oct 30 05:39:47.141598 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.141578 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:39:47.141691 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.141680 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:39:47.141862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.141846 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:39:47.141974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.141941 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:39:47.151281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.146112 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m"] Oct 30 05:39:47.251467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.251430 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m\" (UID: \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" Oct 30 05:39:47.251467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.251471 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m\" (UID: \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" Oct 30 05:39:47.251896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.251653 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4qq7\" (UniqueName: \"kubernetes.io/projected/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-kube-api-access-k4qq7\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m\" (UID: \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" Oct 30 05:39:47.352811 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.352782 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4qq7\" (UniqueName: \"kubernetes.io/projected/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-kube-api-access-k4qq7\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m\" (UID: \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" Oct 30 05:39:47.353024 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.352862 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m\" (UID: \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" Oct 30 05:39:47.353024 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.352883 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m\" (UID: \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" Oct 30 05:39:47.353579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.353556 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m\" (UID: \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" Oct 30 05:39:47.355313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.355290 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m\" (UID: \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" Oct 30 05:39:47.372591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.372562 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4qq7\" (UniqueName: \"kubernetes.io/projected/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-kube-api-access-k4qq7\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m\" (UID: \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" Oct 30 05:39:47.436640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.436606 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" Oct 30 05:39:47.742091 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:47.742064 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m"] Oct 30 05:39:47.743635 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:39:47.743607 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56577f79_3dfd_46ce_adf2_1e7ec2db9fce.slice/crio-8bb4bb80556a1eedf0dae5e573912ff870bd09e8159b08031a62e5d0335c7099 WatchSource:0}: Error finding container 8bb4bb80556a1eedf0dae5e573912ff870bd09e8159b08031a62e5d0335c7099: Status 404 returned error can't find the container with id 8bb4bb80556a1eedf0dae5e573912ff870bd09e8159b08031a62e5d0335c7099 Oct 30 05:39:48.039569 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.039542 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:39:48.044257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.044226 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" event={"ID":"56577f79-3dfd-46ce-adf2-1e7ec2db9fce","Type":"ContainerStarted","Data":"8bb4bb80556a1eedf0dae5e573912ff870bd09e8159b08031a62e5d0335c7099"} Oct 30 05:39:48.046111 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.046085 6587 generic.go:356] "Generic (PLEG): container finished" podID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerID="f977a47e488d84b54b06f1ad51501c5fe4252a314e0f3f720660defa7389b114" exitCode=0 Oct 30 05:39:48.046202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.046130 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" Oct 30 05:39:48.046202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.046163 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" event={"ID":"2e7930f4-1e78-4bd5-be93-010fe7e02d20","Type":"ContainerDied","Data":"f977a47e488d84b54b06f1ad51501c5fe4252a314e0f3f720660defa7389b114"} Oct 30 05:39:48.046302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.046214 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn" event={"ID":"2e7930f4-1e78-4bd5-be93-010fe7e02d20","Type":"ContainerDied","Data":"6d420269333e6fb2003d1260b1beef054b740576da554d62bb853dd87e3505c0"} Oct 30 05:39:48.046302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.046249 6587 scope.go:122] "RemoveContainer" containerID="f977a47e488d84b54b06f1ad51501c5fe4252a314e0f3f720660defa7389b114" Oct 30 05:39:48.060587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.060567 6587 scope.go:122] "RemoveContainer" containerID="250dc0b7b4d0e922ebc9cd1e5dac86cf9d68e5cc75841625e541237809aca257" Oct 30 05:39:48.071144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.071124 6587 scope.go:122] "RemoveContainer" containerID="f977a47e488d84b54b06f1ad51501c5fe4252a314e0f3f720660defa7389b114" Oct 30 05:39:48.071422 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:48.071401 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f977a47e488d84b54b06f1ad51501c5fe4252a314e0f3f720660defa7389b114\": container with ID starting with f977a47e488d84b54b06f1ad51501c5fe4252a314e0f3f720660defa7389b114 not found: ID does not exist" containerID="f977a47e488d84b54b06f1ad51501c5fe4252a314e0f3f720660defa7389b114" Oct 30 05:39:48.071487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.071429 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f977a47e488d84b54b06f1ad51501c5fe4252a314e0f3f720660defa7389b114"} err="failed to get container status \"f977a47e488d84b54b06f1ad51501c5fe4252a314e0f3f720660defa7389b114\": rpc error: code = NotFound desc = could not find container \"f977a47e488d84b54b06f1ad51501c5fe4252a314e0f3f720660defa7389b114\": container with ID starting with f977a47e488d84b54b06f1ad51501c5fe4252a314e0f3f720660defa7389b114 not found: ID does not exist" Oct 30 05:39:48.071487 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.071448 6587 scope.go:122] "RemoveContainer" containerID="250dc0b7b4d0e922ebc9cd1e5dac86cf9d68e5cc75841625e541237809aca257" Oct 30 05:39:48.071689 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:48.071670 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"250dc0b7b4d0e922ebc9cd1e5dac86cf9d68e5cc75841625e541237809aca257\": container with ID starting with 250dc0b7b4d0e922ebc9cd1e5dac86cf9d68e5cc75841625e541237809aca257 not found: ID does not exist" containerID="250dc0b7b4d0e922ebc9cd1e5dac86cf9d68e5cc75841625e541237809aca257" Oct 30 05:39:48.071759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.071703 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250dc0b7b4d0e922ebc9cd1e5dac86cf9d68e5cc75841625e541237809aca257"} err="failed to get container status \"250dc0b7b4d0e922ebc9cd1e5dac86cf9d68e5cc75841625e541237809aca257\": rpc error: code = NotFound desc = could not find container \"250dc0b7b4d0e922ebc9cd1e5dac86cf9d68e5cc75841625e541237809aca257\": container with ID starting with 250dc0b7b4d0e922ebc9cd1e5dac86cf9d68e5cc75841625e541237809aca257 not found: ID does not exist" Oct 30 05:39:48.165580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.165548 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-token\" (UniqueName: \"kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-token\") pod \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " Oct 30 05:39:48.165749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.165606 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-socket\") pod \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " Oct 30 05:39:48.165749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.165642 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-certs\") pod \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " Oct 30 05:39:48.165749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.165673 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-credential-socket\") pod \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " Oct 30 05:39:48.165749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.165713 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istiod-ca-cert\") pod \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " Oct 30 05:39:48.165983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.165770 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-envoy\") pod \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " Oct 30 05:39:48.165983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.165845 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-podinfo\") pod \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " Oct 30 05:39:48.165983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.165915 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-kube-api-access-74mz4\" (UniqueName: \"kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-kube-api-access-74mz4\") pod \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " Oct 30 05:39:48.166160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.165999 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-data\") pod \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\" (UID: \"2e7930f4-1e78-4bd5-be93-010fe7e02d20\") " Oct 30 05:39:48.166160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.166056 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-certs" pod "2e7930f4-1e78-4bd5-be93-010fe7e02d20" (UID: "2e7930f4-1e78-4bd5-be93-010fe7e02d20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:48.166160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.166073 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-credential-socket" pod "2e7930f4-1e78-4bd5-be93-010fe7e02d20" (UID: "2e7930f4-1e78-4bd5-be93-010fe7e02d20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:48.166160 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.166082 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-socket" pod "2e7930f4-1e78-4bd5-be93-010fe7e02d20" (UID: "2e7930f4-1e78-4bd5-be93-010fe7e02d20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:48.166407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.166387 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-data" pod "2e7930f4-1e78-4bd5-be93-010fe7e02d20" (UID: "2e7930f4-1e78-4bd5-be93-010fe7e02d20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:48.166510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.166484 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.166510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.166504 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.166621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.166519 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.166621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.166509 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istiod-ca-cert" pod "2e7930f4-1e78-4bd5-be93-010fe7e02d20" (UID: "2e7930f4-1e78-4bd5-be93-010fe7e02d20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:48.166621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.166528 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.168207 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.168184 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-podinfo" pod "2e7930f4-1e78-4bd5-be93-010fe7e02d20" (UID: "2e7930f4-1e78-4bd5-be93-010fe7e02d20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:39:48.168410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.168338 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-envoy" pod "2e7930f4-1e78-4bd5-be93-010fe7e02d20" (UID: "2e7930f4-1e78-4bd5-be93-010fe7e02d20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:48.168954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.168928 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-token" pod "2e7930f4-1e78-4bd5-be93-010fe7e02d20" (UID: "2e7930f4-1e78-4bd5-be93-010fe7e02d20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:48.168954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.168946 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-kube-api-access-74mz4" pod "2e7930f4-1e78-4bd5-be93-010fe7e02d20" (UID: "2e7930f4-1e78-4bd5-be93-010fe7e02d20"). InnerVolumeSpecName "kube-api-access-74mz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:48.267435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.267395 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.267435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.267426 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74mz4\" (UniqueName: \"kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-kube-api-access-74mz4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.267435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.267436 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.267435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.267446 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.267435 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.267454 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7930f4-1e78-4bd5-be93-010fe7e02d20-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:48.348152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.348112 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8"] Oct 30 05:39:48.429445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.429411 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn"] Oct 30 05:39:48.436766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.436712 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn"] Oct 30 05:39:48.707581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:48.707491 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" podUID="38da181c-3cfa-48da-b727-08064c3d57fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:39:49.056505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:49.056466 6587 generic.go:356] "Generic (PLEG): container finished" podID="56577f79-3dfd-46ce-adf2-1e7ec2db9fce" containerID="65bde700dcc49ac47e19599a45d7c2a18140f68f53be0ca5b8a6d8db7d803644" exitCode=0 Oct 30 05:39:49.056716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:49.056538 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" event={"ID":"56577f79-3dfd-46ce-adf2-1e7ec2db9fce","Type":"ContainerDied","Data":"65bde700dcc49ac47e19599a45d7c2a18140f68f53be0ca5b8a6d8db7d803644"} Oct 30 05:39:49.292441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:49.292407 6587 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:39:49.292820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:49.292472 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" podUID="8994c5b4-d21b-4b3f-864a-68fa175ab737" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:39:49.582165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:49.582131 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" path="/var/lib/kubelet/pods/2e7930f4-1e78-4bd5-be93-010fe7e02d20/volumes" Oct 30 05:39:50.209350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:50.209322 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" Oct 30 05:39:50.293840 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:50.293809 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-xks-ca\") pod \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\" (UID: \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\") " Oct 30 05:39:50.294282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:50.293948 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-openshift-ca\") pod \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\" (UID: \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\") " Oct 30 05:39:50.294282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:50.294090 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-kube-api-access-k4qq7\" (UniqueName: \"kubernetes.io/projected/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-kube-api-access-k4qq7\") pod \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\" (UID: \"56577f79-3dfd-46ce-adf2-1e7ec2db9fce\") " Oct 30 05:39:50.294478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:50.294292 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-openshift-ca" pod "56577f79-3dfd-46ce-adf2-1e7ec2db9fce" (UID: "56577f79-3dfd-46ce-adf2-1e7ec2db9fce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:50.294597 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:50.294576 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:50.295907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:50.295886 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-xks-ca" pod "56577f79-3dfd-46ce-adf2-1e7ec2db9fce" (UID: "56577f79-3dfd-46ce-adf2-1e7ec2db9fce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:39:50.296116 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:50.296094 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-kube-api-access-k4qq7" pod "56577f79-3dfd-46ce-adf2-1e7ec2db9fce" (UID: "56577f79-3dfd-46ce-adf2-1e7ec2db9fce"). InnerVolumeSpecName "kube-api-access-k4qq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:50.400265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:50.396607 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4qq7\" (UniqueName: \"kubernetes.io/projected/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-kube-api-access-k4qq7\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:50.400265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:50.396646 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56577f79-3dfd-46ce-adf2-1e7ec2db9fce-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:51.073615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:51.073587 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" Oct 30 05:39:51.073845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:51.073604 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m" event={"ID":"56577f79-3dfd-46ce-adf2-1e7ec2db9fce","Type":"ContainerDied","Data":"8bb4bb80556a1eedf0dae5e573912ff870bd09e8159b08031a62e5d0335c7099"} Oct 30 05:39:51.073845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:51.073662 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bb4bb80556a1eedf0dae5e573912ff870bd09e8159b08031a62e5d0335c7099" Oct 30 05:39:51.875415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:51.875393 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:39:51.904940 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:51.904916 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:39:52.012314 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012277 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-certs\") pod \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " Oct 30 05:39:52.012522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012332 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-envoy\") pod \"8994c5b4-d21b-4b3f-864a-68fa175ab737\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " Oct 30 05:39:52.012522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012410 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-podinfo\") pod \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " Oct 30 05:39:52.012522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012451 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8994c5b4-d21b-4b3f-864a-68fa175ab737-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8994c5b4-d21b-4b3f-864a-68fa175ab737-istiod-ca-cert\") pod \"8994c5b4-d21b-4b3f-864a-68fa175ab737\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " Oct 30 05:39:52.012711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012523 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-data\") pod \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " Oct 30 05:39:52.012711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012567 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-socket\") pod \"8994c5b4-d21b-4b3f-864a-68fa175ab737\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " Oct 30 05:39:52.012711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012600 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-socket\") pod \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " Oct 30 05:39:52.012711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012609 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-certs" pod "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" (UID: "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.012711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012647 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-kube-api-access-rcltz\" (UniqueName: \"kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-kube-api-access-rcltz\") pod \"8994c5b4-d21b-4b3f-864a-68fa175ab737\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " Oct 30 05:39:52.012711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012690 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-data\") pod \"8994c5b4-d21b-4b3f-864a-68fa175ab737\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " Oct 30 05:39:52.013073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012741 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-credential-socket\") pod \"8994c5b4-d21b-4b3f-864a-68fa175ab737\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " Oct 30 05:39:52.013073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012772 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istiod-ca-cert\") pod \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " Oct 30 05:39:52.013073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012801 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-podinfo\") pod \"8994c5b4-d21b-4b3f-864a-68fa175ab737\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " Oct 30 05:39:52.013073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012906 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-certs\") pod \"8994c5b4-d21b-4b3f-864a-68fa175ab737\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " Oct 30 05:39:52.013073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012920 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-socket" pod "8994c5b4-d21b-4b3f-864a-68fa175ab737" (UID: "8994c5b4-d21b-4b3f-864a-68fa175ab737"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.013073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012927 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-socket" pod "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" (UID: "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.013073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012963 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-envoy\") pod \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " Oct 30 05:39:52.013073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012975 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8994c5b4-d21b-4b3f-864a-68fa175ab737-istiod-ca-cert" pod "8994c5b4-d21b-4b3f-864a-68fa175ab737" (UID: "8994c5b4-d21b-4b3f-864a-68fa175ab737"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:52.013073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.012996 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-token\" (UniqueName: \"kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-token\") pod \"8994c5b4-d21b-4b3f-864a-68fa175ab737\" (UID: \"8994c5b4-d21b-4b3f-864a-68fa175ab737\") " Oct 30 05:39:52.013073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013049 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-credential-socket\") pod \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " Oct 30 05:39:52.013684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013089 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-kube-api-access-k64jw\" (UniqueName: \"kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-kube-api-access-k64jw\") pod \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " Oct 30 05:39:52.013684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013126 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-token\" (UniqueName: \"kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-token\") pod \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\" (UID: \"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a\") " Oct 30 05:39:52.013684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013184 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-credential-socket" pod "8994c5b4-d21b-4b3f-864a-68fa175ab737" (UID: "8994c5b4-d21b-4b3f-864a-68fa175ab737"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.013684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013187 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-data" pod "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" (UID: "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.013684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013442 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-data" pod "8994c5b4-d21b-4b3f-864a-68fa175ab737" (UID: "8994c5b4-d21b-4b3f-864a-68fa175ab737"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.013684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013617 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.013684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013636 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.013684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013652 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.013684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013666 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.013684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013679 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.014238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013694 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.014238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013709 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8994c5b4-d21b-4b3f-864a-68fa175ab737-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.014238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013952 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-credential-socket" pod "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" (UID: "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.014238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.013962 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-certs" pod "8994c5b4-d21b-4b3f-864a-68fa175ab737" (UID: "8994c5b4-d21b-4b3f-864a-68fa175ab737"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.014516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.014278 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istiod-ca-cert" pod "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" (UID: "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:52.015444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.015416 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-envoy" pod "8994c5b4-d21b-4b3f-864a-68fa175ab737" (UID: "8994c5b4-d21b-4b3f-864a-68fa175ab737"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.016558 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.016532 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-token" pod "8994c5b4-d21b-4b3f-864a-68fa175ab737" (UID: "8994c5b4-d21b-4b3f-864a-68fa175ab737"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:52.016659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.016598 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-kube-api-access-k64jw" pod "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" (UID: "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a"). InnerVolumeSpecName "kube-api-access-k64jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:52.016893 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.016870 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-envoy" pod "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" (UID: "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:52.017003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.016893 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-token" pod "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" (UID: "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:52.017257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.017222 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-podinfo" pod "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" (UID: "c24c4b5a-53e8-4f99-ba51-69ec7b6a994a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:39:52.018410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.018385 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-podinfo" pod "8994c5b4-d21b-4b3f-864a-68fa175ab737" (UID: "8994c5b4-d21b-4b3f-864a-68fa175ab737"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:39:52.018519 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.018402 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-kube-api-access-rcltz" pod "8994c5b4-d21b-4b3f-864a-68fa175ab737" (UID: "8994c5b4-d21b-4b3f-864a-68fa175ab737"). InnerVolumeSpecName "kube-api-access-rcltz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:52.081484 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.081450 6587 generic.go:356] "Generic (PLEG): container finished" podID="8994c5b4-d21b-4b3f-864a-68fa175ab737" containerID="24bddc0bd1383638807f8a334f2459e789ec8f708e9e706c2ab76aaf9eab0d7d" exitCode=0 Oct 30 05:39:52.081661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.081499 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" Oct 30 05:39:52.081661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.081522 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" event={"ID":"8994c5b4-d21b-4b3f-864a-68fa175ab737","Type":"ContainerDied","Data":"24bddc0bd1383638807f8a334f2459e789ec8f708e9e706c2ab76aaf9eab0d7d"} Oct 30 05:39:52.081661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.081574 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh" event={"ID":"8994c5b4-d21b-4b3f-864a-68fa175ab737","Type":"ContainerDied","Data":"387ad3c91d14431e8c2d039b9b9021891045a5576763718ad449a7196d9243b2"} Oct 30 05:39:52.081661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.081601 6587 scope.go:122] "RemoveContainer" containerID="24bddc0bd1383638807f8a334f2459e789ec8f708e9e706c2ab76aaf9eab0d7d" Oct 30 05:39:52.083474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.083449 6587 generic.go:356] "Generic (PLEG): container finished" podID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerID="62ed9e0c8d5903e83e05f3186145a82721261c2c668309cdb18537a48a403536" exitCode=0 Oct 30 05:39:52.083583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.083502 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" Oct 30 05:39:52.083583 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.083535 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" event={"ID":"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a","Type":"ContainerDied","Data":"62ed9e0c8d5903e83e05f3186145a82721261c2c668309cdb18537a48a403536"} Oct 30 05:39:52.083670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.083592 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw" event={"ID":"c24c4b5a-53e8-4f99-ba51-69ec7b6a994a","Type":"ContainerDied","Data":"f2f45eddcd0f0f7277db6a84b7867c64b58533d021bf710d3c491141f9f59726"} Oct 30 05:39:52.093858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.093843 6587 scope.go:122] "RemoveContainer" containerID="24bddc0bd1383638807f8a334f2459e789ec8f708e9e706c2ab76aaf9eab0d7d" Oct 30 05:39:52.094133 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:52.094116 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24bddc0bd1383638807f8a334f2459e789ec8f708e9e706c2ab76aaf9eab0d7d\": container with ID starting with 24bddc0bd1383638807f8a334f2459e789ec8f708e9e706c2ab76aaf9eab0d7d not found: ID does not exist" containerID="24bddc0bd1383638807f8a334f2459e789ec8f708e9e706c2ab76aaf9eab0d7d" Oct 30 05:39:52.094184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.094142 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bddc0bd1383638807f8a334f2459e789ec8f708e9e706c2ab76aaf9eab0d7d"} err="failed to get container status \"24bddc0bd1383638807f8a334f2459e789ec8f708e9e706c2ab76aaf9eab0d7d\": rpc error: code = NotFound desc = could not find container \"24bddc0bd1383638807f8a334f2459e789ec8f708e9e706c2ab76aaf9eab0d7d\": container with ID starting with 24bddc0bd1383638807f8a334f2459e789ec8f708e9e706c2ab76aaf9eab0d7d not found: ID does not exist" Oct 30 05:39:52.094184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.094159 6587 scope.go:122] "RemoveContainer" containerID="62ed9e0c8d5903e83e05f3186145a82721261c2c668309cdb18537a48a403536" Oct 30 05:39:52.104109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.104092 6587 scope.go:122] "RemoveContainer" containerID="c3936e4be7a46060b4f86bd849a2f18c5bf415c3bc4e7dc9df76e8c6482bbf4a" Oct 30 05:39:52.113603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.113587 6587 scope.go:122] "RemoveContainer" containerID="62ed9e0c8d5903e83e05f3186145a82721261c2c668309cdb18537a48a403536" Oct 30 05:39:52.113852 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:52.113829 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ed9e0c8d5903e83e05f3186145a82721261c2c668309cdb18537a48a403536\": container with ID starting with 62ed9e0c8d5903e83e05f3186145a82721261c2c668309cdb18537a48a403536 not found: ID does not exist" containerID="62ed9e0c8d5903e83e05f3186145a82721261c2c668309cdb18537a48a403536" Oct 30 05:39:52.113917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.113861 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ed9e0c8d5903e83e05f3186145a82721261c2c668309cdb18537a48a403536"} err="failed to get container status \"62ed9e0c8d5903e83e05f3186145a82721261c2c668309cdb18537a48a403536\": rpc error: code = NotFound desc = could not find container \"62ed9e0c8d5903e83e05f3186145a82721261c2c668309cdb18537a48a403536\": container with ID starting with 62ed9e0c8d5903e83e05f3186145a82721261c2c668309cdb18537a48a403536 not found: ID does not exist" Oct 30 05:39:52.113917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.113880 6587 scope.go:122] "RemoveContainer" containerID="c3936e4be7a46060b4f86bd849a2f18c5bf415c3bc4e7dc9df76e8c6482bbf4a" Oct 30 05:39:52.114147 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:52.114127 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3936e4be7a46060b4f86bd849a2f18c5bf415c3bc4e7dc9df76e8c6482bbf4a\": container with ID starting with c3936e4be7a46060b4f86bd849a2f18c5bf415c3bc4e7dc9df76e8c6482bbf4a not found: ID does not exist" containerID="c3936e4be7a46060b4f86bd849a2f18c5bf415c3bc4e7dc9df76e8c6482bbf4a" Oct 30 05:39:52.114193 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.114154 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3936e4be7a46060b4f86bd849a2f18c5bf415c3bc4e7dc9df76e8c6482bbf4a"} err="failed to get container status \"c3936e4be7a46060b4f86bd849a2f18c5bf415c3bc4e7dc9df76e8c6482bbf4a\": rpc error: code = NotFound desc = could not find container \"c3936e4be7a46060b4f86bd849a2f18c5bf415c3bc4e7dc9df76e8c6482bbf4a\": container with ID starting with c3936e4be7a46060b4f86bd849a2f18c5bf415c3bc4e7dc9df76e8c6482bbf4a not found: ID does not exist" Oct 30 05:39:52.114318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.114302 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.114382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.114323 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.114382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.114332 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.114382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.114341 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.114382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.114351 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k64jw\" (UniqueName: \"kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-kube-api-access-k64jw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.114549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.114395 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.114549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.114408 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.114549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.114420 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.114549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.114431 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcltz\" (UniqueName: \"kubernetes.io/projected/8994c5b4-d21b-4b3f-864a-68fa175ab737-kube-api-access-rcltz\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.114549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.114443 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.114549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.114455 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8994c5b4-d21b-4b3f-864a-68fa175ab737-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:52.135697 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.135671 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh"] Oct 30 05:39:52.140000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.139974 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh"] Oct 30 05:39:52.175129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.175105 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw"] Oct 30 05:39:52.177467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:52.177432 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw"] Oct 30 05:39:53.581855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:53.581816 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8994c5b4-d21b-4b3f-864a-68fa175ab737" path="/var/lib/kubelet/pods/8994c5b4-d21b-4b3f-864a-68fa175ab737/volumes" Oct 30 05:39:53.582517 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:53.582499 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" path="/var/lib/kubelet/pods/c24c4b5a-53e8-4f99-ba51-69ec7b6a994a/volumes" Oct 30 05:39:53.933212 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:53.933124 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp"] Oct 30 05:39:54.887791 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:39:54.887768 6587 logging.go:55] [core] [Channel #16461 SubChannel #16463]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.32:50052", ServerName: "10.134.1.32:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.32:50052: connect: connection refused" Oct 30 05:39:54.888230 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:39:54.887803 6587 logging.go:55] [core] [Channel #16460 SubChannel #16462]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.33:50052", ServerName: "10.134.1.33:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.33:50052: connect: connection refused" Oct 30 05:39:55.058064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.058037 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" Oct 30 05:39:55.105437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.105413 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" Oct 30 05:39:55.111186 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.111159 6587 generic.go:356] "Generic (PLEG): container finished" podID="3d330d3a-1853-4bc5-92a6-d7b13bcf2466" containerID="c5ce09fcf677c8c7754a797f2504397e08430bdd9aac783d4e6941e006dcface" exitCode=0 Oct 30 05:39:55.111318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.111200 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" Oct 30 05:39:55.111318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.111222 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" event={"ID":"3d330d3a-1853-4bc5-92a6-d7b13bcf2466","Type":"ContainerDied","Data":"c5ce09fcf677c8c7754a797f2504397e08430bdd9aac783d4e6941e006dcface"} Oct 30 05:39:55.111318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.111274 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" event={"ID":"3d330d3a-1853-4bc5-92a6-d7b13bcf2466","Type":"ContainerDied","Data":"fa71f06a6edee646b7b1db561ae61478bc29d20124a5f8d9053745c1d63105ab"} Oct 30 05:39:55.111318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.111302 6587 scope.go:122] "RemoveContainer" containerID="c5ce09fcf677c8c7754a797f2504397e08430bdd9aac783d4e6941e006dcface" Oct 30 05:39:55.112815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.112790 6587 generic.go:356] "Generic (PLEG): container finished" podID="3c5ff08b-f239-4e07-863a-3653c81a6708" containerID="3b5a472e2c0750c8b11ad491bf2ca7da9e9f3bd5c5214959f3f5f5e67dd4f931" exitCode=0 Oct 30 05:39:55.112938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.112830 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" Oct 30 05:39:55.112938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.112895 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" event={"ID":"3c5ff08b-f239-4e07-863a-3653c81a6708","Type":"ContainerDied","Data":"3b5a472e2c0750c8b11ad491bf2ca7da9e9f3bd5c5214959f3f5f5e67dd4f931"} Oct 30 05:39:55.113034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.112947 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" event={"ID":"3c5ff08b-f239-4e07-863a-3653c81a6708","Type":"ContainerDied","Data":"b0bd31a676b030ca2d2c100da272723a3dcadecacaace560043b43b6ce1eedd9"} Oct 30 05:39:55.124000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.123981 6587 scope.go:122] "RemoveContainer" containerID="c5ce09fcf677c8c7754a797f2504397e08430bdd9aac783d4e6941e006dcface" Oct 30 05:39:55.124314 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:55.124288 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ce09fcf677c8c7754a797f2504397e08430bdd9aac783d4e6941e006dcface\": container with ID starting with c5ce09fcf677c8c7754a797f2504397e08430bdd9aac783d4e6941e006dcface not found: ID does not exist" containerID="c5ce09fcf677c8c7754a797f2504397e08430bdd9aac783d4e6941e006dcface" Oct 30 05:39:55.124428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.124324 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ce09fcf677c8c7754a797f2504397e08430bdd9aac783d4e6941e006dcface"} err="failed to get container status \"c5ce09fcf677c8c7754a797f2504397e08430bdd9aac783d4e6941e006dcface\": rpc error: code = NotFound desc = could not find container \"c5ce09fcf677c8c7754a797f2504397e08430bdd9aac783d4e6941e006dcface\": container with ID starting with c5ce09fcf677c8c7754a797f2504397e08430bdd9aac783d4e6941e006dcface not found: ID does not exist" Oct 30 05:39:55.124428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.124345 6587 scope.go:122] "RemoveContainer" containerID="3b5a472e2c0750c8b11ad491bf2ca7da9e9f3bd5c5214959f3f5f5e67dd4f931" Oct 30 05:39:55.135466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.135401 6587 scope.go:122] "RemoveContainer" containerID="3b5a472e2c0750c8b11ad491bf2ca7da9e9f3bd5c5214959f3f5f5e67dd4f931" Oct 30 05:39:55.135715 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:55.135692 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5a472e2c0750c8b11ad491bf2ca7da9e9f3bd5c5214959f3f5f5e67dd4f931\": container with ID starting with 3b5a472e2c0750c8b11ad491bf2ca7da9e9f3bd5c5214959f3f5f5e67dd4f931 not found: ID does not exist" containerID="3b5a472e2c0750c8b11ad491bf2ca7da9e9f3bd5c5214959f3f5f5e67dd4f931" Oct 30 05:39:55.135853 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.135719 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5a472e2c0750c8b11ad491bf2ca7da9e9f3bd5c5214959f3f5f5e67dd4f931"} err="failed to get container status \"3b5a472e2c0750c8b11ad491bf2ca7da9e9f3bd5c5214959f3f5f5e67dd4f931\": rpc error: code = NotFound desc = could not find container \"3b5a472e2c0750c8b11ad491bf2ca7da9e9f3bd5c5214959f3f5f5e67dd4f931\": container with ID starting with 3b5a472e2c0750c8b11ad491bf2ca7da9e9f3bd5c5214959f3f5f5e67dd4f931 not found: ID does not exist" Oct 30 05:39:55.147065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.147037 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-plugins-config-volume\") pod \"3d330d3a-1853-4bc5-92a6-d7b13bcf2466\" (UID: \"3d330d3a-1853-4bc5-92a6-d7b13bcf2466\") " Oct 30 05:39:55.147511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.147205 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-kube-api-access-f2zxz\" (UniqueName: \"kubernetes.io/projected/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-kube-api-access-f2zxz\") pod \"3d330d3a-1853-4bc5-92a6-d7b13bcf2466\" (UID: \"3d330d3a-1853-4bc5-92a6-d7b13bcf2466\") " Oct 30 05:39:55.147511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.147478 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-plugins-config-volume" pod "3d330d3a-1853-4bc5-92a6-d7b13bcf2466" (UID: "3d330d3a-1853-4bc5-92a6-d7b13bcf2466"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:55.147789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.147767 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:55.149492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.149454 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-kube-api-access-f2zxz" pod "3d330d3a-1853-4bc5-92a6-d7b13bcf2466" (UID: "3d330d3a-1853-4bc5-92a6-d7b13bcf2466"). InnerVolumeSpecName "kube-api-access-f2zxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:55.248870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.248838 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c5ff08b-f239-4e07-863a-3653c81a6708-kube-api-access-pttjj\" (UniqueName: \"kubernetes.io/projected/3c5ff08b-f239-4e07-863a-3653c81a6708-kube-api-access-pttjj\") pod \"3c5ff08b-f239-4e07-863a-3653c81a6708\" (UID: \"3c5ff08b-f239-4e07-863a-3653c81a6708\") " Oct 30 05:39:55.249044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.248964 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c5ff08b-f239-4e07-863a-3653c81a6708-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5ff08b-f239-4e07-863a-3653c81a6708-plugins-config-volume\") pod \"3c5ff08b-f239-4e07-863a-3653c81a6708\" (UID: \"3c5ff08b-f239-4e07-863a-3653c81a6708\") " Oct 30 05:39:55.249436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.249411 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5ff08b-f239-4e07-863a-3653c81a6708-plugins-config-volume" pod "3c5ff08b-f239-4e07-863a-3653c81a6708" (UID: "3c5ff08b-f239-4e07-863a-3653c81a6708"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:55.249509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.249494 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2zxz\" (UniqueName: \"kubernetes.io/projected/3d330d3a-1853-4bc5-92a6-d7b13bcf2466-kube-api-access-f2zxz\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:55.250956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.250931 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5ff08b-f239-4e07-863a-3653c81a6708-kube-api-access-pttjj" pod "3c5ff08b-f239-4e07-863a-3653c81a6708" (UID: "3c5ff08b-f239-4e07-863a-3653c81a6708"). InnerVolumeSpecName "kube-api-access-pttjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:55.351047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.351010 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5ff08b-f239-4e07-863a-3653c81a6708-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:55.351047 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.351041 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pttjj\" (UniqueName: \"kubernetes.io/projected/3c5ff08b-f239-4e07-863a-3653c81a6708-kube-api-access-pttjj\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:55.465747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.465711 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd"] Oct 30 05:39:55.472950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.472918 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd"] Oct 30 05:39:55.504038 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.504003 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz"] Oct 30 05:39:55.509261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.509227 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz"] Oct 30 05:39:55.582450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.582407 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c5ff08b-f239-4e07-863a-3653c81a6708" path="/var/lib/kubelet/pods/3c5ff08b-f239-4e07-863a-3653c81a6708/volumes" Oct 30 05:39:55.583023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.582995 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d330d3a-1853-4bc5-92a6-d7b13bcf2466" path="/var/lib/kubelet/pods/3d330d3a-1853-4bc5-92a6-d7b13bcf2466/volumes" Oct 30 05:39:55.887719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.887670 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd" podUID="3d330d3a-1853-4bc5-92a6-d7b13bcf2466" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.33:50052\" within 1s: context deadline exceeded" Oct 30 05:39:55.887905 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:55.887680 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz" podUID="3c5ff08b-f239-4e07-863a-3653c81a6708" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.32:50052\" within 1s: context deadline exceeded" Oct 30 05:39:55.985416 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:55.985348 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:39:56.788230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:56.788165 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" Oct 30 05:39:56.867317 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:56.867285 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38da181c-3cfa-48da-b727-08064c3d57fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38da181c-3cfa-48da-b727-08064c3d57fe-plugins-config-volume\") pod \"38da181c-3cfa-48da-b727-08064c3d57fe\" (UID: \"38da181c-3cfa-48da-b727-08064c3d57fe\") " Oct 30 05:39:56.867493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:56.867436 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38da181c-3cfa-48da-b727-08064c3d57fe-kube-api-access-rpzfk\" (UniqueName: \"kubernetes.io/projected/38da181c-3cfa-48da-b727-08064c3d57fe-kube-api-access-rpzfk\") pod \"38da181c-3cfa-48da-b727-08064c3d57fe\" (UID: \"38da181c-3cfa-48da-b727-08064c3d57fe\") " Oct 30 05:39:56.867741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:56.867710 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38da181c-3cfa-48da-b727-08064c3d57fe-plugins-config-volume" pod "38da181c-3cfa-48da-b727-08064c3d57fe" (UID: "38da181c-3cfa-48da-b727-08064c3d57fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:56.870050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:56.870027 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38da181c-3cfa-48da-b727-08064c3d57fe-kube-api-access-rpzfk" pod "38da181c-3cfa-48da-b727-08064c3d57fe" (UID: "38da181c-3cfa-48da-b727-08064c3d57fe"). InnerVolumeSpecName "kube-api-access-rpzfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:56.969299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:56.969267 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38da181c-3cfa-48da-b727-08064c3d57fe-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:56.969299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:56.969295 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpzfk\" (UniqueName: \"kubernetes.io/projected/38da181c-3cfa-48da-b727-08064c3d57fe-kube-api-access-rpzfk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:57.131796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:57.131762 6587 generic.go:356] "Generic (PLEG): container finished" podID="38da181c-3cfa-48da-b727-08064c3d57fe" containerID="b6bd2e82274909902811212bc020634f12a2f9fe6574ad117156183873129f7e" exitCode=0 Oct 30 05:39:57.132021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:57.131807 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" event={"ID":"38da181c-3cfa-48da-b727-08064c3d57fe","Type":"ContainerDied","Data":"b6bd2e82274909902811212bc020634f12a2f9fe6574ad117156183873129f7e"} Oct 30 05:39:57.132021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:57.131817 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" Oct 30 05:39:57.132021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:57.131847 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv" event={"ID":"38da181c-3cfa-48da-b727-08064c3d57fe","Type":"ContainerDied","Data":"914ce144dfa2de8ef6f6ac8685e6994fb548b53c45e4eaa4f44a585beba6034c"} Oct 30 05:39:57.132021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:57.131868 6587 scope.go:122] "RemoveContainer" containerID="b6bd2e82274909902811212bc020634f12a2f9fe6574ad117156183873129f7e" Oct 30 05:39:57.147636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:57.147616 6587 scope.go:122] "RemoveContainer" containerID="b6bd2e82274909902811212bc020634f12a2f9fe6574ad117156183873129f7e" Oct 30 05:39:57.147947 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:39:57.147924 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6bd2e82274909902811212bc020634f12a2f9fe6574ad117156183873129f7e\": container with ID starting with b6bd2e82274909902811212bc020634f12a2f9fe6574ad117156183873129f7e not found: ID does not exist" containerID="b6bd2e82274909902811212bc020634f12a2f9fe6574ad117156183873129f7e" Oct 30 05:39:57.148018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:57.147954 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6bd2e82274909902811212bc020634f12a2f9fe6574ad117156183873129f7e"} err="failed to get container status \"b6bd2e82274909902811212bc020634f12a2f9fe6574ad117156183873129f7e\": rpc error: code = NotFound desc = could not find container \"b6bd2e82274909902811212bc020634f12a2f9fe6574ad117156183873129f7e\": container with ID starting with b6bd2e82274909902811212bc020634f12a2f9fe6574ad117156183873129f7e not found: ID does not exist" Oct 30 05:39:57.183001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:57.182969 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv"] Oct 30 05:39:57.189688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:57.189661 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv"] Oct 30 05:39:57.581650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:57.581612 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38da181c-3cfa-48da-b727-08064c3d57fe" path="/var/lib/kubelet/pods/38da181c-3cfa-48da-b727-08064c3d57fe/volumes" Oct 30 05:39:57.620230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:57.620198 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h"] Oct 30 05:39:57.667478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:57.667441 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb"] Oct 30 05:39:58.030784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:58.030692 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt"] Oct 30 05:39:58.033500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:58.033469 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f"] Oct 30 05:39:58.035205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:58.035189 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d98b018-fbq6f"] Oct 30 05:39:58.037001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:58.036974 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372fea20-wxtgt"] Oct 30 05:39:58.934408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:58.934345 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" podUID="82856c63-29d3-42d2-9078-3880374c9ebc" containerName="payload-processing" containerID="cri-o://e6bae3b936acd7fd399ca27429e6e33590a0d228e516f3573c88fd3eecc64e8f" gracePeriod=25 Oct 30 05:39:59.157303 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.157266 6587 generic.go:356] "Generic (PLEG): container finished" podID="82856c63-29d3-42d2-9078-3880374c9ebc" containerID="e6bae3b936acd7fd399ca27429e6e33590a0d228e516f3573c88fd3eecc64e8f" exitCode=0 Oct 30 05:39:59.157814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.157480 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" event={"ID":"82856c63-29d3-42d2-9078-3880374c9ebc","Type":"ContainerDied","Data":"e6bae3b936acd7fd399ca27429e6e33590a0d228e516f3573c88fd3eecc64e8f"} Oct 30 05:39:59.201756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.201688 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" Oct 30 05:39:59.293481 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.293442 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82856c63-29d3-42d2-9078-3880374c9ebc-kube-api-access-xbtnc\" (UniqueName: \"kubernetes.io/projected/82856c63-29d3-42d2-9078-3880374c9ebc-kube-api-access-xbtnc\") pod \"82856c63-29d3-42d2-9078-3880374c9ebc\" (UID: \"82856c63-29d3-42d2-9078-3880374c9ebc\") " Oct 30 05:39:59.293651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.293587 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82856c63-29d3-42d2-9078-3880374c9ebc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82856c63-29d3-42d2-9078-3880374c9ebc-plugins-config-volume\") pod \"82856c63-29d3-42d2-9078-3880374c9ebc\" (UID: \"82856c63-29d3-42d2-9078-3880374c9ebc\") " Oct 30 05:39:59.293985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.293960 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82856c63-29d3-42d2-9078-3880374c9ebc-plugins-config-volume" pod "82856c63-29d3-42d2-9078-3880374c9ebc" (UID: "82856c63-29d3-42d2-9078-3880374c9ebc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:39:59.295682 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.295657 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82856c63-29d3-42d2-9078-3880374c9ebc-kube-api-access-xbtnc" pod "82856c63-29d3-42d2-9078-3880374c9ebc" (UID: "82856c63-29d3-42d2-9078-3880374c9ebc"). InnerVolumeSpecName "kube-api-access-xbtnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:59.395145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.395107 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82856c63-29d3-42d2-9078-3880374c9ebc-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:59.395145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.395140 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbtnc\" (UniqueName: \"kubernetes.io/projected/82856c63-29d3-42d2-9078-3880374c9ebc-kube-api-access-xbtnc\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:39:59.581836 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.581804 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e40fbdc-a209-4661-8315-6bee6a146ffb" path="/var/lib/kubelet/pods/5e40fbdc-a209-4661-8315-6bee6a146ffb/volumes" Oct 30 05:39:59.582249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.582236 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55" path="/var/lib/kubelet/pods/7bbe4fa6-2b31-4b1d-867b-2a2c26c87f55/volumes" Oct 30 05:39:59.656134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656099 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp"] Oct 30 05:39:59.656602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656587 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" Oct 30 05:39:59.656649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656603 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" Oct 30 05:39:59.656649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656632 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3c5ff08b-f239-4e07-863a-3653c81a6708" containerName="payload-pre-processing" Oct 30 05:39:59.656649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656638 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c5ff08b-f239-4e07-863a-3653c81a6708" containerName="payload-pre-processing" Oct 30 05:39:59.656649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656645 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" Oct 30 05:39:59.656649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656650 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656663 6587 cpu_manager.go:414] "containerMap: removing container" podUID="8994c5b4-d21b-4b3f-864a-68fa175ab737" containerName="istio-proxy" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656669 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8994c5b4-d21b-4b3f-864a-68fa175ab737" containerName="istio-proxy" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656678 6587 cpu_manager.go:414] "containerMap: removing container" podUID="82856c63-29d3-42d2-9078-3880374c9ebc" containerName="payload-processing" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656683 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82856c63-29d3-42d2-9078-3880374c9ebc" containerName="payload-processing" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656694 6587 cpu_manager.go:414] "containerMap: removing container" podUID="38da181c-3cfa-48da-b727-08064c3d57fe" containerName="payload-processing" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656699 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38da181c-3cfa-48da-b727-08064c3d57fe" containerName="payload-processing" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656706 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3d330d3a-1853-4bc5-92a6-d7b13bcf2466" containerName="payload-processing" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656710 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d330d3a-1853-4bc5-92a6-d7b13bcf2466" containerName="payload-processing" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656720 6587 cpu_manager.go:414] "containerMap: removing container" podUID="56577f79-3dfd-46ce-adf2-1e7ec2db9fce" containerName="revoke-keys" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656724 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56577f79-3dfd-46ce-adf2-1e7ec2db9fce" containerName="revoke-keys" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656782 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d330d3a-1853-4bc5-92a6-d7b13bcf2466" containerName="payload-processing" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656791 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656797 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="56577f79-3dfd-46ce-adf2-1e7ec2db9fce" containerName="revoke-keys" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656804 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656813 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c5ff08b-f239-4e07-863a-3653c81a6708" containerName="payload-pre-processing" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656818 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656824 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="8994c5b4-d21b-4b3f-864a-68fa175ab737" containerName="istio-proxy" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656830 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="82856c63-29d3-42d2-9078-3880374c9ebc" containerName="payload-processing" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656837 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" Oct 30 05:39:59.656839 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.656843 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="38da181c-3cfa-48da-b727-08064c3d57fe" containerName="payload-processing" Oct 30 05:39:59.661308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.661288 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" Oct 30 05:39:59.680684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.680661 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:39:59.680918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.680900 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:39:59.692483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.692450 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:39:59.692619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.692464 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:39:59.696041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.696016 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp"] Oct 30 05:39:59.801199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.801163 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e28d932-4790-4f6b-880c-338534347104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f697f9f9-7q9rp\" (UID: \"6e28d932-4790-4f6b-880c-338534347104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" Oct 30 05:39:59.801408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.801206 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e28d932-4790-4f6b-880c-338534347104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f697f9f9-7q9rp\" (UID: \"6e28d932-4790-4f6b-880c-338534347104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" Oct 30 05:39:59.801408 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.801399 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbdzk\" (UniqueName: \"kubernetes.io/projected/6e28d932-4790-4f6b-880c-338534347104-kube-api-access-zbdzk\") pod \"maas-api-revoke-sub-f697f9f9-7q9rp\" (UID: \"6e28d932-4790-4f6b-880c-338534347104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" Oct 30 05:39:59.902573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.902483 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e28d932-4790-4f6b-880c-338534347104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f697f9f9-7q9rp\" (UID: \"6e28d932-4790-4f6b-880c-338534347104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" Oct 30 05:39:59.902756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.902677 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbdzk\" (UniqueName: \"kubernetes.io/projected/6e28d932-4790-4f6b-880c-338534347104-kube-api-access-zbdzk\") pod \"maas-api-revoke-sub-f697f9f9-7q9rp\" (UID: \"6e28d932-4790-4f6b-880c-338534347104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" Oct 30 05:39:59.902756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.902711 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e28d932-4790-4f6b-880c-338534347104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f697f9f9-7q9rp\" (UID: \"6e28d932-4790-4f6b-880c-338534347104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" Oct 30 05:39:59.903266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.903248 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e28d932-4790-4f6b-880c-338534347104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f697f9f9-7q9rp\" (UID: \"6e28d932-4790-4f6b-880c-338534347104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" Oct 30 05:39:59.905096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.905068 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e28d932-4790-4f6b-880c-338534347104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f697f9f9-7q9rp\" (UID: \"6e28d932-4790-4f6b-880c-338534347104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" Oct 30 05:39:59.924232 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.924209 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbdzk\" (UniqueName: \"kubernetes.io/projected/6e28d932-4790-4f6b-880c-338534347104-kube-api-access-zbdzk\") pod \"maas-api-revoke-sub-f697f9f9-7q9rp\" (UID: \"6e28d932-4790-4f6b-880c-338534347104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" Oct 30 05:39:59.975609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:39:59.975587 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" Oct 30 05:40:00.165042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:00.164965 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" Oct 30 05:40:00.165042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:00.164980 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp" event={"ID":"82856c63-29d3-42d2-9078-3880374c9ebc","Type":"ContainerDied","Data":"be6e6e717c1d05c257ede3475511b32c0ac2b2c3de495736262b2a7dead8794b"} Oct 30 05:40:00.165042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:00.165035 6587 scope.go:122] "RemoveContainer" containerID="e6bae3b936acd7fd399ca27429e6e33590a0d228e516f3573c88fd3eecc64e8f" Oct 30 05:40:00.220552 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:00.220526 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp"] Oct 30 05:40:00.225645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:00.225622 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp"] Oct 30 05:40:00.275872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:00.275845 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp"] Oct 30 05:40:00.281202 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:40:00.281170 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e28d932_4790_4f6b_880c_338534347104.slice/crio-6c2ff5a97fe1861ee5370a9054afa7021531e532d58e8869bec6d9d2bfe704fe WatchSource:0}: Error finding container 6c2ff5a97fe1861ee5370a9054afa7021531e532d58e8869bec6d9d2bfe704fe: Status 404 returned error can't find the container with id 6c2ff5a97fe1861ee5370a9054afa7021531e532d58e8869bec6d9d2bfe704fe Oct 30 05:40:01.174443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:01.174413 6587 generic.go:356] "Generic (PLEG): container finished" podID="6e28d932-4790-4f6b-880c-338534347104" containerID="b54ef70f7a713f535267aea340d64f1dfeb7476fddf2f9f8f8c88b633830c5e6" exitCode=0 Oct 30 05:40:01.174778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:01.174494 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" event={"ID":"6e28d932-4790-4f6b-880c-338534347104","Type":"ContainerDied","Data":"b54ef70f7a713f535267aea340d64f1dfeb7476fddf2f9f8f8c88b633830c5e6"} Oct 30 05:40:01.174778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:01.174542 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" event={"ID":"6e28d932-4790-4f6b-880c-338534347104","Type":"ContainerStarted","Data":"6c2ff5a97fe1861ee5370a9054afa7021531e532d58e8869bec6d9d2bfe704fe"} Oct 30 05:40:01.581316 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:01.581281 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82856c63-29d3-42d2-9078-3880374c9ebc" path="/var/lib/kubelet/pods/82856c63-29d3-42d2-9078-3880374c9ebc/volumes" Oct 30 05:40:02.281828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.281797 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m"] Oct 30 05:40:02.293450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.293410 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9dc17a-smd8m"] Oct 30 05:40:02.320062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.320036 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" Oct 30 05:40:02.428071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.428033 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e28d932-4790-4f6b-880c-338534347104-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e28d932-4790-4f6b-880c-338534347104-maas-api-openshift-ca\") pod \"6e28d932-4790-4f6b-880c-338534347104\" (UID: \"6e28d932-4790-4f6b-880c-338534347104\") " Oct 30 05:40:02.428286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.428163 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e28d932-4790-4f6b-880c-338534347104-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e28d932-4790-4f6b-880c-338534347104-maas-api-xks-ca\") pod \"6e28d932-4790-4f6b-880c-338534347104\" (UID: \"6e28d932-4790-4f6b-880c-338534347104\") " Oct 30 05:40:02.428351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.428313 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e28d932-4790-4f6b-880c-338534347104-kube-api-access-zbdzk\" (UniqueName: \"kubernetes.io/projected/6e28d932-4790-4f6b-880c-338534347104-kube-api-access-zbdzk\") pod \"6e28d932-4790-4f6b-880c-338534347104\" (UID: \"6e28d932-4790-4f6b-880c-338534347104\") " Oct 30 05:40:02.428536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.428508 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e28d932-4790-4f6b-880c-338534347104-maas-api-openshift-ca" pod "6e28d932-4790-4f6b-880c-338534347104" (UID: "6e28d932-4790-4f6b-880c-338534347104"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:02.428776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.428754 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e28d932-4790-4f6b-880c-338534347104-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:02.430436 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.430407 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e28d932-4790-4f6b-880c-338534347104-maas-api-xks-ca" pod "6e28d932-4790-4f6b-880c-338534347104" (UID: "6e28d932-4790-4f6b-880c-338534347104"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:02.430565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.430541 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e28d932-4790-4f6b-880c-338534347104-kube-api-access-zbdzk" pod "6e28d932-4790-4f6b-880c-338534347104" (UID: "6e28d932-4790-4f6b-880c-338534347104"). InnerVolumeSpecName "kube-api-access-zbdzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:02.529574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.529538 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e28d932-4790-4f6b-880c-338534347104-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:02.529574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.529572 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbdzk\" (UniqueName: \"kubernetes.io/projected/6e28d932-4790-4f6b-880c-338534347104-kube-api-access-zbdzk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:02.620850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.620787 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" podUID="573bf036-fa9c-437d-bb41-8c93e82cb568" containerName="payload-processing" containerID="cri-o://44955a785f067f7768fe299ca8a4626208e6dab73568b43355a178577236f20a" gracePeriod=25 Oct 30 05:40:02.667974 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:02.667928 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" podUID="3baee78b-9919-4aab-a971-7251ce5ecd29" containerName="payload-pre-processing" containerID="cri-o://6e22817b359afc91383c98a429fbdc0ec8328068e508c3a7e88820d081b19c20" gracePeriod=25 Oct 30 05:40:03.190208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.190171 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" event={"ID":"6e28d932-4790-4f6b-880c-338534347104","Type":"ContainerDied","Data":"6c2ff5a97fe1861ee5370a9054afa7021531e532d58e8869bec6d9d2bfe704fe"} Oct 30 05:40:03.190381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.190218 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c2ff5a97fe1861ee5370a9054afa7021531e532d58e8869bec6d9d2bfe704fe" Oct 30 05:40:03.190381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.190186 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp" Oct 30 05:40:03.370326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.370286 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" containerID="cri-o://b45cb13399d9a75e6266dd02367a70f6835d224d4e9f9d12368aecfd15c9a3ac" gracePeriod=45 Oct 30 05:40:03.585060 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.585025 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56577f79-3dfd-46ce-adf2-1e7ec2db9fce" path="/var/lib/kubelet/pods/56577f79-3dfd-46ce-adf2-1e7ec2db9fce/volumes" Oct 30 05:40:03.638466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.638439 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:40:03.742879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.742265 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-dshm\") pod \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " Oct 30 05:40:03.742879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.742350 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tmp-dir\") pod \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " Oct 30 05:40:03.742879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.742465 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tls-certs\" (UniqueName: \"kubernetes.io/secret/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tls-certs\") pod \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " Oct 30 05:40:03.742879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.742498 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-model-cache\") pod \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " Oct 30 05:40:03.742879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.742548 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-kube-api-access-ktthr\" (UniqueName: \"kubernetes.io/projected/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-kube-api-access-ktthr\") pod \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " Oct 30 05:40:03.742879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.742626 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-home\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-home\") pod \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\" (UID: \"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31\") " Oct 30 05:40:03.742879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.742685 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tmp-dir" pod "fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" (UID: "fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:03.742879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.742881 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-model-cache" pod "fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" (UID: "fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:03.743635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.743074 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-home" pod "fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" (UID: "fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:03.743635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.743124 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:03.743635 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.743144 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:03.745307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.745275 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-dshm" pod "fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" (UID: "fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:03.745476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.745311 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-kube-api-access-ktthr" pod "fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" (UID: "fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31"). InnerVolumeSpecName "kube-api-access-ktthr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:03.745581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.745562 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tls-certs" pod "fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" (UID: "fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:03.843746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.843708 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:03.843746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.843734 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:03.843746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.843746 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktthr\" (UniqueName: \"kubernetes.io/projected/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-kube-api-access-ktthr\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:03.843746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:03.843754 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:04.198325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:04.198281 6587 generic.go:356] "Generic (PLEG): container finished" podID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerID="b45cb13399d9a75e6266dd02367a70f6835d224d4e9f9d12368aecfd15c9a3ac" exitCode=0 Oct 30 05:40:04.198514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:04.198343 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" Oct 30 05:40:04.198514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:04.198370 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" event={"ID":"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31","Type":"ContainerDied","Data":"b45cb13399d9a75e6266dd02367a70f6835d224d4e9f9d12368aecfd15c9a3ac"} Oct 30 05:40:04.198514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:04.198414 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8" event={"ID":"fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31","Type":"ContainerDied","Data":"25f5cdf2a1d61b4ea5fdbe406920ab4f55e5d6b2390c64d2e0a5adb960afe40f"} Oct 30 05:40:04.198514 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:04.198431 6587 scope.go:122] "RemoveContainer" containerID="b45cb13399d9a75e6266dd02367a70f6835d224d4e9f9d12368aecfd15c9a3ac" Oct 30 05:40:04.209080 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:04.209062 6587 scope.go:122] "RemoveContainer" containerID="b45cb13399d9a75e6266dd02367a70f6835d224d4e9f9d12368aecfd15c9a3ac" Oct 30 05:40:04.209497 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:40:04.209383 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45cb13399d9a75e6266dd02367a70f6835d224d4e9f9d12368aecfd15c9a3ac\": container with ID starting with b45cb13399d9a75e6266dd02367a70f6835d224d4e9f9d12368aecfd15c9a3ac not found: ID does not exist" containerID="b45cb13399d9a75e6266dd02367a70f6835d224d4e9f9d12368aecfd15c9a3ac" Oct 30 05:40:04.209497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:04.209419 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45cb13399d9a75e6266dd02367a70f6835d224d4e9f9d12368aecfd15c9a3ac"} err="failed to get container status \"b45cb13399d9a75e6266dd02367a70f6835d224d4e9f9d12368aecfd15c9a3ac\": rpc error: code = NotFound desc = could not find container \"b45cb13399d9a75e6266dd02367a70f6835d224d4e9f9d12368aecfd15c9a3ac\": container with ID starting with b45cb13399d9a75e6266dd02367a70f6835d224d4e9f9d12368aecfd15c9a3ac not found: ID does not exist" Oct 30 05:40:04.254880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:04.254845 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8"] Oct 30 05:40:04.260446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:04.260416 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/simulator-w5-0b21ba-kserve-74f4b4b78b-nt7f8"] Oct 30 05:40:04.878985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:04.878944 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" podUID="573bf036-fa9c-437d-bb41-8c93e82cb568" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:40:04.879430 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:04.878958 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" podUID="3baee78b-9919-4aab-a971-7251ce5ecd29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:40:05.580876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:05.580837 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" path="/var/lib/kubelet/pods/fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31/volumes" Oct 30 05:40:09.815290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:09.815256 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd"] Oct 30 05:40:09.815792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:09.815477 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerName="istio-proxy" containerID="cri-o://1d0990593626ef7e662f82833f26945287eb71cf8aa26217fe72f1b6dd78ba65" gracePeriod=30 Oct 30 05:40:09.827250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:09.827221 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q"] Oct 30 05:40:09.827478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:09.827439 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" podUID="9c59e0f2-7aca-4982-a50f-9b28382c6e8e" containerName="istio-proxy" containerID="cri-o://4e04d7d67806e2be6ce9dcba8b6fc196f8c11e33fa874d9d6c87af421eb9e702" gracePeriod=30 Oct 30 05:40:09.839457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:09.839416 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9"] Oct 30 05:40:09.839658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:09.839622 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" podUID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" containerName="istio-proxy" containerID="cri-o://3878a63e317838cea37715cd6c8c7286cf4849d2fa2038ba086adfdae74ff79f" gracePeriod=30 Oct 30 05:40:10.554219 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.554186 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5"] Oct 30 05:40:10.554747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.554732 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" Oct 30 05:40:10.554793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.554749 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24c4b5a-53e8-4f99-ba51-69ec7b6a994a" containerName="istio-proxy" Oct 30 05:40:10.554793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.554769 6587 cpu_manager.go:414] "containerMap: removing container" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" Oct 30 05:40:10.554793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.554777 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" Oct 30 05:40:10.554896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.554798 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" Oct 30 05:40:10.554896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.554803 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7930f4-1e78-4bd5-be93-010fe7e02d20" containerName="istio-proxy" Oct 30 05:40:10.554896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.554814 6587 cpu_manager.go:414] "containerMap: removing container" podUID="6e28d932-4790-4f6b-880c-338534347104" containerName="revoke-keys" Oct 30 05:40:10.554896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.554819 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e28d932-4790-4f6b-880c-338534347104" containerName="revoke-keys" Oct 30 05:40:10.555023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.554905 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e28d932-4790-4f6b-880c-338534347104" containerName="revoke-keys" Oct 30 05:40:10.555023 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.554920 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef5d50e-c6e1-4f7e-ad51-af4edc5c1a31" containerName="main" Oct 30 05:40:10.558120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.558100 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" Oct 30 05:40:10.580727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.580694 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5"] Oct 30 05:40:10.585827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.585806 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:40:10.585964 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.585846 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:40:10.604714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.604683 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5\" (UID: \"55b3053d-c050-4d4c-bca3-22b3905f20eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" Oct 30 05:40:10.604868 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.604759 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2kf4\" (UniqueName: \"kubernetes.io/projected/55b3053d-c050-4d4c-bca3-22b3905f20eb-kube-api-access-s2kf4\") pod \"maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5\" (UID: \"55b3053d-c050-4d4c-bca3-22b3905f20eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" Oct 30 05:40:10.604868 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.604846 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5\" (UID: \"55b3053d-c050-4d4c-bca3-22b3905f20eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" Oct 30 05:40:10.607466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.607441 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:40:10.608103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.607486 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:40:10.706305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.706265 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2kf4\" (UniqueName: \"kubernetes.io/projected/55b3053d-c050-4d4c-bca3-22b3905f20eb-kube-api-access-s2kf4\") pod \"maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5\" (UID: \"55b3053d-c050-4d4c-bca3-22b3905f20eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" Oct 30 05:40:10.706500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.706312 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5\" (UID: \"55b3053d-c050-4d4c-bca3-22b3905f20eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" Oct 30 05:40:10.706500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.706477 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5\" (UID: \"55b3053d-c050-4d4c-bca3-22b3905f20eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" Oct 30 05:40:10.707097 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.707073 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5\" (UID: \"55b3053d-c050-4d4c-bca3-22b3905f20eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" Oct 30 05:40:10.708754 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.708730 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5\" (UID: \"55b3053d-c050-4d4c-bca3-22b3905f20eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" Oct 30 05:40:10.725793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.725767 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2kf4\" (UniqueName: \"kubernetes.io/projected/55b3053d-c050-4d4c-bca3-22b3905f20eb-kube-api-access-s2kf4\") pod \"maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5\" (UID: \"55b3053d-c050-4d4c-bca3-22b3905f20eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" Oct 30 05:40:10.871223 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:10.871139 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" Oct 30 05:40:11.171041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:11.171009 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5"] Oct 30 05:40:11.172674 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:40:11.172636 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b3053d_c050_4d4c_bca3_22b3905f20eb.slice/crio-39a93355a46b7eb64815ce9b9fdc7100bff8732da142aaa670a1674cde8ea996 WatchSource:0}: Error finding container 39a93355a46b7eb64815ce9b9fdc7100bff8732da142aaa670a1674cde8ea996: Status 404 returned error can't find the container with id 39a93355a46b7eb64815ce9b9fdc7100bff8732da142aaa670a1674cde8ea996 Oct 30 05:40:11.252438 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:11.252406 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" event={"ID":"55b3053d-c050-4d4c-bca3-22b3905f20eb","Type":"ContainerStarted","Data":"39a93355a46b7eb64815ce9b9fdc7100bff8732da142aaa670a1674cde8ea996"} Oct 30 05:40:12.261426 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:12.261389 6587 generic.go:356] "Generic (PLEG): container finished" podID="55b3053d-c050-4d4c-bca3-22b3905f20eb" containerID="f9ac45b56c69f4d9a7b3770da2df80fd52f00bcab3892d0b4d23d91bf97c42f8" exitCode=0 Oct 30 05:40:12.261833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:12.261461 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" event={"ID":"55b3053d-c050-4d4c-bca3-22b3905f20eb","Type":"ContainerDied","Data":"f9ac45b56c69f4d9a7b3770da2df80fd52f00bcab3892d0b4d23d91bf97c42f8"} Oct 30 05:40:13.249859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:13.249824 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:40:13.250166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:13.249887 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" podUID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:40:13.410922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:13.410899 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" Oct 30 05:40:13.437774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:13.437745 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-xks-ca\") pod \"55b3053d-c050-4d4c-bca3-22b3905f20eb\" (UID: \"55b3053d-c050-4d4c-bca3-22b3905f20eb\") " Oct 30 05:40:13.437919 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:13.437856 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-openshift-ca\") pod \"55b3053d-c050-4d4c-bca3-22b3905f20eb\" (UID: \"55b3053d-c050-4d4c-bca3-22b3905f20eb\") " Oct 30 05:40:13.438003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:13.437945 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b3053d-c050-4d4c-bca3-22b3905f20eb-kube-api-access-s2kf4\" (UniqueName: \"kubernetes.io/projected/55b3053d-c050-4d4c-bca3-22b3905f20eb-kube-api-access-s2kf4\") pod \"55b3053d-c050-4d4c-bca3-22b3905f20eb\" (UID: \"55b3053d-c050-4d4c-bca3-22b3905f20eb\") " Oct 30 05:40:13.438258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:13.438228 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-openshift-ca" pod "55b3053d-c050-4d4c-bca3-22b3905f20eb" (UID: "55b3053d-c050-4d4c-bca3-22b3905f20eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:13.438442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:13.438416 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:13.439971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:13.439944 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-xks-ca" pod "55b3053d-c050-4d4c-bca3-22b3905f20eb" (UID: "55b3053d-c050-4d4c-bca3-22b3905f20eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:13.440083 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:13.440030 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b3053d-c050-4d4c-bca3-22b3905f20eb-kube-api-access-s2kf4" pod "55b3053d-c050-4d4c-bca3-22b3905f20eb" (UID: "55b3053d-c050-4d4c-bca3-22b3905f20eb"). InnerVolumeSpecName "kube-api-access-s2kf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:13.540902 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:13.540816 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b3053d-c050-4d4c-bca3-22b3905f20eb-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:13.540902 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:13.540857 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2kf4\" (UniqueName: \"kubernetes.io/projected/55b3053d-c050-4d4c-bca3-22b3905f20eb-kube-api-access-s2kf4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:14.034205 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:14.034172 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9"] Oct 30 05:40:14.037370 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:14.037326 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b21ba-4qzs9"] Oct 30 05:40:14.278195 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:14.278166 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" Oct 30 05:40:14.278391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:14.278202 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5" event={"ID":"55b3053d-c050-4d4c-bca3-22b3905f20eb","Type":"ContainerDied","Data":"39a93355a46b7eb64815ce9b9fdc7100bff8732da142aaa670a1674cde8ea996"} Oct 30 05:40:14.278391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:14.278246 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39a93355a46b7eb64815ce9b9fdc7100bff8732da142aaa670a1674cde8ea996" Oct 30 05:40:14.654174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:14.654140 6587 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:40:14.654581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:14.654220 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" podUID="9c59e0f2-7aca-4982-a50f-9b28382c6e8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:40:14.879044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:14.879014 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" podUID="3baee78b-9919-4aab-a971-7251ce5ecd29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:40:14.879166 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:14.879027 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" podUID="573bf036-fa9c-437d-bb41-8c93e82cb568" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:40:15.111608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.111580 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:40:15.158032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.157981 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-token\" (UniqueName: \"kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-token\") pod \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " Oct 30 05:40:15.158168 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.158070 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-data\") pod \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " Oct 30 05:40:15.158284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.158264 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-socket\") pod \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " Oct 30 05:40:15.159433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.158485 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-kube-api-access-l48tn\" (UniqueName: \"kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-kube-api-access-l48tn\") pod \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " Oct 30 05:40:15.159433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.158528 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-podinfo\") pod \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " Oct 30 05:40:15.159433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.158588 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-envoy\") pod \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " Oct 30 05:40:15.159433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.158631 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-credential-socket\") pod \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " Oct 30 05:40:15.159433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.158686 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istiod-ca-cert\") pod \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " Oct 30 05:40:15.159433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.158718 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-certs\") pod \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\" (UID: \"9c59e0f2-7aca-4982-a50f-9b28382c6e8e\") " Oct 30 05:40:15.159967 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.159936 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-certs" pod "9c59e0f2-7aca-4982-a50f-9b28382c6e8e" (UID: "9c59e0f2-7aca-4982-a50f-9b28382c6e8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.166141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.161132 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-credential-socket" pod "9c59e0f2-7aca-4982-a50f-9b28382c6e8e" (UID: "9c59e0f2-7aca-4982-a50f-9b28382c6e8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.166141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.161308 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-socket" pod "9c59e0f2-7aca-4982-a50f-9b28382c6e8e" (UID: "9c59e0f2-7aca-4982-a50f-9b28382c6e8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.166141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.161510 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istiod-ca-cert" pod "9c59e0f2-7aca-4982-a50f-9b28382c6e8e" (UID: "9c59e0f2-7aca-4982-a50f-9b28382c6e8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:15.166141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.161866 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-data" pod "9c59e0f2-7aca-4982-a50f-9b28382c6e8e" (UID: "9c59e0f2-7aca-4982-a50f-9b28382c6e8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.166741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.166643 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-kube-api-access-l48tn" pod "9c59e0f2-7aca-4982-a50f-9b28382c6e8e" (UID: "9c59e0f2-7aca-4982-a50f-9b28382c6e8e"). InnerVolumeSpecName "kube-api-access-l48tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:15.166741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.166699 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-token" pod "9c59e0f2-7aca-4982-a50f-9b28382c6e8e" (UID: "9c59e0f2-7aca-4982-a50f-9b28382c6e8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:15.166741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.166711 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-envoy" pod "9c59e0f2-7aca-4982-a50f-9b28382c6e8e" (UID: "9c59e0f2-7aca-4982-a50f-9b28382c6e8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.166962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.166676 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-podinfo" pod "9c59e0f2-7aca-4982-a50f-9b28382c6e8e" (UID: "9c59e0f2-7aca-4982-a50f-9b28382c6e8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:40:15.175124 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.175091 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:40:15.177366 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.177302 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:40:15.259806 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.259762 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-kube-api-access-2c2xb\" (UniqueName: \"kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-kube-api-access-2c2xb\") pod \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " Oct 30 05:40:15.260137 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260104 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-podinfo\") pod \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " Oct 30 05:40:15.260282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260165 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-credential-socket\") pod \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " Oct 30 05:40:15.260282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260198 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-kube-api-access-xjw24\" (UniqueName: \"kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-kube-api-access-xjw24\") pod \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " Oct 30 05:40:15.260282 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260259 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-credential-socket\") pod \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " Oct 30 05:40:15.260495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260293 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-data\") pod \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " Oct 30 05:40:15.260495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260344 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istiod-ca-cert\") pod \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " Oct 30 05:40:15.260495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260418 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istiod-ca-cert\") pod \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " Oct 30 05:40:15.260495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260448 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-envoy\") pod \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " Oct 30 05:40:15.260495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260493 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-socket\") pod \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " Oct 30 05:40:15.260782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260534 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-data\") pod \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " Oct 30 05:40:15.260782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260566 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-envoy\") pod \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " Oct 30 05:40:15.260782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260607 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-certs\") pod \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " Oct 30 05:40:15.260782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260663 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-token\" (UniqueName: \"kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-token\") pod \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\" (UID: \"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce\") " Oct 30 05:40:15.260782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260693 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-podinfo\") pod \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " Oct 30 05:40:15.260782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260740 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-certs\") pod \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " Oct 30 05:40:15.260782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260775 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-token\" (UniqueName: \"kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-token\") pod \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " Oct 30 05:40:15.261188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.260806 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-socket\") pod \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\" (UID: \"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64\") " Oct 30 05:40:15.261264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.261230 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.261264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.261261 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.261420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.261275 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.261420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.261288 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.261420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.261303 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.261420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.261317 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l48tn\" (UniqueName: \"kubernetes.io/projected/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-kube-api-access-l48tn\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.261420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.261332 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.261420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.261345 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.261420 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.261376 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c59e0f2-7aca-4982-a50f-9b28382c6e8e-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.261796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.261620 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-socket" pod "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" (UID: "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.261875 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.261845 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-credential-socket" pod "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" (UID: "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.262292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.262247 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-data" pod "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" (UID: "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.262737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.262681 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-credential-socket" pod "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" (UID: "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.263030 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.263008 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-data" pod "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" (UID: "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.264541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.263607 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istiod-ca-cert" pod "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" (UID: "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:15.264541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.263873 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-socket" pod "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" (UID: "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.264541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.263987 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istiod-ca-cert" pod "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" (UID: "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:15.264541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.264135 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-certs" pod "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" (UID: "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.264541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.264333 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-kube-api-access-xjw24" pod "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" (UID: "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64"). InnerVolumeSpecName "kube-api-access-xjw24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:15.264541 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.264510 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-certs" pod "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" (UID: "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.266789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.266765 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-envoy" pod "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" (UID: "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.271975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.271926 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-token" pod "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" (UID: "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:15.272222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.272201 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-kube-api-access-2c2xb" pod "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" (UID: "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce"). InnerVolumeSpecName "kube-api-access-2c2xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:15.273131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.273108 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-podinfo" pod "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" (UID: "ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:40:15.273262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.273248 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-podinfo" pod "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" (UID: "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:40:15.273639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.273613 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-token" pod "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" (UID: "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:15.274510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.274482 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-envoy" pod "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" (UID: "d02ff6cc-a5bb-40de-a4e3-9cd3058abc64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:15.290133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.290098 6587 generic.go:356] "Generic (PLEG): container finished" podID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerID="1d0990593626ef7e662f82833f26945287eb71cf8aa26217fe72f1b6dd78ba65" exitCode=0 Oct 30 05:40:15.290276 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.290195 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" event={"ID":"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce","Type":"ContainerDied","Data":"1d0990593626ef7e662f82833f26945287eb71cf8aa26217fe72f1b6dd78ba65"} Oct 30 05:40:15.290345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.290278 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" event={"ID":"ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce","Type":"ContainerDied","Data":"bf21dc9579e27b1e916fe70d05d94aa6194118d050bd18ee389085f1dbff67b1"} Oct 30 05:40:15.290345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.290308 6587 scope.go:122] "RemoveContainer" containerID="1d0990593626ef7e662f82833f26945287eb71cf8aa26217fe72f1b6dd78ba65" Oct 30 05:40:15.290537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.290516 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd" Oct 30 05:40:15.297577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.297555 6587 generic.go:356] "Generic (PLEG): container finished" podID="9c59e0f2-7aca-4982-a50f-9b28382c6e8e" containerID="4e04d7d67806e2be6ce9dcba8b6fc196f8c11e33fa874d9d6c87af421eb9e702" exitCode=0 Oct 30 05:40:15.297721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.297632 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" Oct 30 05:40:15.297721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.297690 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" event={"ID":"9c59e0f2-7aca-4982-a50f-9b28382c6e8e","Type":"ContainerDied","Data":"4e04d7d67806e2be6ce9dcba8b6fc196f8c11e33fa874d9d6c87af421eb9e702"} Oct 30 05:40:15.297858 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.297722 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q" event={"ID":"9c59e0f2-7aca-4982-a50f-9b28382c6e8e","Type":"ContainerDied","Data":"cbb15c5cfea48749fc629941a4dde9d5670b1599160ac4c5c00656de2ecf8bcf"} Oct 30 05:40:15.299488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.299457 6587 generic.go:356] "Generic (PLEG): container finished" podID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" containerID="3878a63e317838cea37715cd6c8c7286cf4849d2fa2038ba086adfdae74ff79f" exitCode=0 Oct 30 05:40:15.299603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.299568 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" Oct 30 05:40:15.299686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.299667 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" event={"ID":"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64","Type":"ContainerDied","Data":"3878a63e317838cea37715cd6c8c7286cf4849d2fa2038ba086adfdae74ff79f"} Oct 30 05:40:15.299745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.299713 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9" event={"ID":"d02ff6cc-a5bb-40de-a4e3-9cd3058abc64","Type":"ContainerDied","Data":"eaaf2163e542bac75a91d78c68cb6af8c71ee7455016a5a091f84a6e86c5a252"} Oct 30 05:40:15.310811 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.310784 6587 scope.go:122] "RemoveContainer" containerID="74a7eb4b2804ef7810b12a38693e254f7ba3c99d867fec953b2e2ac263e973ca" Oct 30 05:40:15.326988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.326964 6587 scope.go:122] "RemoveContainer" containerID="1d0990593626ef7e662f82833f26945287eb71cf8aa26217fe72f1b6dd78ba65" Oct 30 05:40:15.328028 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:40:15.328006 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0990593626ef7e662f82833f26945287eb71cf8aa26217fe72f1b6dd78ba65\": container with ID starting with 1d0990593626ef7e662f82833f26945287eb71cf8aa26217fe72f1b6dd78ba65 not found: ID does not exist" containerID="1d0990593626ef7e662f82833f26945287eb71cf8aa26217fe72f1b6dd78ba65" Oct 30 05:40:15.328096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.328037 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0990593626ef7e662f82833f26945287eb71cf8aa26217fe72f1b6dd78ba65"} err="failed to get container status \"1d0990593626ef7e662f82833f26945287eb71cf8aa26217fe72f1b6dd78ba65\": rpc error: code = NotFound desc = could not find container \"1d0990593626ef7e662f82833f26945287eb71cf8aa26217fe72f1b6dd78ba65\": container with ID starting with 1d0990593626ef7e662f82833f26945287eb71cf8aa26217fe72f1b6dd78ba65 not found: ID does not exist" Oct 30 05:40:15.328096 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.328054 6587 scope.go:122] "RemoveContainer" containerID="74a7eb4b2804ef7810b12a38693e254f7ba3c99d867fec953b2e2ac263e973ca" Oct 30 05:40:15.328306 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:40:15.328289 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a7eb4b2804ef7810b12a38693e254f7ba3c99d867fec953b2e2ac263e973ca\": container with ID starting with 74a7eb4b2804ef7810b12a38693e254f7ba3c99d867fec953b2e2ac263e973ca not found: ID does not exist" containerID="74a7eb4b2804ef7810b12a38693e254f7ba3c99d867fec953b2e2ac263e973ca" Oct 30 05:40:15.328372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.328313 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a7eb4b2804ef7810b12a38693e254f7ba3c99d867fec953b2e2ac263e973ca"} err="failed to get container status \"74a7eb4b2804ef7810b12a38693e254f7ba3c99d867fec953b2e2ac263e973ca\": rpc error: code = NotFound desc = could not find container \"74a7eb4b2804ef7810b12a38693e254f7ba3c99d867fec953b2e2ac263e973ca\": container with ID starting with 74a7eb4b2804ef7810b12a38693e254f7ba3c99d867fec953b2e2ac263e973ca not found: ID does not exist" Oct 30 05:40:15.328372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.328329 6587 scope.go:122] "RemoveContainer" containerID="4e04d7d67806e2be6ce9dcba8b6fc196f8c11e33fa874d9d6c87af421eb9e702" Oct 30 05:40:15.341169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.341150 6587 scope.go:122] "RemoveContainer" containerID="4e04d7d67806e2be6ce9dcba8b6fc196f8c11e33fa874d9d6c87af421eb9e702" Oct 30 05:40:15.341506 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:40:15.341486 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e04d7d67806e2be6ce9dcba8b6fc196f8c11e33fa874d9d6c87af421eb9e702\": container with ID starting with 4e04d7d67806e2be6ce9dcba8b6fc196f8c11e33fa874d9d6c87af421eb9e702 not found: ID does not exist" containerID="4e04d7d67806e2be6ce9dcba8b6fc196f8c11e33fa874d9d6c87af421eb9e702" Oct 30 05:40:15.341593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.341516 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e04d7d67806e2be6ce9dcba8b6fc196f8c11e33fa874d9d6c87af421eb9e702"} err="failed to get container status \"4e04d7d67806e2be6ce9dcba8b6fc196f8c11e33fa874d9d6c87af421eb9e702\": rpc error: code = NotFound desc = could not find container \"4e04d7d67806e2be6ce9dcba8b6fc196f8c11e33fa874d9d6c87af421eb9e702\": container with ID starting with 4e04d7d67806e2be6ce9dcba8b6fc196f8c11e33fa874d9d6c87af421eb9e702 not found: ID does not exist" Oct 30 05:40:15.341593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.341539 6587 scope.go:122] "RemoveContainer" containerID="3878a63e317838cea37715cd6c8c7286cf4849d2fa2038ba086adfdae74ff79f" Oct 30 05:40:15.354139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.354109 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd"] Oct 30 05:40:15.354502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.354483 6587 scope.go:122] "RemoveContainer" containerID="3878a63e317838cea37715cd6c8c7286cf4849d2fa2038ba086adfdae74ff79f" Oct 30 05:40:15.354920 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:40:15.354897 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3878a63e317838cea37715cd6c8c7286cf4849d2fa2038ba086adfdae74ff79f\": container with ID starting with 3878a63e317838cea37715cd6c8c7286cf4849d2fa2038ba086adfdae74ff79f not found: ID does not exist" containerID="3878a63e317838cea37715cd6c8c7286cf4849d2fa2038ba086adfdae74ff79f" Oct 30 05:40:15.355003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.354928 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3878a63e317838cea37715cd6c8c7286cf4849d2fa2038ba086adfdae74ff79f"} err="failed to get container status \"3878a63e317838cea37715cd6c8c7286cf4849d2fa2038ba086adfdae74ff79f\": rpc error: code = NotFound desc = could not find container \"3878a63e317838cea37715cd6c8c7286cf4849d2fa2038ba086adfdae74ff79f\": container with ID starting with 3878a63e317838cea37715cd6c8c7286cf4849d2fa2038ba086adfdae74ff79f not found: ID does not exist" Oct 30 05:40:15.362562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362535 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362562 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362564 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362576 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362585 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362595 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362603 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c2xb\" (UniqueName: \"kubernetes.io/projected/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-kube-api-access-2c2xb\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362613 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362627 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362636 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjw24\" (UniqueName: \"kubernetes.io/projected/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-kube-api-access-xjw24\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362646 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362654 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362662 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362669 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362677 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362686 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362693 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362702 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.362734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.362709 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:15.374521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.374492 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst"] Oct 30 05:40:15.377555 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.377528 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd"] Oct 30 05:40:15.491411 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.491313 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q"] Oct 30 05:40:15.495544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.495517 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q"] Oct 30 05:40:15.528170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.528135 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9"] Oct 30 05:40:15.534865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.534837 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9"] Oct 30 05:40:15.581392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.581345 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6904fb31-6918-4dd1-bb40-d683a6bde3a3" path="/var/lib/kubelet/pods/6904fb31-6918-4dd1-bb40-d683a6bde3a3/volumes" Oct 30 05:40:15.581835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.581821 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c59e0f2-7aca-4982-a50f-9b28382c6e8e" path="/var/lib/kubelet/pods/9c59e0f2-7aca-4982-a50f-9b28382c6e8e/volumes" Oct 30 05:40:15.582236 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.582223 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" path="/var/lib/kubelet/pods/d02ff6cc-a5bb-40de-a4e3-9cd3058abc64/volumes" Oct 30 05:40:15.582951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:15.582935 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" path="/var/lib/kubelet/pods/ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce/volumes" Oct 30 05:40:21.045073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.044995 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr"] Oct 30 05:40:21.077108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.077074 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn"] Oct 30 05:40:21.582102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582063 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26"] Oct 30 05:40:21.582732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582712 6587 cpu_manager.go:414] "containerMap: removing container" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerName="istio-proxy" Oct 30 05:40:21.582804 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582734 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerName="istio-proxy" Oct 30 05:40:21.582804 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582764 6587 cpu_manager.go:414] "containerMap: removing container" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerName="istio-proxy" Oct 30 05:40:21.582804 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582773 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerName="istio-proxy" Oct 30 05:40:21.582804 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582788 6587 cpu_manager.go:414] "containerMap: removing container" podUID="55b3053d-c050-4d4c-bca3-22b3905f20eb" containerName="revoke-keys" Oct 30 05:40:21.582804 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582795 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b3053d-c050-4d4c-bca3-22b3905f20eb" containerName="revoke-keys" Oct 30 05:40:21.582986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582815 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" containerName="istio-proxy" Oct 30 05:40:21.582986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582822 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" containerName="istio-proxy" Oct 30 05:40:21.582986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582855 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9c59e0f2-7aca-4982-a50f-9b28382c6e8e" containerName="istio-proxy" Oct 30 05:40:21.582986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582862 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c59e0f2-7aca-4982-a50f-9b28382c6e8e" containerName="istio-proxy" Oct 30 05:40:21.582986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582964 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerName="istio-proxy" Oct 30 05:40:21.582986 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582979 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b3053d-c050-4d4c-bca3-22b3905f20eb" containerName="revoke-keys" Oct 30 05:40:21.583190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.582993 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02ff6cc-a5bb-40de-a4e3-9cd3058abc64" containerName="istio-proxy" Oct 30 05:40:21.583190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.583005 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c59e0f2-7aca-4982-a50f-9b28382c6e8e" containerName="istio-proxy" Oct 30 05:40:21.588709 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.588689 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" Oct 30 05:40:21.608545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.608515 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:40:21.608715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.608687 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26"] Oct 30 05:40:21.608845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.608757 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:40:21.617027 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.616992 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:40:21.629129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.629100 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ppd\" (UniqueName: \"kubernetes.io/projected/d44fd769-67e4-411a-8a2e-ee50329ec6a0-kube-api-access-q8ppd\") pod \"maas-api-revoke-sub-a2bc3343-9lz26\" (UID: \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" Oct 30 05:40:21.629309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.629168 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2bc3343-9lz26\" (UID: \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" Oct 30 05:40:21.629421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.629345 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2bc3343-9lz26\" (UID: \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" Oct 30 05:40:21.713783 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.713752 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:40:21.730832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.730796 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2bc3343-9lz26\" (UID: \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" Oct 30 05:40:21.730832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.730837 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ppd\" (UniqueName: \"kubernetes.io/projected/d44fd769-67e4-411a-8a2e-ee50329ec6a0-kube-api-access-q8ppd\") pod \"maas-api-revoke-sub-a2bc3343-9lz26\" (UID: \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" Oct 30 05:40:21.731044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.730884 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2bc3343-9lz26\" (UID: \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" Oct 30 05:40:21.731625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.731600 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2bc3343-9lz26\" (UID: \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" Oct 30 05:40:21.733226 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.733206 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2bc3343-9lz26\" (UID: \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" Oct 30 05:40:21.750264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.750240 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ppd\" (UniqueName: \"kubernetes.io/projected/d44fd769-67e4-411a-8a2e-ee50329ec6a0-kube-api-access-q8ppd\") pod \"maas-api-revoke-sub-a2bc3343-9lz26\" (UID: \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" Oct 30 05:40:21.901890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:21.901807 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" Oct 30 05:40:22.218587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:22.218557 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26"] Oct 30 05:40:22.221224 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:40:22.221177 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44fd769_67e4_411a_8a2e_ee50329ec6a0.slice/crio-36514076ffb26afc956b5b14837f86b494e24b5964095c8ae3398b7b352c0343 WatchSource:0}: Error finding container 36514076ffb26afc956b5b14837f86b494e24b5964095c8ae3398b7b352c0343: Status 404 returned error can't find the container with id 36514076ffb26afc956b5b14837f86b494e24b5964095c8ae3398b7b352c0343 Oct 30 05:40:22.354815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:22.354780 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" event={"ID":"d44fd769-67e4-411a-8a2e-ee50329ec6a0","Type":"ContainerStarted","Data":"36514076ffb26afc956b5b14837f86b494e24b5964095c8ae3398b7b352c0343"} Oct 30 05:40:22.607229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:22.607194 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f"] Oct 30 05:40:22.607471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:22.607411 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" podUID="9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" containerName="istio-proxy" containerID="cri-o://2f0504d501789f9135992f3f10898db5782c6dce766da487ad5f745b04eababd" gracePeriod=30 Oct 30 05:40:22.947591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:22.947569 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" Oct 30 05:40:22.969257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:22.969228 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" Oct 30 05:40:23.044348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.044258 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/573bf036-fa9c-437d-bb41-8c93e82cb568-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/573bf036-fa9c-437d-bb41-8c93e82cb568-plugins-config-volume\") pod \"573bf036-fa9c-437d-bb41-8c93e82cb568\" (UID: \"573bf036-fa9c-437d-bb41-8c93e82cb568\") " Oct 30 05:40:23.044348 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.044304 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/573bf036-fa9c-437d-bb41-8c93e82cb568-kube-api-access-cpgdg\" (UniqueName: \"kubernetes.io/projected/573bf036-fa9c-437d-bb41-8c93e82cb568-kube-api-access-cpgdg\") pod \"573bf036-fa9c-437d-bb41-8c93e82cb568\" (UID: \"573bf036-fa9c-437d-bb41-8c93e82cb568\") " Oct 30 05:40:23.044613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.044400 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3baee78b-9919-4aab-a971-7251ce5ecd29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3baee78b-9919-4aab-a971-7251ce5ecd29-plugins-config-volume\") pod \"3baee78b-9919-4aab-a971-7251ce5ecd29\" (UID: \"3baee78b-9919-4aab-a971-7251ce5ecd29\") " Oct 30 05:40:23.044613 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.044434 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3baee78b-9919-4aab-a971-7251ce5ecd29-kube-api-access-z284l\" (UniqueName: \"kubernetes.io/projected/3baee78b-9919-4aab-a971-7251ce5ecd29-kube-api-access-z284l\") pod \"3baee78b-9919-4aab-a971-7251ce5ecd29\" (UID: \"3baee78b-9919-4aab-a971-7251ce5ecd29\") " Oct 30 05:40:23.044847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.044818 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573bf036-fa9c-437d-bb41-8c93e82cb568-plugins-config-volume" pod "573bf036-fa9c-437d-bb41-8c93e82cb568" (UID: "573bf036-fa9c-437d-bb41-8c93e82cb568"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:23.044918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.044847 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3baee78b-9919-4aab-a971-7251ce5ecd29-plugins-config-volume" pod "3baee78b-9919-4aab-a971-7251ce5ecd29" (UID: "3baee78b-9919-4aab-a971-7251ce5ecd29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:23.046618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.046592 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573bf036-fa9c-437d-bb41-8c93e82cb568-kube-api-access-cpgdg" pod "573bf036-fa9c-437d-bb41-8c93e82cb568" (UID: "573bf036-fa9c-437d-bb41-8c93e82cb568"). InnerVolumeSpecName "kube-api-access-cpgdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:23.047151 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.047124 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3baee78b-9919-4aab-a971-7251ce5ecd29-kube-api-access-z284l" pod "3baee78b-9919-4aab-a971-7251ce5ecd29" (UID: "3baee78b-9919-4aab-a971-7251ce5ecd29"). InnerVolumeSpecName "kube-api-access-z284l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:23.145625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.145597 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/573bf036-fa9c-437d-bb41-8c93e82cb568-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:23.145625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.145625 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpgdg\" (UniqueName: \"kubernetes.io/projected/573bf036-fa9c-437d-bb41-8c93e82cb568-kube-api-access-cpgdg\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:23.145819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.145636 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3baee78b-9919-4aab-a971-7251ce5ecd29-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:23.145819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.145646 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z284l\" (UniqueName: \"kubernetes.io/projected/3baee78b-9919-4aab-a971-7251ce5ecd29-kube-api-access-z284l\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:23.364141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.364107 6587 generic.go:356] "Generic (PLEG): container finished" podID="573bf036-fa9c-437d-bb41-8c93e82cb568" containerID="44955a785f067f7768fe299ca8a4626208e6dab73568b43355a178577236f20a" exitCode=0 Oct 30 05:40:23.364661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.364151 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" Oct 30 05:40:23.364661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.364180 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" event={"ID":"573bf036-fa9c-437d-bb41-8c93e82cb568","Type":"ContainerDied","Data":"44955a785f067f7768fe299ca8a4626208e6dab73568b43355a178577236f20a"} Oct 30 05:40:23.364661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.364233 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h" event={"ID":"573bf036-fa9c-437d-bb41-8c93e82cb568","Type":"ContainerDied","Data":"0a729eb7a00b91d957b31e55066295d00a430e01d8242c42a00f264a1ec2eff3"} Oct 30 05:40:23.364661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.364259 6587 scope.go:122] "RemoveContainer" containerID="44955a785f067f7768fe299ca8a4626208e6dab73568b43355a178577236f20a" Oct 30 05:40:23.366338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.366312 6587 generic.go:356] "Generic (PLEG): container finished" podID="d44fd769-67e4-411a-8a2e-ee50329ec6a0" containerID="25f11695e8d936f1b098da0ab27888853c0f9c3558c060bd7c9c5e37be74e3f4" exitCode=0 Oct 30 05:40:23.366500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.366396 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" event={"ID":"d44fd769-67e4-411a-8a2e-ee50329ec6a0","Type":"ContainerDied","Data":"25f11695e8d936f1b098da0ab27888853c0f9c3558c060bd7c9c5e37be74e3f4"} Oct 30 05:40:23.367957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.367936 6587 generic.go:356] "Generic (PLEG): container finished" podID="3baee78b-9919-4aab-a971-7251ce5ecd29" containerID="6e22817b359afc91383c98a429fbdc0ec8328068e508c3a7e88820d081b19c20" exitCode=0 Oct 30 05:40:23.368102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.367958 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" event={"ID":"3baee78b-9919-4aab-a971-7251ce5ecd29","Type":"ContainerDied","Data":"6e22817b359afc91383c98a429fbdc0ec8328068e508c3a7e88820d081b19c20"} Oct 30 05:40:23.368102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.367987 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" Oct 30 05:40:23.368102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.367999 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb" event={"ID":"3baee78b-9919-4aab-a971-7251ce5ecd29","Type":"ContainerDied","Data":"ac10d6036a68c667802e76c7643dc5842b2550a2aa2509ddbd591b6288351bc3"} Oct 30 05:40:23.376518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.376501 6587 scope.go:122] "RemoveContainer" containerID="44955a785f067f7768fe299ca8a4626208e6dab73568b43355a178577236f20a" Oct 30 05:40:23.376791 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:40:23.376765 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44955a785f067f7768fe299ca8a4626208e6dab73568b43355a178577236f20a\": container with ID starting with 44955a785f067f7768fe299ca8a4626208e6dab73568b43355a178577236f20a not found: ID does not exist" containerID="44955a785f067f7768fe299ca8a4626208e6dab73568b43355a178577236f20a" Oct 30 05:40:23.376912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.376796 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44955a785f067f7768fe299ca8a4626208e6dab73568b43355a178577236f20a"} err="failed to get container status \"44955a785f067f7768fe299ca8a4626208e6dab73568b43355a178577236f20a\": rpc error: code = NotFound desc = could not find container \"44955a785f067f7768fe299ca8a4626208e6dab73568b43355a178577236f20a\": container with ID starting with 44955a785f067f7768fe299ca8a4626208e6dab73568b43355a178577236f20a not found: ID does not exist" Oct 30 05:40:23.376912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.376810 6587 scope.go:122] "RemoveContainer" containerID="6e22817b359afc91383c98a429fbdc0ec8328068e508c3a7e88820d081b19c20" Oct 30 05:40:23.386791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.386772 6587 scope.go:122] "RemoveContainer" containerID="6e22817b359afc91383c98a429fbdc0ec8328068e508c3a7e88820d081b19c20" Oct 30 05:40:23.387058 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:40:23.387034 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e22817b359afc91383c98a429fbdc0ec8328068e508c3a7e88820d081b19c20\": container with ID starting with 6e22817b359afc91383c98a429fbdc0ec8328068e508c3a7e88820d081b19c20 not found: ID does not exist" containerID="6e22817b359afc91383c98a429fbdc0ec8328068e508c3a7e88820d081b19c20" Oct 30 05:40:23.387107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.387070 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e22817b359afc91383c98a429fbdc0ec8328068e508c3a7e88820d081b19c20"} err="failed to get container status \"6e22817b359afc91383c98a429fbdc0ec8328068e508c3a7e88820d081b19c20\": rpc error: code = NotFound desc = could not find container \"6e22817b359afc91383c98a429fbdc0ec8328068e508c3a7e88820d081b19c20\": container with ID starting with 6e22817b359afc91383c98a429fbdc0ec8328068e508c3a7e88820d081b19c20 not found: ID does not exist" Oct 30 05:40:23.460261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.460233 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h"] Oct 30 05:40:23.466491 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.466464 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h"] Oct 30 05:40:23.500509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.500481 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb"] Oct 30 05:40:23.504505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.504480 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb"] Oct 30 05:40:23.580070 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.580030 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3baee78b-9919-4aab-a971-7251ce5ecd29" path="/var/lib/kubelet/pods/3baee78b-9919-4aab-a971-7251ce5ecd29/volumes" Oct 30 05:40:23.580525 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:23.580503 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="573bf036-fa9c-437d-bb41-8c93e82cb568" path="/var/lib/kubelet/pods/573bf036-fa9c-437d-bb41-8c93e82cb568/volumes" Oct 30 05:40:24.524295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:24.524253 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" Oct 30 05:40:24.662229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:24.662134 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-openshift-ca\") pod \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\" (UID: \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\") " Oct 30 05:40:24.662229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:24.662202 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44fd769-67e4-411a-8a2e-ee50329ec6a0-kube-api-access-q8ppd\" (UniqueName: \"kubernetes.io/projected/d44fd769-67e4-411a-8a2e-ee50329ec6a0-kube-api-access-q8ppd\") pod \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\" (UID: \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\") " Oct 30 05:40:24.662512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:24.662264 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-xks-ca\") pod \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\" (UID: \"d44fd769-67e4-411a-8a2e-ee50329ec6a0\") " Oct 30 05:40:24.662626 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:24.662598 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-openshift-ca" pod "d44fd769-67e4-411a-8a2e-ee50329ec6a0" (UID: "d44fd769-67e4-411a-8a2e-ee50329ec6a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:24.664476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:24.664446 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-xks-ca" pod "d44fd769-67e4-411a-8a2e-ee50329ec6a0" (UID: "d44fd769-67e4-411a-8a2e-ee50329ec6a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:24.664624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:24.664564 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44fd769-67e4-411a-8a2e-ee50329ec6a0-kube-api-access-q8ppd" pod "d44fd769-67e4-411a-8a2e-ee50329ec6a0" (UID: "d44fd769-67e4-411a-8a2e-ee50329ec6a0"). InnerVolumeSpecName "kube-api-access-q8ppd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:24.763305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:24.763272 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:24.763305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:24.763301 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44fd769-67e4-411a-8a2e-ee50329ec6a0-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:24.763305 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:24.763312 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8ppd\" (UniqueName: \"kubernetes.io/projected/d44fd769-67e4-411a-8a2e-ee50329ec6a0-kube-api-access-q8ppd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:25.385095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:25.385055 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" event={"ID":"d44fd769-67e4-411a-8a2e-ee50329ec6a0","Type":"ContainerDied","Data":"36514076ffb26afc956b5b14837f86b494e24b5964095c8ae3398b7b352c0343"} Oct 30 05:40:25.385284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:25.385114 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36514076ffb26afc956b5b14837f86b494e24b5964095c8ae3398b7b352c0343" Oct 30 05:40:25.385284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:25.385067 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26" Oct 30 05:40:25.985889 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:40:25.985855 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:40:26.046263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.046224 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" podUID="06b19257-a9ac-42dc-a991-ce01ca2d6414" containerName="payload-processing" containerID="cri-o://cce08864701e4f421e69d4ff6f0eba5ea38aa4a180c8d3bbe11f41b9daa854a0" gracePeriod=25 Oct 30 05:40:26.078218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.078177 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" podUID="263665e1-3eeb-43a5-adb8-82a0d9269a0b" containerName="payload-pre-processing" containerID="cri-o://a41cac8a4f65730abbc6f8933a631e2eaee9e3d7a691cc9b8f9aa8f609201151" gracePeriod=25 Oct 30 05:40:26.309744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.309714 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" Oct 30 05:40:26.381238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.381204 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06b19257-a9ac-42dc-a991-ce01ca2d6414-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b19257-a9ac-42dc-a991-ce01ca2d6414-plugins-config-volume\") pod \"06b19257-a9ac-42dc-a991-ce01ca2d6414\" (UID: \"06b19257-a9ac-42dc-a991-ce01ca2d6414\") " Oct 30 05:40:26.381471 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.381249 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b19257-a9ac-42dc-a991-ce01ca2d6414-kube-api-access-scjrh\" (UniqueName: \"kubernetes.io/projected/06b19257-a9ac-42dc-a991-ce01ca2d6414-kube-api-access-scjrh\") pod \"06b19257-a9ac-42dc-a991-ce01ca2d6414\" (UID: \"06b19257-a9ac-42dc-a991-ce01ca2d6414\") " Oct 30 05:40:26.381694 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.381667 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b19257-a9ac-42dc-a991-ce01ca2d6414-plugins-config-volume" pod "06b19257-a9ac-42dc-a991-ce01ca2d6414" (UID: "06b19257-a9ac-42dc-a991-ce01ca2d6414"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:26.383458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.383438 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b19257-a9ac-42dc-a991-ce01ca2d6414-kube-api-access-scjrh" pod "06b19257-a9ac-42dc-a991-ce01ca2d6414" (UID: "06b19257-a9ac-42dc-a991-ce01ca2d6414"). InnerVolumeSpecName "kube-api-access-scjrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:26.393896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.393872 6587 generic.go:356] "Generic (PLEG): container finished" podID="06b19257-a9ac-42dc-a991-ce01ca2d6414" containerID="cce08864701e4f421e69d4ff6f0eba5ea38aa4a180c8d3bbe11f41b9daa854a0" exitCode=0 Oct 30 05:40:26.394014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.393914 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" Oct 30 05:40:26.394014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.393937 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" event={"ID":"06b19257-a9ac-42dc-a991-ce01ca2d6414","Type":"ContainerDied","Data":"cce08864701e4f421e69d4ff6f0eba5ea38aa4a180c8d3bbe11f41b9daa854a0"} Oct 30 05:40:26.394014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.393982 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr" event={"ID":"06b19257-a9ac-42dc-a991-ce01ca2d6414","Type":"ContainerDied","Data":"4d4455fa0d4937391196f49ca8aecbb628c32e5c3aef25416d80ddb4c1eaae9e"} Oct 30 05:40:26.394014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.394003 6587 scope.go:122] "RemoveContainer" containerID="cce08864701e4f421e69d4ff6f0eba5ea38aa4a180c8d3bbe11f41b9daa854a0" Oct 30 05:40:26.438650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.438627 6587 scope.go:122] "RemoveContainer" containerID="cce08864701e4f421e69d4ff6f0eba5ea38aa4a180c8d3bbe11f41b9daa854a0" Oct 30 05:40:26.438972 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:40:26.438951 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce08864701e4f421e69d4ff6f0eba5ea38aa4a180c8d3bbe11f41b9daa854a0\": container with ID starting with cce08864701e4f421e69d4ff6f0eba5ea38aa4a180c8d3bbe11f41b9daa854a0 not found: ID does not exist" containerID="cce08864701e4f421e69d4ff6f0eba5ea38aa4a180c8d3bbe11f41b9daa854a0" Oct 30 05:40:26.439045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.438982 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce08864701e4f421e69d4ff6f0eba5ea38aa4a180c8d3bbe11f41b9daa854a0"} err="failed to get container status \"cce08864701e4f421e69d4ff6f0eba5ea38aa4a180c8d3bbe11f41b9daa854a0\": rpc error: code = NotFound desc = could not find container \"cce08864701e4f421e69d4ff6f0eba5ea38aa4a180c8d3bbe11f41b9daa854a0\": container with ID starting with cce08864701e4f421e69d4ff6f0eba5ea38aa4a180c8d3bbe11f41b9daa854a0 not found: ID does not exist" Oct 30 05:40:26.445870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.445841 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr"] Oct 30 05:40:26.447629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.447607 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr"] Oct 30 05:40:26.482874 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.482850 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b19257-a9ac-42dc-a991-ce01ca2d6414-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:26.482979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:26.482875 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scjrh\" (UniqueName: \"kubernetes.io/projected/06b19257-a9ac-42dc-a991-ce01ca2d6414-kube-api-access-scjrh\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:27.579648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:27.579616 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b19257-a9ac-42dc-a991-ce01ca2d6414" path="/var/lib/kubelet/pods/06b19257-a9ac-42dc-a991-ce01ca2d6414/volumes" Oct 30 05:40:27.736131 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:27.736088 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" podUID="263665e1-3eeb-43a5-adb8-82a0d9269a0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:40:27.877127 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:27.877102 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:40:27.999852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:27.999820 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-envoy\") pod \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " Oct 30 05:40:28.000076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:27.999891 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-kube-api-access-g4f7k\" (UniqueName: \"kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-kube-api-access-g4f7k\") pod \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " Oct 30 05:40:28.000076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:27.999918 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istiod-ca-cert\") pod \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " Oct 30 05:40:28.000076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000000 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-data\") pod \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " Oct 30 05:40:28.000076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000043 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-podinfo\") pod \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " Oct 30 05:40:28.000322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000127 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-token\" (UniqueName: \"kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-token\") pod \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " Oct 30 05:40:28.000322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000185 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-credential-socket\") pod \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " Oct 30 05:40:28.000322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000232 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-socket\") pod \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " Oct 30 05:40:28.000322 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000273 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-certs\") pod \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\" (UID: \"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35\") " Oct 30 05:40:28.000658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000312 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-data" pod "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" (UID: "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:28.000658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000341 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istiod-ca-cert" pod "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" (UID: "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:28.000658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000527 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-credential-socket" pod "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" (UID: "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:28.000658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000609 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-socket" pod "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" (UID: "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:28.000658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000652 6587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istiod-ca-cert\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:28.000951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000674 6587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-data\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:28.000951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000689 6587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-credential-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:28.000951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.000723 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-certs" pod "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" (UID: "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:28.002328 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.002301 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-envoy" pod "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" (UID: "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:28.002477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.002457 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-kube-api-access-g4f7k" pod "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" (UID: "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35"). InnerVolumeSpecName "kube-api-access-g4f7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:28.002860 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.002835 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-podinfo" pod "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" (UID: "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:40:28.002944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.002872 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-token" pod "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" (UID: "9cd62d9d-4486-47d9-8256-f6f4cd9d4f35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:28.102039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.101948 6587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-socket\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:28.102039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.101989 6587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-workload-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:28.102039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.102004 6587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-envoy\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:28.102039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.102017 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4f7k\" (UniqueName: \"kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-kube-api-access-g4f7k\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:28.102039 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.102033 6587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-podinfo\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:28.102341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.102047 6587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35-istio-token\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:28.409938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.409845 6587 generic.go:356] "Generic (PLEG): container finished" podID="9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" containerID="2f0504d501789f9135992f3f10898db5782c6dce766da487ad5f745b04eababd" exitCode=0 Oct 30 05:40:28.409938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.409903 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" Oct 30 05:40:28.409938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.409928 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" event={"ID":"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35","Type":"ContainerDied","Data":"2f0504d501789f9135992f3f10898db5782c6dce766da487ad5f745b04eababd"} Oct 30 05:40:28.410211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.409985 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f" event={"ID":"9cd62d9d-4486-47d9-8256-f6f4cd9d4f35","Type":"ContainerDied","Data":"0ff993309cda266fe9895eebdd30296ee0d39ce0ba2b4b4d65be67c3dea9f241"} Oct 30 05:40:28.410211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.410008 6587 scope.go:122] "RemoveContainer" containerID="2f0504d501789f9135992f3f10898db5782c6dce766da487ad5f745b04eababd" Oct 30 05:40:28.421800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.421779 6587 scope.go:122] "RemoveContainer" containerID="2f0504d501789f9135992f3f10898db5782c6dce766da487ad5f745b04eababd" Oct 30 05:40:28.422092 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:40:28.422070 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0504d501789f9135992f3f10898db5782c6dce766da487ad5f745b04eababd\": container with ID starting with 2f0504d501789f9135992f3f10898db5782c6dce766da487ad5f745b04eababd not found: ID does not exist" containerID="2f0504d501789f9135992f3f10898db5782c6dce766da487ad5f745b04eababd" Oct 30 05:40:28.422147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.422104 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0504d501789f9135992f3f10898db5782c6dce766da487ad5f745b04eababd"} err="failed to get container status \"2f0504d501789f9135992f3f10898db5782c6dce766da487ad5f745b04eababd\": rpc error: code = NotFound desc = could not find container \"2f0504d501789f9135992f3f10898db5782c6dce766da487ad5f745b04eababd\": container with ID starting with 2f0504d501789f9135992f3f10898db5782c6dce766da487ad5f745b04eababd not found: ID does not exist" Oct 30 05:40:28.473423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.473375 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f"] Oct 30 05:40:28.484218 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:28.484189 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f"] Oct 30 05:40:29.580154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:29.580110 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" path="/var/lib/kubelet/pods/9cd62d9d-4486-47d9-8256-f6f4cd9d4f35/volumes" Oct 30 05:40:30.395101 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.395034 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" containerID="cri-o://16ce9a6d9bbae1d4dd4a9ea05fd5020eae94c018ffe07f56cdfee6835856c975" gracePeriod=45 Oct 30 05:40:30.660070 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.660009 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:40:30.727388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.727337 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-model-cache\") pod \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " Oct 30 05:40:30.727718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.727691 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tmp-dir\") pod \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " Oct 30 05:40:30.727855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.727738 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tls-certs\") pod \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " Oct 30 05:40:30.727855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.727781 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faad5ec5-3afc-4be7-8a65-6072b41c51a7-kube-api-access-fwrwg\" (UniqueName: \"kubernetes.io/projected/faad5ec5-3afc-4be7-8a65-6072b41c51a7-kube-api-access-fwrwg\") pod \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " Oct 30 05:40:30.728017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.727890 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-home\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-home\") pod \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " Oct 30 05:40:30.728017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.727922 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-dshm\") pod \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\" (UID: \"faad5ec5-3afc-4be7-8a65-6072b41c51a7\") " Oct 30 05:40:30.728852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.727693 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-model-cache" pod "faad5ec5-3afc-4be7-8a65-6072b41c51a7" (UID: "faad5ec5-3afc-4be7-8a65-6072b41c51a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:30.728852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.728388 6587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-model-cache\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.728852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.727930 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tmp-dir" pod "faad5ec5-3afc-4be7-8a65-6072b41c51a7" (UID: "faad5ec5-3afc-4be7-8a65-6072b41c51a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:30.729507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.729171 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-home" pod "faad5ec5-3afc-4be7-8a65-6072b41c51a7" (UID: "faad5ec5-3afc-4be7-8a65-6072b41c51a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:30.731161 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.731130 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tls-certs" pod "faad5ec5-3afc-4be7-8a65-6072b41c51a7" (UID: "faad5ec5-3afc-4be7-8a65-6072b41c51a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:30.731871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.731847 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faad5ec5-3afc-4be7-8a65-6072b41c51a7-kube-api-access-fwrwg" pod "faad5ec5-3afc-4be7-8a65-6072b41c51a7" (UID: "faad5ec5-3afc-4be7-8a65-6072b41c51a7"). InnerVolumeSpecName "kube-api-access-fwrwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:30.731982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.731876 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-dshm" pod "faad5ec5-3afc-4be7-8a65-6072b41c51a7" (UID: "faad5ec5-3afc-4be7-8a65-6072b41c51a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:30.829900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.829861 6587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tmp-dir\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.829900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.829901 6587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faad5ec5-3afc-4be7-8a65-6072b41c51a7-tls-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.830134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.829915 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwrwg\" (UniqueName: \"kubernetes.io/projected/faad5ec5-3afc-4be7-8a65-6072b41c51a7-kube-api-access-fwrwg\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.830134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.829927 6587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-home\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.830134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:30.829938 6587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faad5ec5-3afc-4be7-8a65-6072b41c51a7-dshm\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:31.432009 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:31.431977 6587 generic.go:356] "Generic (PLEG): container finished" podID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerID="16ce9a6d9bbae1d4dd4a9ea05fd5020eae94c018ffe07f56cdfee6835856c975" exitCode=0 Oct 30 05:40:31.432211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:31.432020 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" Oct 30 05:40:31.432211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:31.432065 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" event={"ID":"faad5ec5-3afc-4be7-8a65-6072b41c51a7","Type":"ContainerDied","Data":"16ce9a6d9bbae1d4dd4a9ea05fd5020eae94c018ffe07f56cdfee6835856c975"} Oct 30 05:40:31.432211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:31.432109 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst" event={"ID":"faad5ec5-3afc-4be7-8a65-6072b41c51a7","Type":"ContainerDied","Data":"d37c4fdbaa5418237112e1b1a35afd0b9e9e0c68dedbb771234f9a613c77f42a"} Oct 30 05:40:31.432211 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:31.432128 6587 scope.go:122] "RemoveContainer" containerID="16ce9a6d9bbae1d4dd4a9ea05fd5020eae94c018ffe07f56cdfee6835856c975" Oct 30 05:40:31.442984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:31.442968 6587 scope.go:122] "RemoveContainer" containerID="16ce9a6d9bbae1d4dd4a9ea05fd5020eae94c018ffe07f56cdfee6835856c975" Oct 30 05:40:31.443224 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:40:31.443207 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ce9a6d9bbae1d4dd4a9ea05fd5020eae94c018ffe07f56cdfee6835856c975\": container with ID starting with 16ce9a6d9bbae1d4dd4a9ea05fd5020eae94c018ffe07f56cdfee6835856c975 not found: ID does not exist" containerID="16ce9a6d9bbae1d4dd4a9ea05fd5020eae94c018ffe07f56cdfee6835856c975" Oct 30 05:40:31.443286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:31.443230 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ce9a6d9bbae1d4dd4a9ea05fd5020eae94c018ffe07f56cdfee6835856c975"} err="failed to get container status \"16ce9a6d9bbae1d4dd4a9ea05fd5020eae94c018ffe07f56cdfee6835856c975\": rpc error: code = NotFound desc = could not find container \"16ce9a6d9bbae1d4dd4a9ea05fd5020eae94c018ffe07f56cdfee6835856c975\": container with ID starting with 16ce9a6d9bbae1d4dd4a9ea05fd5020eae94c018ffe07f56cdfee6835856c975 not found: ID does not exist" Oct 30 05:40:31.480255 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:31.480213 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst"] Oct 30 05:40:31.492195 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:31.492171 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/simulator-w1-cff4ac-kserve-5756c7bc77-wlvst"] Oct 30 05:40:31.579669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:31.579629 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" path="/var/lib/kubelet/pods/faad5ec5-3afc-4be7-8a65-6072b41c51a7/volumes" Oct 30 05:40:37.457647 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:37.457612 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5"] Oct 30 05:40:37.469961 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:37.469924 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cff4ac-hr5r5"] Oct 30 05:40:37.579444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:37.579410 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b3053d-c050-4d4c-bca3-22b3905f20eb" path="/var/lib/kubelet/pods/55b3053d-c050-4d4c-bca3-22b3905f20eb/volumes" Oct 30 05:40:37.735254 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:37.735169 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" podUID="263665e1-3eeb-43a5-adb8-82a0d9269a0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:40:44.525655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:44.525620 6587 scope.go:122] "RemoveContainer" containerID="3d00a673f3f95c5a443fdea3bdb8b092ae1f798bc74c296b55f3f2bc49b64ed2" Oct 30 05:40:44.535137 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:44.535114 6587 scope.go:122] "RemoveContainer" containerID="f6febf50708ca8eb30653c0ee7955fc3e508a7db7e95c62a70cbf008ff780d7a" Oct 30 05:40:44.543747 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:44.543727 6587 scope.go:122] "RemoveContainer" containerID="2b03bfaba61d21185de28acccfdfb5e7bcbb6017fe0fb8d352ccfed2cd82386d" Oct 30 05:40:44.552069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:44.552053 6587 scope.go:122] "RemoveContainer" containerID="a50e8f21b0ff89224d0e6fefb7b563a49f0a06ed52d8f05e935c929efa7e9f8a" Oct 30 05:40:44.560590 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:44.560570 6587 scope.go:122] "RemoveContainer" containerID="2ae9b87260dec2eaee468d8e01115514d3f5fc534a336ecfc0cce15bc2d9274c" Oct 30 05:40:44.569634 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:44.569614 6587 scope.go:122] "RemoveContainer" containerID="ab206525956691618afe6f00447677c3e7a6a80e94fde926a86bc01a2fd6122f" Oct 30 05:40:46.345376 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.345295 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" Oct 30 05:40:46.481462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.481423 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263665e1-3eeb-43a5-adb8-82a0d9269a0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263665e1-3eeb-43a5-adb8-82a0d9269a0b-plugins-config-volume\") pod \"263665e1-3eeb-43a5-adb8-82a0d9269a0b\" (UID: \"263665e1-3eeb-43a5-adb8-82a0d9269a0b\") " Oct 30 05:40:46.481670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.481534 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263665e1-3eeb-43a5-adb8-82a0d9269a0b-kube-api-access-k79rx\" (UniqueName: \"kubernetes.io/projected/263665e1-3eeb-43a5-adb8-82a0d9269a0b-kube-api-access-k79rx\") pod \"263665e1-3eeb-43a5-adb8-82a0d9269a0b\" (UID: \"263665e1-3eeb-43a5-adb8-82a0d9269a0b\") " Oct 30 05:40:46.481819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.481796 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263665e1-3eeb-43a5-adb8-82a0d9269a0b-plugins-config-volume" pod "263665e1-3eeb-43a5-adb8-82a0d9269a0b" (UID: "263665e1-3eeb-43a5-adb8-82a0d9269a0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:46.481899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.481886 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263665e1-3eeb-43a5-adb8-82a0d9269a0b-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:46.483642 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.483622 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263665e1-3eeb-43a5-adb8-82a0d9269a0b-kube-api-access-k79rx" pod "263665e1-3eeb-43a5-adb8-82a0d9269a0b" (UID: "263665e1-3eeb-43a5-adb8-82a0d9269a0b"). InnerVolumeSpecName "kube-api-access-k79rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:46.530352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.530320 6587 generic.go:356] "Generic (PLEG): container finished" podID="263665e1-3eeb-43a5-adb8-82a0d9269a0b" containerID="a41cac8a4f65730abbc6f8933a631e2eaee9e3d7a691cc9b8f9aa8f609201151" exitCode=0 Oct 30 05:40:46.530535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.530394 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" event={"ID":"263665e1-3eeb-43a5-adb8-82a0d9269a0b","Type":"ContainerDied","Data":"a41cac8a4f65730abbc6f8933a631e2eaee9e3d7a691cc9b8f9aa8f609201151"} Oct 30 05:40:46.530535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.530411 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" Oct 30 05:40:46.530535 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.530501 6587 scope.go:122] "RemoveContainer" containerID="a41cac8a4f65730abbc6f8933a631e2eaee9e3d7a691cc9b8f9aa8f609201151" Oct 30 05:40:46.530661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.530491 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn" event={"ID":"263665e1-3eeb-43a5-adb8-82a0d9269a0b","Type":"ContainerDied","Data":"5f2a0a9eb607dea9f5913925dd1cf1beab7f7fc604903878ffe6a69b3e847cb2"} Oct 30 05:40:46.542529 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.542506 6587 scope.go:122] "RemoveContainer" containerID="a41cac8a4f65730abbc6f8933a631e2eaee9e3d7a691cc9b8f9aa8f609201151" Oct 30 05:40:46.542844 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:40:46.542825 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41cac8a4f65730abbc6f8933a631e2eaee9e3d7a691cc9b8f9aa8f609201151\": container with ID starting with a41cac8a4f65730abbc6f8933a631e2eaee9e3d7a691cc9b8f9aa8f609201151 not found: ID does not exist" containerID="a41cac8a4f65730abbc6f8933a631e2eaee9e3d7a691cc9b8f9aa8f609201151" Oct 30 05:40:46.542902 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.542856 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41cac8a4f65730abbc6f8933a631e2eaee9e3d7a691cc9b8f9aa8f609201151"} err="failed to get container status \"a41cac8a4f65730abbc6f8933a631e2eaee9e3d7a691cc9b8f9aa8f609201151\": rpc error: code = NotFound desc = could not find container \"a41cac8a4f65730abbc6f8933a631e2eaee9e3d7a691cc9b8f9aa8f609201151\": container with ID starting with a41cac8a4f65730abbc6f8933a631e2eaee9e3d7a691cc9b8f9aa8f609201151 not found: ID does not exist" Oct 30 05:40:46.579567 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.579537 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn"] Oct 30 05:40:46.582511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.582488 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k79rx\" (UniqueName: \"kubernetes.io/projected/263665e1-3eeb-43a5-adb8-82a0d9269a0b-kube-api-access-k79rx\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:40:46.584546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:46.584526 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn"] Oct 30 05:40:47.580203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:40:47.580170 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263665e1-3eeb-43a5-adb8-82a0d9269a0b" path="/var/lib/kubelet/pods/263665e1-3eeb-43a5-adb8-82a0d9269a0b/volumes" Oct 30 05:40:55.986137 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:40:55.986093 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:41:04.705961 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.705928 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92"] Oct 30 05:41:04.706379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706339 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3baee78b-9919-4aab-a971-7251ce5ecd29" containerName="payload-pre-processing" Oct 30 05:41:04.706379 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706351 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3baee78b-9919-4aab-a971-7251ce5ecd29" containerName="payload-pre-processing" Oct 30 05:41:04.706496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706390 6587 cpu_manager.go:414] "containerMap: removing container" podUID="573bf036-fa9c-437d-bb41-8c93e82cb568" containerName="payload-processing" Oct 30 05:41:04.706496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706396 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="573bf036-fa9c-437d-bb41-8c93e82cb568" containerName="payload-processing" Oct 30 05:41:04.706496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706417 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" containerName="istio-proxy" Oct 30 05:41:04.706496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706422 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" containerName="istio-proxy" Oct 30 05:41:04.706496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706431 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d44fd769-67e4-411a-8a2e-ee50329ec6a0" containerName="revoke-keys" Oct 30 05:41:04.706496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706436 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44fd769-67e4-411a-8a2e-ee50329ec6a0" containerName="revoke-keys" Oct 30 05:41:04.706496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706447 6587 cpu_manager.go:414] "containerMap: removing container" podUID="263665e1-3eeb-43a5-adb8-82a0d9269a0b" containerName="payload-pre-processing" Oct 30 05:41:04.706496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706452 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263665e1-3eeb-43a5-adb8-82a0d9269a0b" containerName="payload-pre-processing" Oct 30 05:41:04.706496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706460 6587 cpu_manager.go:414] "containerMap: removing container" podUID="06b19257-a9ac-42dc-a991-ce01ca2d6414" containerName="payload-processing" Oct 30 05:41:04.706496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706464 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b19257-a9ac-42dc-a991-ce01ca2d6414" containerName="payload-processing" Oct 30 05:41:04.706496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706471 6587 cpu_manager.go:414] "containerMap: removing container" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" Oct 30 05:41:04.706496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706476 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" Oct 30 05:41:04.706899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706535 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="263665e1-3eeb-43a5-adb8-82a0d9269a0b" containerName="payload-pre-processing" Oct 30 05:41:04.706899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706545 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b19257-a9ac-42dc-a991-ce01ca2d6414" containerName="payload-processing" Oct 30 05:41:04.706899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706552 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="faad5ec5-3afc-4be7-8a65-6072b41c51a7" containerName="main" Oct 30 05:41:04.706899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706560 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3baee78b-9919-4aab-a971-7251ce5ecd29" containerName="payload-pre-processing" Oct 30 05:41:04.706899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706567 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="573bf036-fa9c-437d-bb41-8c93e82cb568" containerName="payload-processing" Oct 30 05:41:04.706899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706575 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd62d9d-4486-47d9-8256-f6f4cd9d4f35" containerName="istio-proxy" Oct 30 05:41:04.706899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706582 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44fd769-67e4-411a-8a2e-ee50329ec6a0" containerName="revoke-keys" Oct 30 05:41:04.706899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.706587 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9e5f93-4df2-4f1d-90a7-5e5e5ddf22ce" containerName="istio-proxy" Oct 30 05:41:04.711029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.711008 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.737079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.737046 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92"] Oct 30 05:41:04.761035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.761000 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.761278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.761253 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.761395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.761295 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.761499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.761467 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.761561 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.761517 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.761618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.761602 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.761656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.761639 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.761778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.761751 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb86n\" (UniqueName: \"kubernetes.io/projected/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-kube-api-access-vb86n\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.761871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.761852 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.862522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.862488 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.862522 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.862525 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.862773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.862551 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.862773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.862574 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.862773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.862620 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb86n\" (UniqueName: \"kubernetes.io/projected/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-kube-api-access-vb86n\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.862773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.862669 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.862773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.862698 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.862773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.862736 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.862773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.862765 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.863185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.862972 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.863185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.863046 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.863185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.863134 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.863406 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.863216 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.863640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.863615 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.864979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.864954 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.865082 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.865062 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.888094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.888068 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb86n\" (UniqueName: \"kubernetes.io/projected/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-kube-api-access-vb86n\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:04.889640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:04.889621 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-jbc92\" (UID: \"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:05.025476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:05.025393 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:05.331665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:05.331638 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92"] Oct 30 05:41:05.333337 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:41:05.333307 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41789fb_0a8e_48dd_8cb0_8b9af7d70ffa.slice/crio-a1a07b7cf1114ff6a65a277b5d5902cd45407845821ca50ea8cb10345615cc7f WatchSource:0}: Error finding container a1a07b7cf1114ff6a65a277b5d5902cd45407845821ca50ea8cb10345615cc7f: Status 404 returned error can't find the container with id a1a07b7cf1114ff6a65a277b5d5902cd45407845821ca50ea8cb10345615cc7f Oct 30 05:41:05.335651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:05.335621 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:41:05.335768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:05.335697 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:41:05.335882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:05.335775 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:41:05.649391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:05.649280 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" event={"ID":"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa","Type":"ContainerStarted","Data":"245b1a3a7beb24f0a19032241c3f9e19d956e9fbfa895dc04b43f7a9cfe39af1"} Oct 30 05:41:05.649391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:05.649372 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" event={"ID":"b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa","Type":"ContainerStarted","Data":"a1a07b7cf1114ff6a65a277b5d5902cd45407845821ca50ea8cb10345615cc7f"} Oct 30 05:41:05.695058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:05.694986 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" podStartSLOduration=1.694966114 podStartE2EDuration="1.694966114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:05.693933107 +0000 UTC m=+3390.221699894" watchObservedRunningTime="2026-10-30 05:41:05.694966114 +0000 UTC m=+3390.222732902" Oct 30 05:41:06.031442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:06.028786 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:06.040553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:06.040507 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" start-of-body= Oct 30 05:41:06.040745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:06.040597 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" podUID="b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" Oct 30 05:41:07.040381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:07.032787 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" start-of-body= Oct 30 05:41:07.040381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:07.032868 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" podUID="b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" Oct 30 05:41:08.040501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:08.040461 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" start-of-body= Oct 30 05:41:08.040931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:08.040532 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" podUID="b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" Oct 30 05:41:09.028395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:09.026828 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" start-of-body= Oct 30 05:41:09.028395 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:09.026906 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" podUID="b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" Oct 30 05:41:10.038381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:10.037838 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" start-of-body= Oct 30 05:41:10.038381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:10.037897 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" podUID="b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" Oct 30 05:41:11.037768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:11.037728 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" start-of-body= Oct 30 05:41:11.038066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:11.038034 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" podUID="b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" Oct 30 05:41:12.026085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:12.026044 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" start-of-body= Oct 30 05:41:12.026513 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:12.026110 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" podUID="b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.166:15021/healthz/ready\": dial tcp 10.134.1.166:15021: connect: connection refused" Oct 30 05:41:13.692475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:13.692437 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:13.692953 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:13.692929 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:13.693874 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:13.693852 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" Oct 30 05:41:14.215074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.215036 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m"] Oct 30 05:41:14.218595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.218577 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" Oct 30 05:41:14.238174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.238149 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:41:14.238313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.238258 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:41:14.238313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.238266 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:41:14.238313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.238283 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:41:14.247933 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.247908 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m"] Oct 30 05:41:14.380819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.380783 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-h6z2m\" (UID: \"774c4c38-642b-44a1-8424-373c0c5ab10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" Oct 30 05:41:14.381003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.380834 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr7pp\" (UniqueName: \"kubernetes.io/projected/774c4c38-642b-44a1-8424-373c0c5ab10b-kube-api-access-dr7pp\") pod \"maas-api-revoke-sub-85237e45-h6z2m\" (UID: \"774c4c38-642b-44a1-8424-373c0c5ab10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" Oct 30 05:41:14.381003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.380890 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-h6z2m\" (UID: \"774c4c38-642b-44a1-8424-373c0c5ab10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" Oct 30 05:41:14.482326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.482247 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-h6z2m\" (UID: \"774c4c38-642b-44a1-8424-373c0c5ab10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" Oct 30 05:41:14.482326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.482293 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr7pp\" (UniqueName: \"kubernetes.io/projected/774c4c38-642b-44a1-8424-373c0c5ab10b-kube-api-access-dr7pp\") pod \"maas-api-revoke-sub-85237e45-h6z2m\" (UID: \"774c4c38-642b-44a1-8424-373c0c5ab10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" Oct 30 05:41:14.482326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.482310 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-h6z2m\" (UID: \"774c4c38-642b-44a1-8424-373c0c5ab10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" Oct 30 05:41:14.483035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.483011 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-h6z2m\" (UID: \"774c4c38-642b-44a1-8424-373c0c5ab10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" Oct 30 05:41:14.484736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.484707 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-h6z2m\" (UID: \"774c4c38-642b-44a1-8424-373c0c5ab10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" Oct 30 05:41:14.502657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.502627 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr7pp\" (UniqueName: \"kubernetes.io/projected/774c4c38-642b-44a1-8424-373c0c5ab10b-kube-api-access-dr7pp\") pod \"maas-api-revoke-sub-85237e45-h6z2m\" (UID: \"774c4c38-642b-44a1-8424-373c0c5ab10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" Oct 30 05:41:14.532639 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.532614 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" Oct 30 05:41:14.829621 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:14.829589 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m"] Oct 30 05:41:14.830776 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:41:14.830748 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774c4c38_642b_44a1_8424_373c0c5ab10b.slice/crio-bad692cc4f3e54902c9b4e7aa84535d30f7787babc006be3624bc5c7e83ca1b5 WatchSource:0}: Error finding container bad692cc4f3e54902c9b4e7aa84535d30f7787babc006be3624bc5c7e83ca1b5: Status 404 returned error can't find the container with id bad692cc4f3e54902c9b4e7aa84535d30f7787babc006be3624bc5c7e83ca1b5 Oct 30 05:41:15.716421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:15.716371 6587 generic.go:356] "Generic (PLEG): container finished" podID="774c4c38-642b-44a1-8424-373c0c5ab10b" containerID="8549ce085a3cd882bedfd1ad204faf94f8a9b85ae63f9fa1c1ffffe449c436e6" exitCode=0 Oct 30 05:41:15.716638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:15.716461 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" event={"ID":"774c4c38-642b-44a1-8424-373c0c5ab10b","Type":"ContainerDied","Data":"8549ce085a3cd882bedfd1ad204faf94f8a9b85ae63f9fa1c1ffffe449c436e6"} Oct 30 05:41:15.716638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:15.716518 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" event={"ID":"774c4c38-642b-44a1-8424-373c0c5ab10b","Type":"ContainerStarted","Data":"bad692cc4f3e54902c9b4e7aa84535d30f7787babc006be3624bc5c7e83ca1b5"} Oct 30 05:41:16.874320 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:16.874297 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" Oct 30 05:41:17.007617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:17.007525 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-openshift-ca\") pod \"774c4c38-642b-44a1-8424-373c0c5ab10b\" (UID: \"774c4c38-642b-44a1-8424-373c0c5ab10b\") " Oct 30 05:41:17.007617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:17.007580 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774c4c38-642b-44a1-8424-373c0c5ab10b-kube-api-access-dr7pp\" (UniqueName: \"kubernetes.io/projected/774c4c38-642b-44a1-8424-373c0c5ab10b-kube-api-access-dr7pp\") pod \"774c4c38-642b-44a1-8424-373c0c5ab10b\" (UID: \"774c4c38-642b-44a1-8424-373c0c5ab10b\") " Oct 30 05:41:17.007863 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:17.007710 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-xks-ca\") pod \"774c4c38-642b-44a1-8424-373c0c5ab10b\" (UID: \"774c4c38-642b-44a1-8424-373c0c5ab10b\") " Oct 30 05:41:17.007946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:17.007923 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-openshift-ca" pod "774c4c38-642b-44a1-8424-373c0c5ab10b" (UID: "774c4c38-642b-44a1-8424-373c0c5ab10b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:41:17.008099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:17.008073 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:41:17.009688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:17.009663 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-xks-ca" pod "774c4c38-642b-44a1-8424-373c0c5ab10b" (UID: "774c4c38-642b-44a1-8424-373c0c5ab10b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:41:17.009780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:17.009710 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774c4c38-642b-44a1-8424-373c0c5ab10b-kube-api-access-dr7pp" pod "774c4c38-642b-44a1-8424-373c0c5ab10b" (UID: "774c4c38-642b-44a1-8424-373c0c5ab10b"). InnerVolumeSpecName "kube-api-access-dr7pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:41:17.108652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:17.108616 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774c4c38-642b-44a1-8424-373c0c5ab10b-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:41:17.108652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:17.108649 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr7pp\" (UniqueName: \"kubernetes.io/projected/774c4c38-642b-44a1-8424-373c0c5ab10b-kube-api-access-dr7pp\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:41:17.732576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:17.732540 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" Oct 30 05:41:17.732795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:17.732576 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m" event={"ID":"774c4c38-642b-44a1-8424-373c0c5ab10b","Type":"ContainerDied","Data":"bad692cc4f3e54902c9b4e7aa84535d30f7787babc006be3624bc5c7e83ca1b5"} Oct 30 05:41:17.732795 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:17.732623 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bad692cc4f3e54902c9b4e7aa84535d30f7787babc006be3624bc5c7e83ca1b5" Oct 30 05:41:25.984458 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:41:25.984421 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:41:27.031661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:27.031621 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6"] Oct 30 05:41:27.043540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:27.043510 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0439879d-zmhv6"] Oct 30 05:41:27.580053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:27.580019 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa50cecf-fcd8-4ada-98f2-787c4d9942ac" path="/var/lib/kubelet/pods/aa50cecf-fcd8-4ada-98f2-787c4d9942ac/volumes" Oct 30 05:41:28.030741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:28.030657 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6"] Oct 30 05:41:28.042281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:28.042251 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79946a68-kdhb6"] Oct 30 05:41:29.581193 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:29.581159 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e12583-5094-4e4d-9593-c71070951f96" path="/var/lib/kubelet/pods/f1e12583-5094-4e4d-9593-c71070951f96/volumes" Oct 30 05:41:44.697489 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:44.697456 6587 scope.go:122] "RemoveContainer" containerID="17917854a53a29fdc86d92df1a90880b3fc5fa801dbf353cb75d73128a3a1869" Oct 30 05:41:44.706812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:44.706787 6587 scope.go:122] "RemoveContainer" containerID="4b60bf0dd542950ce8b20fd7f10f0955189404b1bdbcfafb866f8a8c26b5ba26" Oct 30 05:41:44.715683 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:44.715664 6587 scope.go:122] "RemoveContainer" containerID="6ad95de9f665e51f6c54d0681b7e75f6756a1684ec0d4c64c119648f6c8ed63e" Oct 30 05:41:44.724347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:41:44.724327 6587 scope.go:122] "RemoveContainer" containerID="2b1d9bcb5d3d89ad7a8180d271d4116381574cd60c04ee1e67ee6d5d7ff419c2" Oct 30 05:41:55.997007 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:41:55.996913 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:42:02.409905 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.409868 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd"] Oct 30 05:42:02.410457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.410289 6587 cpu_manager.go:414] "containerMap: removing container" podUID="774c4c38-642b-44a1-8424-373c0c5ab10b" containerName="revoke-keys" Oct 30 05:42:02.410457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.410300 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774c4c38-642b-44a1-8424-373c0c5ab10b" containerName="revoke-keys" Oct 30 05:42:02.410457 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.410410 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="774c4c38-642b-44a1-8424-373c0c5ab10b" containerName="revoke-keys" Oct 30 05:42:02.415399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.415378 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" Oct 30 05:42:02.435458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.435435 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:42:02.447234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.447214 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:42:02.447445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.447432 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:42:02.447549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.447503 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:42:02.449546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.449529 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd"] Oct 30 05:42:02.452881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.452860 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qcvtd\" (UID: \"336378ec-ba3e-45d8-a954-f57cdc98c79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" Oct 30 05:42:02.453000 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.452905 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qcvtd\" (UID: \"336378ec-ba3e-45d8-a954-f57cdc98c79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" Oct 30 05:42:02.453063 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.453011 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmvrj\" (UniqueName: \"kubernetes.io/projected/336378ec-ba3e-45d8-a954-f57cdc98c79d-kube-api-access-tmvrj\") pod \"maas-api-revoke-sub-c1e271a9-qcvtd\" (UID: \"336378ec-ba3e-45d8-a954-f57cdc98c79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" Oct 30 05:42:02.554107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.554071 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmvrj\" (UniqueName: \"kubernetes.io/projected/336378ec-ba3e-45d8-a954-f57cdc98c79d-kube-api-access-tmvrj\") pod \"maas-api-revoke-sub-c1e271a9-qcvtd\" (UID: \"336378ec-ba3e-45d8-a954-f57cdc98c79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" Oct 30 05:42:02.554260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.554188 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qcvtd\" (UID: \"336378ec-ba3e-45d8-a954-f57cdc98c79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" Oct 30 05:42:02.554260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.554242 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qcvtd\" (UID: \"336378ec-ba3e-45d8-a954-f57cdc98c79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" Oct 30 05:42:02.554927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.554904 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qcvtd\" (UID: \"336378ec-ba3e-45d8-a954-f57cdc98c79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" Oct 30 05:42:02.556645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.556623 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qcvtd\" (UID: \"336378ec-ba3e-45d8-a954-f57cdc98c79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" Oct 30 05:42:02.573686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.573658 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmvrj\" (UniqueName: \"kubernetes.io/projected/336378ec-ba3e-45d8-a954-f57cdc98c79d-kube-api-access-tmvrj\") pod \"maas-api-revoke-sub-c1e271a9-qcvtd\" (UID: \"336378ec-ba3e-45d8-a954-f57cdc98c79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" Oct 30 05:42:02.727929 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:02.727870 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" Oct 30 05:42:03.025146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:03.025115 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd"] Oct 30 05:42:03.026683 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:42:03.026653 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336378ec_ba3e_45d8_a954_f57cdc98c79d.slice/crio-ef043b0fb750e457501190c1d7b0e722ead87268f08117ac07df5da897e40ad9 WatchSource:0}: Error finding container ef043b0fb750e457501190c1d7b0e722ead87268f08117ac07df5da897e40ad9: Status 404 returned error can't find the container with id ef043b0fb750e457501190c1d7b0e722ead87268f08117ac07df5da897e40ad9 Oct 30 05:42:04.026960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:04.026881 6587 generic.go:356] "Generic (PLEG): container finished" podID="336378ec-ba3e-45d8-a954-f57cdc98c79d" containerID="7176c5be6a960e1f40e0224a76ab44dfab8c757e69875e25777b3fd99f109e7c" exitCode=0 Oct 30 05:42:04.027392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:04.026962 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" event={"ID":"336378ec-ba3e-45d8-a954-f57cdc98c79d","Type":"ContainerDied","Data":"7176c5be6a960e1f40e0224a76ab44dfab8c757e69875e25777b3fd99f109e7c"} Oct 30 05:42:04.027392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:04.027016 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" event={"ID":"336378ec-ba3e-45d8-a954-f57cdc98c79d","Type":"ContainerStarted","Data":"ef043b0fb750e457501190c1d7b0e722ead87268f08117ac07df5da897e40ad9"} Oct 30 05:42:05.175652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:05.175626 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" Oct 30 05:42:05.281401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:05.281349 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336378ec-ba3e-45d8-a954-f57cdc98c79d-kube-api-access-tmvrj\" (UniqueName: \"kubernetes.io/projected/336378ec-ba3e-45d8-a954-f57cdc98c79d-kube-api-access-tmvrj\") pod \"336378ec-ba3e-45d8-a954-f57cdc98c79d\" (UID: \"336378ec-ba3e-45d8-a954-f57cdc98c79d\") " Oct 30 05:42:05.281595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:05.281452 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-openshift-ca\") pod \"336378ec-ba3e-45d8-a954-f57cdc98c79d\" (UID: \"336378ec-ba3e-45d8-a954-f57cdc98c79d\") " Oct 30 05:42:05.281595 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:05.281551 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-xks-ca\") pod \"336378ec-ba3e-45d8-a954-f57cdc98c79d\" (UID: \"336378ec-ba3e-45d8-a954-f57cdc98c79d\") " Oct 30 05:42:05.281843 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:05.281821 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-openshift-ca" pod "336378ec-ba3e-45d8-a954-f57cdc98c79d" (UID: "336378ec-ba3e-45d8-a954-f57cdc98c79d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:42:05.283601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:05.283549 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-xks-ca" pod "336378ec-ba3e-45d8-a954-f57cdc98c79d" (UID: "336378ec-ba3e-45d8-a954-f57cdc98c79d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:42:05.283601 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:05.283590 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336378ec-ba3e-45d8-a954-f57cdc98c79d-kube-api-access-tmvrj" pod "336378ec-ba3e-45d8-a954-f57cdc98c79d" (UID: "336378ec-ba3e-45d8-a954-f57cdc98c79d"). InnerVolumeSpecName "kube-api-access-tmvrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:42:05.382324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:05.382288 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmvrj\" (UniqueName: \"kubernetes.io/projected/336378ec-ba3e-45d8-a954-f57cdc98c79d-kube-api-access-tmvrj\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:42:05.382324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:05.382317 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:42:05.382324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:05.382329 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336378ec-ba3e-45d8-a954-f57cdc98c79d-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:42:06.039917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:06.039827 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" Oct 30 05:42:06.039917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:06.039848 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd" event={"ID":"336378ec-ba3e-45d8-a954-f57cdc98c79d","Type":"ContainerDied","Data":"ef043b0fb750e457501190c1d7b0e722ead87268f08117ac07df5da897e40ad9"} Oct 30 05:42:06.039917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:06.039890 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef043b0fb750e457501190c1d7b0e722ead87268f08117ac07df5da897e40ad9" Oct 30 05:42:19.932710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:19.930965 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s"] Oct 30 05:42:19.934585 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:19.934556 6587 cpu_manager.go:414] "containerMap: removing container" podUID="336378ec-ba3e-45d8-a954-f57cdc98c79d" containerName="revoke-keys" Oct 30 05:42:19.934762 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:19.934745 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336378ec-ba3e-45d8-a954-f57cdc98c79d" containerName="revoke-keys" Oct 30 05:42:19.934999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:19.934981 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="336378ec-ba3e-45d8-a954-f57cdc98c79d" containerName="revoke-keys" Oct 30 05:42:19.941663 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:19.941642 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:19.993264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:19.993221 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s"] Oct 30 05:42:20.071526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.071417 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35dc1c80-11e5-4647-b911-bf84b7e3336f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.071871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.071847 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.072143 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.072121 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.072377 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.072334 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35dc1c80-11e5-4647-b911-bf84b7e3336f-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.072587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.072566 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.072784 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.072766 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7jmx\" (UniqueName: \"kubernetes.io/projected/35dc1c80-11e5-4647-b911-bf84b7e3336f-kube-api-access-v7jmx\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.072990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.072971 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.073195 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.073177 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.073449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.073427 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35dc1c80-11e5-4647-b911-bf84b7e3336f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.183831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.183735 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35dc1c80-11e5-4647-b911-bf84b7e3336f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.184970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.184862 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35dc1c80-11e5-4647-b911-bf84b7e3336f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.185265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.185235 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.185829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.185705 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.186003 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.185984 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.186454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.186428 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35dc1c80-11e5-4647-b911-bf84b7e3336f-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.186609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.186472 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.186609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.186499 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7jmx\" (UniqueName: \"kubernetes.io/projected/35dc1c80-11e5-4647-b911-bf84b7e3336f-kube-api-access-v7jmx\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.186609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.186540 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.186609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.186577 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.186935 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.186642 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35dc1c80-11e5-4647-b911-bf84b7e3336f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.187470 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.187431 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.187628 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.187593 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.187787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.186372 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.189820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.189800 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35dc1c80-11e5-4647-b911-bf84b7e3336f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.190290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.190271 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35dc1c80-11e5-4647-b911-bf84b7e3336f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.208425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.208403 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35dc1c80-11e5-4647-b911-bf84b7e3336f-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.208563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.208545 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7jmx\" (UniqueName: \"kubernetes.io/projected/35dc1c80-11e5-4647-b911-bf84b7e3336f-kube-api-access-v7jmx\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-db25s\" (UID: \"35dc1c80-11e5-4647-b911-bf84b7e3336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.304908 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.304871 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:20.662789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.662745 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s"] Oct 30 05:42:20.666272 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:42:20.666241 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35dc1c80_11e5_4647_b911_bf84b7e3336f.slice/crio-692be25a354150b0cb423b8bebcb9f081bf4b0d3bf93af83bbbce5a326e11e4e WatchSource:0}: Error finding container 692be25a354150b0cb423b8bebcb9f081bf4b0d3bf93af83bbbce5a326e11e4e: Status 404 returned error can't find the container with id 692be25a354150b0cb423b8bebcb9f081bf4b0d3bf93af83bbbce5a326e11e4e Oct 30 05:42:20.668423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.668389 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:42:20.668527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.668454 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:42:20.668527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:20.668484 6587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:42:21.157296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:21.157244 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" event={"ID":"35dc1c80-11e5-4647-b911-bf84b7e3336f","Type":"ContainerStarted","Data":"f9f5c7f141894b74afc9bb6be1c81445ec576e02d60f1a1f30d31f2a8950c7b6"} Oct 30 05:42:21.157796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:21.157321 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" event={"ID":"35dc1c80-11e5-4647-b911-bf84b7e3336f","Type":"ContainerStarted","Data":"692be25a354150b0cb423b8bebcb9f081bf4b0d3bf93af83bbbce5a326e11e4e"} Oct 30 05:42:21.201728 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:21.201677 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" podStartSLOduration=2.201664043 podStartE2EDuration="2.201664043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:21.199708874 +0000 UTC m=+3465.727475659" watchObservedRunningTime="2026-10-30 05:42:21.201664043 +0000 UTC m=+3465.729430829" Oct 30 05:42:21.305133 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:21.305094 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:21.306502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:21.306472 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" start-of-body= Oct 30 05:42:21.306658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:21.306538 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" podUID="35dc1c80-11e5-4647-b911-bf84b7e3336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" Oct 30 05:42:22.305269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:22.305221 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" start-of-body= Oct 30 05:42:22.305790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:22.305293 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" podUID="35dc1c80-11e5-4647-b911-bf84b7e3336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" Oct 30 05:42:22.835382 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:22.826845 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq"] Oct 30 05:42:22.846641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:22.846601 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" Oct 30 05:42:22.855748 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:22.855191 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq"] Oct 30 05:42:22.863441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:22.863410 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:42:22.863842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:22.863656 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:42:22.863842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:22.863725 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:42:22.873656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:22.873628 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:42:23.039171 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:23.039123 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww62w\" (UniqueName: \"kubernetes.io/projected/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-kube-api-access-ww62w\") pod \"maas-api-revoke-sub-2e1f5b5f-92ncq\" (UID: \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" Oct 30 05:42:23.040565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:23.040528 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-92ncq\" (UID: \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" Oct 30 05:42:23.040713 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:23.040672 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-92ncq\" (UID: \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" Oct 30 05:42:23.145447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:23.141915 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-92ncq\" (UID: \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" Oct 30 05:42:23.145447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:23.142035 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww62w\" (UniqueName: \"kubernetes.io/projected/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-kube-api-access-ww62w\") pod \"maas-api-revoke-sub-2e1f5b5f-92ncq\" (UID: \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" Oct 30 05:42:23.145447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:23.142136 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-92ncq\" (UID: \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" Oct 30 05:42:23.145447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:23.142810 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-92ncq\" (UID: \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" Oct 30 05:42:23.145447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:23.144729 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-92ncq\" (UID: \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" Oct 30 05:42:23.163152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:23.163121 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww62w\" (UniqueName: \"kubernetes.io/projected/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-kube-api-access-ww62w\") pod \"maas-api-revoke-sub-2e1f5b5f-92ncq\" (UID: \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" Oct 30 05:42:23.305699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:23.305665 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" start-of-body= Oct 30 05:42:23.306081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:23.305726 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" podUID="35dc1c80-11e5-4647-b911-bf84b7e3336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" Oct 30 05:42:23.462577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:23.462480 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" Oct 30 05:42:23.785388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:23.785344 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq"] Oct 30 05:42:23.786837 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:42:23.786811 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4fb7b8_d2cd_4eff_899a_7c559a6706cf.slice/crio-cb181be7d60aecbe32c79463c0d808c664ffdca9d38ee9607cdc9552ca1301a0 WatchSource:0}: Error finding container cb181be7d60aecbe32c79463c0d808c664ffdca9d38ee9607cdc9552ca1301a0: Status 404 returned error can't find the container with id cb181be7d60aecbe32c79463c0d808c664ffdca9d38ee9607cdc9552ca1301a0 Oct 30 05:42:24.186465 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:24.186422 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" event={"ID":"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf","Type":"ContainerStarted","Data":"583fa83b5882193857a754e5c7a5bfb5b70e167c8633f5fd46470604cc256624"} Oct 30 05:42:24.186651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:24.186480 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" event={"ID":"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf","Type":"ContainerStarted","Data":"cb181be7d60aecbe32c79463c0d808c664ffdca9d38ee9607cdc9552ca1301a0"} Oct 30 05:42:24.234384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:24.230454 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" podStartSLOduration=2.230432957 podStartE2EDuration="2.230432957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:24.227726479 +0000 UTC m=+3468.755493266" watchObservedRunningTime="2026-10-30 05:42:24.230432957 +0000 UTC m=+3468.758199744" Oct 30 05:42:24.305917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:24.305880 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" start-of-body= Oct 30 05:42:24.306445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:24.305937 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" podUID="35dc1c80-11e5-4647-b911-bf84b7e3336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" Oct 30 05:42:25.193916 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:25.193827 6587 generic.go:356] "Generic (PLEG): container finished" podID="9d4fb7b8-d2cd-4eff-899a-7c559a6706cf" containerID="583fa83b5882193857a754e5c7a5bfb5b70e167c8633f5fd46470604cc256624" exitCode=0 Oct 30 05:42:25.194104 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:25.193909 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" event={"ID":"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf","Type":"ContainerDied","Data":"583fa83b5882193857a754e5c7a5bfb5b70e167c8633f5fd46470604cc256624"} Oct 30 05:42:25.305584 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:25.305548 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" start-of-body= Oct 30 05:42:25.305764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:25.305604 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" podUID="35dc1c80-11e5-4647-b911-bf84b7e3336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" Oct 30 05:42:25.985305 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:42:25.985268 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:42:26.305698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:26.305664 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" start-of-body= Oct 30 05:42:26.305864 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:26.305721 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" podUID="35dc1c80-11e5-4647-b911-bf84b7e3336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" Oct 30 05:42:26.374841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:26.374811 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" Oct 30 05:42:26.471664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:26.471633 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-openshift-ca\") pod \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\" (UID: \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\") " Oct 30 05:42:26.471871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:26.471757 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-kube-api-access-ww62w\" (UniqueName: \"kubernetes.io/projected/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-kube-api-access-ww62w\") pod \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\" (UID: \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\") " Oct 30 05:42:26.471871 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:26.471816 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-xks-ca\") pod \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\" (UID: \"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf\") " Oct 30 05:42:26.472044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:26.472017 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-openshift-ca" pod "9d4fb7b8-d2cd-4eff-899a-7c559a6706cf" (UID: "9d4fb7b8-d2cd-4eff-899a-7c559a6706cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:42:26.472231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:26.472215 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:42:26.473872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:26.473847 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-xks-ca" pod "9d4fb7b8-d2cd-4eff-899a-7c559a6706cf" (UID: "9d4fb7b8-d2cd-4eff-899a-7c559a6706cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:42:26.473872 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:26.473857 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-kube-api-access-ww62w" pod "9d4fb7b8-d2cd-4eff-899a-7c559a6706cf" (UID: "9d4fb7b8-d2cd-4eff-899a-7c559a6706cf"). InnerVolumeSpecName "kube-api-access-ww62w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:42:26.573344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:26.573309 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:42:26.573344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:26.573339 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww62w\" (UniqueName: \"kubernetes.io/projected/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf-kube-api-access-ww62w\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:42:27.208511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:27.208472 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" Oct 30 05:42:27.208913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:27.208483 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq" event={"ID":"9d4fb7b8-d2cd-4eff-899a-7c559a6706cf","Type":"ContainerDied","Data":"cb181be7d60aecbe32c79463c0d808c664ffdca9d38ee9607cdc9552ca1301a0"} Oct 30 05:42:27.208913 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:27.208610 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb181be7d60aecbe32c79463c0d808c664ffdca9d38ee9607cdc9552ca1301a0" Oct 30 05:42:27.305825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:27.305797 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" start-of-body= Oct 30 05:42:27.306017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:27.305848 6587 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" podUID="35dc1c80-11e5-4647-b911-bf84b7e3336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" Oct 30 05:42:28.822548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:28.822508 6587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:28.823049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:28.822836 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:28.823900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:28.823877 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" Oct 30 05:42:34.125530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.125488 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l"] Oct 30 05:42:34.126154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.126138 6587 cpu_manager.go:414] "containerMap: removing container" podUID="9d4fb7b8-d2cd-4eff-899a-7c559a6706cf" containerName="revoke-keys" Oct 30 05:42:34.126202 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.126156 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d4fb7b8-d2cd-4eff-899a-7c559a6706cf" containerName="revoke-keys" Oct 30 05:42:34.126271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.126261 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d4fb7b8-d2cd-4eff-899a-7c559a6706cf" containerName="revoke-keys" Oct 30 05:42:34.130064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.130040 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" Oct 30 05:42:34.146026 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.145997 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:42:34.148732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.148701 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l"] Oct 30 05:42:34.157456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.157416 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:42:34.157456 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.157421 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:42:34.157759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.157731 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:42:34.249907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.249876 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4q7\" (UniqueName: \"kubernetes.io/projected/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-kube-api-access-km4q7\") pod \"maas-api-revoke-sub-ec843018-95d4l\" (UID: \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" Oct 30 05:42:34.249907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.249914 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-95d4l\" (UID: \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" Oct 30 05:42:34.250137 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.250032 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-95d4l\" (UID: \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" Oct 30 05:42:34.350849 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.350812 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4q7\" (UniqueName: \"kubernetes.io/projected/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-kube-api-access-km4q7\") pod \"maas-api-revoke-sub-ec843018-95d4l\" (UID: \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" Oct 30 05:42:34.351065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.350870 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-95d4l\" (UID: \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" Oct 30 05:42:34.351065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.350929 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-95d4l\" (UID: \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" Oct 30 05:42:34.351732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.351701 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-95d4l\" (UID: \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" Oct 30 05:42:34.353407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.353350 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-95d4l\" (UID: \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" Oct 30 05:42:34.370991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.370961 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4q7\" (UniqueName: \"kubernetes.io/projected/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-kube-api-access-km4q7\") pod \"maas-api-revoke-sub-ec843018-95d4l\" (UID: \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" Oct 30 05:42:34.445838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.445748 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" Oct 30 05:42:34.751596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:34.751564 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l"] Oct 30 05:42:34.754538 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:42:34.754502 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7030f58_ff6c_4166_9033_4f97c6f5f2a1.slice/crio-3114624596d1d6e1efef9e22198ab23e7674abbd866811875e5aafc043473600 WatchSource:0}: Error finding container 3114624596d1d6e1efef9e22198ab23e7674abbd866811875e5aafc043473600: Status 404 returned error can't find the container with id 3114624596d1d6e1efef9e22198ab23e7674abbd866811875e5aafc043473600 Oct 30 05:42:35.256847 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:35.256802 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" event={"ID":"c7030f58-ff6c-4166-9033-4f97c6f5f2a1","Type":"ContainerStarted","Data":"2eceafad4caf206a4f3c611629e0cf2f33aa1db49880ce96c323e988ac16f686"} Oct 30 05:42:35.257271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:35.256869 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" event={"ID":"c7030f58-ff6c-4166-9033-4f97c6f5f2a1","Type":"ContainerStarted","Data":"3114624596d1d6e1efef9e22198ab23e7674abbd866811875e5aafc043473600"} Oct 30 05:42:35.297347 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:35.297207 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" podStartSLOduration=1.297187609 podStartE2EDuration="1.297187609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:35.295273323 +0000 UTC m=+3479.823040110" watchObservedRunningTime="2026-10-30 05:42:35.297187609 +0000 UTC m=+3479.824954395" Oct 30 05:42:36.268407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:36.268344 6587 generic.go:356] "Generic (PLEG): container finished" podID="c7030f58-ff6c-4166-9033-4f97c6f5f2a1" containerID="2eceafad4caf206a4f3c611629e0cf2f33aa1db49880ce96c323e988ac16f686" exitCode=0 Oct 30 05:42:36.268947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:36.268491 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" event={"ID":"c7030f58-ff6c-4166-9033-4f97c6f5f2a1","Type":"ContainerDied","Data":"2eceafad4caf206a4f3c611629e0cf2f33aa1db49880ce96c323e988ac16f686"} Oct 30 05:42:37.407782 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:37.407754 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" Oct 30 05:42:37.483118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:37.483082 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-xks-ca\") pod \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\" (UID: \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\") " Oct 30 05:42:37.483118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:37.483122 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-kube-api-access-km4q7\" (UniqueName: \"kubernetes.io/projected/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-kube-api-access-km4q7\") pod \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\" (UID: \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\") " Oct 30 05:42:37.483415 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:37.483155 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-openshift-ca\") pod \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\" (UID: \"c7030f58-ff6c-4166-9033-4f97c6f5f2a1\") " Oct 30 05:42:37.483594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:37.483568 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-openshift-ca" pod "c7030f58-ff6c-4166-9033-4f97c6f5f2a1" (UID: "c7030f58-ff6c-4166-9033-4f97c6f5f2a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:42:37.485234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:37.485213 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-kube-api-access-km4q7" pod "c7030f58-ff6c-4166-9033-4f97c6f5f2a1" (UID: "c7030f58-ff6c-4166-9033-4f97c6f5f2a1"). InnerVolumeSpecName "kube-api-access-km4q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:42:37.485345 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:37.485263 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-xks-ca" pod "c7030f58-ff6c-4166-9033-4f97c6f5f2a1" (UID: "c7030f58-ff6c-4166-9033-4f97c6f5f2a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:42:37.583929 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:37.583893 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:42:37.583929 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:37.583919 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:42:37.583929 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:37.583929 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km4q7\" (UniqueName: \"kubernetes.io/projected/c7030f58-ff6c-4166-9033-4f97c6f5f2a1-kube-api-access-km4q7\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:42:38.282885 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:38.282853 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" Oct 30 05:42:38.283066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:38.282886 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l" event={"ID":"c7030f58-ff6c-4166-9033-4f97c6f5f2a1","Type":"ContainerDied","Data":"3114624596d1d6e1efef9e22198ab23e7674abbd866811875e5aafc043473600"} Oct 30 05:42:38.283066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:38.282940 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3114624596d1d6e1efef9e22198ab23e7674abbd866811875e5aafc043473600" Oct 30 05:42:44.762400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:44.762350 6587 scope.go:122] "RemoveContainer" containerID="7f7dcf7618a96faedd5d471580855de53a475d4e6671cfbc5a88b174a094a626" Oct 30 05:42:44.772046 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:44.772028 6587 scope.go:122] "RemoveContainer" containerID="064c76ebe104a40ff1f2110648bc69592d450806b91e0fef82449c8e3fba67ed" Oct 30 05:42:44.780652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:44.780629 6587 scope.go:122] "RemoveContainer" containerID="1ee13303596c09f598758d7ec4308a5e878ddb2c60d2588d25636adb13e3ec8b" Oct 30 05:42:44.788711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:44.788692 6587 scope.go:122] "RemoveContainer" containerID="b876d3262a6ed5c3448c399adfaf5d59c49068025d0dc8453f01182e804030f8" Oct 30 05:42:44.796832 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:44.796814 6587 scope.go:122] "RemoveContainer" containerID="c1c9328a2f9bbfb6f768cca1923822974bb05d8520f771e752041123508d1974" Oct 30 05:42:44.808125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:42:44.808104 6587 scope.go:122] "RemoveContainer" containerID="b69cdc867d68eb979af67687f366ef701e5ed6d667e35fb44836173862e8acfc" Oct 30 05:42:55.997313 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:42:55.997273 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:43:02.750499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.750465 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb"] Oct 30 05:43:02.751002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.750952 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c7030f58-ff6c-4166-9033-4f97c6f5f2a1" containerName="revoke-keys" Oct 30 05:43:02.751002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.750971 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7030f58-ff6c-4166-9033-4f97c6f5f2a1" containerName="revoke-keys" Oct 30 05:43:02.751156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.751123 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7030f58-ff6c-4166-9033-4f97c6f5f2a1" containerName="revoke-keys" Oct 30 05:43:02.756264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.756245 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" Oct 30 05:43:02.772907 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.772875 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb"] Oct 30 05:43:02.782257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.782233 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:43:02.782257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.782258 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:43:02.783228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.783209 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:43:02.783312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.783287 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:43:02.797828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.797799 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjjh4\" (UniqueName: \"kubernetes.io/projected/daf8aee3-edcd-4277-a648-d4719042ef5e-kube-api-access-gjjh4\") pod \"maas-api-revoke-sub-046e1343-6vswb\" (UID: \"daf8aee3-edcd-4277-a648-d4719042ef5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" Oct 30 05:43:02.798017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.797867 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6vswb\" (UID: \"daf8aee3-edcd-4277-a648-d4719042ef5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" Oct 30 05:43:02.798017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.797993 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6vswb\" (UID: \"daf8aee3-edcd-4277-a648-d4719042ef5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" Oct 30 05:43:02.898831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.898786 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjjh4\" (UniqueName: \"kubernetes.io/projected/daf8aee3-edcd-4277-a648-d4719042ef5e-kube-api-access-gjjh4\") pod \"maas-api-revoke-sub-046e1343-6vswb\" (UID: \"daf8aee3-edcd-4277-a648-d4719042ef5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" Oct 30 05:43:02.899034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.898936 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6vswb\" (UID: \"daf8aee3-edcd-4277-a648-d4719042ef5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" Oct 30 05:43:02.899120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.899063 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6vswb\" (UID: \"daf8aee3-edcd-4277-a648-d4719042ef5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" Oct 30 05:43:02.899780 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.899756 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6vswb\" (UID: \"daf8aee3-edcd-4277-a648-d4719042ef5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" Oct 30 05:43:02.901303 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.901284 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6vswb\" (UID: \"daf8aee3-edcd-4277-a648-d4719042ef5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" Oct 30 05:43:02.918398 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:02.918345 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjjh4\" (UniqueName: \"kubernetes.io/projected/daf8aee3-edcd-4277-a648-d4719042ef5e-kube-api-access-gjjh4\") pod \"maas-api-revoke-sub-046e1343-6vswb\" (UID: \"daf8aee3-edcd-4277-a648-d4719042ef5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" Oct 30 05:43:03.069556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:03.069525 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" Oct 30 05:43:03.365998 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:03.365969 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb"] Oct 30 05:43:03.367509 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:43:03.367480 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf8aee3_edcd_4277_a648_d4719042ef5e.slice/crio-69dd9ddbe8a2f8e5f52e8b8932588b05d8416b487a25fcd5372a4e78de40f8d7 WatchSource:0}: Error finding container 69dd9ddbe8a2f8e5f52e8b8932588b05d8416b487a25fcd5372a4e78de40f8d7: Status 404 returned error can't find the container with id 69dd9ddbe8a2f8e5f52e8b8932588b05d8416b487a25fcd5372a4e78de40f8d7 Oct 30 05:43:03.460341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:03.460302 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" event={"ID":"daf8aee3-edcd-4277-a648-d4719042ef5e","Type":"ContainerStarted","Data":"69dd9ddbe8a2f8e5f52e8b8932588b05d8416b487a25fcd5372a4e78de40f8d7"} Oct 30 05:43:04.467717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:04.467628 6587 generic.go:356] "Generic (PLEG): container finished" podID="daf8aee3-edcd-4277-a648-d4719042ef5e" containerID="554a66e8f51a9168c0dfd1a548a5ef9cd6842f2e98ad64c87815dd79a220e9ca" exitCode=0 Oct 30 05:43:04.467717 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:04.467695 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" event={"ID":"daf8aee3-edcd-4277-a648-d4719042ef5e","Type":"ContainerDied","Data":"554a66e8f51a9168c0dfd1a548a5ef9cd6842f2e98ad64c87815dd79a220e9ca"} Oct 30 05:43:05.618547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:05.618521 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" Oct 30 05:43:05.727059 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:05.726983 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-xks-ca\") pod \"daf8aee3-edcd-4277-a648-d4719042ef5e\" (UID: \"daf8aee3-edcd-4277-a648-d4719042ef5e\") " Oct 30 05:43:05.727059 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:05.727055 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-openshift-ca\") pod \"daf8aee3-edcd-4277-a648-d4719042ef5e\" (UID: \"daf8aee3-edcd-4277-a648-d4719042ef5e\") " Oct 30 05:43:05.727288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:05.727116 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf8aee3-edcd-4277-a648-d4719042ef5e-kube-api-access-gjjh4\" (UniqueName: \"kubernetes.io/projected/daf8aee3-edcd-4277-a648-d4719042ef5e-kube-api-access-gjjh4\") pod \"daf8aee3-edcd-4277-a648-d4719042ef5e\" (UID: \"daf8aee3-edcd-4277-a648-d4719042ef5e\") " Oct 30 05:43:05.727476 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:05.727443 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-openshift-ca" pod "daf8aee3-edcd-4277-a648-d4719042ef5e" (UID: "daf8aee3-edcd-4277-a648-d4719042ef5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:43:05.729191 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:05.729167 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-xks-ca" pod "daf8aee3-edcd-4277-a648-d4719042ef5e" (UID: "daf8aee3-edcd-4277-a648-d4719042ef5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:43:05.729278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:05.729220 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf8aee3-edcd-4277-a648-d4719042ef5e-kube-api-access-gjjh4" pod "daf8aee3-edcd-4277-a648-d4719042ef5e" (UID: "daf8aee3-edcd-4277-a648-d4719042ef5e"). InnerVolumeSpecName "kube-api-access-gjjh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:43:05.828055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:05.828014 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjjh4\" (UniqueName: \"kubernetes.io/projected/daf8aee3-edcd-4277-a648-d4719042ef5e-kube-api-access-gjjh4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:43:05.828055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:05.828047 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:43:05.828055 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:05.828062 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf8aee3-edcd-4277-a648-d4719042ef5e-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:43:06.481897 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:06.481866 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" Oct 30 05:43:06.482088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:06.481899 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb" event={"ID":"daf8aee3-edcd-4277-a648-d4719042ef5e","Type":"ContainerDied","Data":"69dd9ddbe8a2f8e5f52e8b8932588b05d8416b487a25fcd5372a4e78de40f8d7"} Oct 30 05:43:06.482088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:06.481945 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69dd9ddbe8a2f8e5f52e8b8932588b05d8416b487a25fcd5372a4e78de40f8d7" Oct 30 05:43:13.088295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.088252 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz"] Oct 30 05:43:13.089032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.088997 6587 cpu_manager.go:414] "containerMap: removing container" podUID="daf8aee3-edcd-4277-a648-d4719042ef5e" containerName="revoke-keys" Oct 30 05:43:13.089032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.089027 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf8aee3-edcd-4277-a648-d4719042ef5e" containerName="revoke-keys" Oct 30 05:43:13.089251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.089134 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf8aee3-edcd-4277-a648-d4719042ef5e" containerName="revoke-keys" Oct 30 05:43:13.092877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.092856 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" Oct 30 05:43:13.107936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.107911 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:43:13.119826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.119791 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:43:13.122007 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.121978 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz"] Oct 30 05:43:13.163852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.163814 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:43:13.163852 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.163818 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:43:13.206342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.206313 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4wwdz\" (UID: \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" Oct 30 05:43:13.206342 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.206379 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4wwdz\" (UID: \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" Oct 30 05:43:13.206593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.206509 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88s7m\" (UniqueName: \"kubernetes.io/projected/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-kube-api-access-88s7m\") pod \"maas-api-revoke-sub-d3fba056-4wwdz\" (UID: \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" Oct 30 05:43:13.308019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.307985 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4wwdz\" (UID: \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" Oct 30 05:43:13.308019 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.308030 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4wwdz\" (UID: \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" Oct 30 05:43:13.308251 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.308066 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88s7m\" (UniqueName: \"kubernetes.io/projected/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-kube-api-access-88s7m\") pod \"maas-api-revoke-sub-d3fba056-4wwdz\" (UID: \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" Oct 30 05:43:13.308666 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.308639 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4wwdz\" (UID: \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" Oct 30 05:43:13.310549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.310521 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4wwdz\" (UID: \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" Oct 30 05:43:13.328268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.328228 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88s7m\" (UniqueName: \"kubernetes.io/projected/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-kube-api-access-88s7m\") pod \"maas-api-revoke-sub-d3fba056-4wwdz\" (UID: \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" Oct 30 05:43:13.408303 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.408207 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" Oct 30 05:43:13.702931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:13.702901 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz"] Oct 30 05:43:13.704704 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:43:13.704673 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0630b7b_ed66_48bf_9aff_d7fc9ebe29d4.slice/crio-fca9ae46322be22ad137df3d6be608f02e272a1f8ac9b59586544476691358aa WatchSource:0}: Error finding container fca9ae46322be22ad137df3d6be608f02e272a1f8ac9b59586544476691358aa: Status 404 returned error can't find the container with id fca9ae46322be22ad137df3d6be608f02e272a1f8ac9b59586544476691358aa Oct 30 05:43:14.531851 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:14.531804 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" event={"ID":"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4","Type":"ContainerStarted","Data":"74527878621b75a8c13942bc50345a845c0bf92f0846c3ffc20d0f55ef9c0482"} Oct 30 05:43:14.532387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:14.531876 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" event={"ID":"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4","Type":"ContainerStarted","Data":"fca9ae46322be22ad137df3d6be608f02e272a1f8ac9b59586544476691358aa"} Oct 30 05:43:15.604641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:15.604058 6587 generic.go:356] "Generic (PLEG): container finished" podID="b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4" containerID="74527878621b75a8c13942bc50345a845c0bf92f0846c3ffc20d0f55ef9c0482" exitCode=0 Oct 30 05:43:15.604641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:15.604400 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" event={"ID":"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4","Type":"ContainerDied","Data":"74527878621b75a8c13942bc50345a845c0bf92f0846c3ffc20d0f55ef9c0482"} Oct 30 05:43:16.859010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:16.858984 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" Oct 30 05:43:17.008446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:17.008340 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-kube-api-access-88s7m\" (UniqueName: \"kubernetes.io/projected/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-kube-api-access-88s7m\") pod \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\" (UID: \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\") " Oct 30 05:43:17.008446 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:17.008427 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-openshift-ca\") pod \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\" (UID: \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\") " Oct 30 05:43:17.008704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:17.008529 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-xks-ca\") pod \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\" (UID: \"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4\") " Oct 30 05:43:17.008894 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:17.008864 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-openshift-ca" pod "b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4" (UID: "b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:43:17.010565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:17.010534 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-xks-ca" pod "b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4" (UID: "b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:43:17.010689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:17.010661 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-kube-api-access-88s7m" pod "b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4" (UID: "b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4"). InnerVolumeSpecName "kube-api-access-88s7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:43:17.109796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:17.109763 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88s7m\" (UniqueName: \"kubernetes.io/projected/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-kube-api-access-88s7m\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:43:17.109796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:17.109794 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:43:17.110180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:17.109809 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:43:17.621848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:17.621808 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" event={"ID":"b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4","Type":"ContainerDied","Data":"fca9ae46322be22ad137df3d6be608f02e272a1f8ac9b59586544476691358aa"} Oct 30 05:43:17.622037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:17.621869 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fca9ae46322be22ad137df3d6be608f02e272a1f8ac9b59586544476691358aa" Oct 30 05:43:17.622037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:17.621877 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz" Oct 30 05:43:25.985410 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:43:25.985348 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:43:35.207992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.207925 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr"] Oct 30 05:43:35.208478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.208395 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4" containerName="revoke-keys" Oct 30 05:43:35.208478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.208409 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4" containerName="revoke-keys" Oct 30 05:43:35.208573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.208505 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4" containerName="revoke-keys" Oct 30 05:43:35.211746 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.211730 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" Oct 30 05:43:35.229135 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.229108 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr"] Oct 30 05:43:35.234134 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.234105 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-whddr\" (UID: \"5e3e63a2-bef9-492a-939d-37d408ddf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" Oct 30 05:43:35.234280 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.234233 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-whddr\" (UID: \"5e3e63a2-bef9-492a-939d-37d408ddf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" Oct 30 05:43:35.234370 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.234281 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpfd\" (UniqueName: \"kubernetes.io/projected/5e3e63a2-bef9-492a-939d-37d408ddf38b-kube-api-access-kbpfd\") pod \"maas-api-revoke-sub-ae594979-whddr\" (UID: \"5e3e63a2-bef9-492a-939d-37d408ddf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" Oct 30 05:43:35.238006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.237983 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:43:35.238125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.238084 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:43:35.238125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.238117 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:43:35.238253 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.238118 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:43:35.335228 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.335199 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpfd\" (UniqueName: \"kubernetes.io/projected/5e3e63a2-bef9-492a-939d-37d408ddf38b-kube-api-access-kbpfd\") pod \"maas-api-revoke-sub-ae594979-whddr\" (UID: \"5e3e63a2-bef9-492a-939d-37d408ddf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" Oct 30 05:43:35.335442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.335275 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-whddr\" (UID: \"5e3e63a2-bef9-492a-939d-37d408ddf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" Oct 30 05:43:35.335442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.335321 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-whddr\" (UID: \"5e3e63a2-bef9-492a-939d-37d408ddf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" Oct 30 05:43:35.336069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.336046 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-whddr\" (UID: \"5e3e63a2-bef9-492a-939d-37d408ddf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" Oct 30 05:43:35.337670 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.337647 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-whddr\" (UID: \"5e3e63a2-bef9-492a-939d-37d408ddf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" Oct 30 05:43:35.353990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.353961 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpfd\" (UniqueName: \"kubernetes.io/projected/5e3e63a2-bef9-492a-939d-37d408ddf38b-kube-api-access-kbpfd\") pod \"maas-api-revoke-sub-ae594979-whddr\" (UID: \"5e3e63a2-bef9-492a-939d-37d408ddf38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" Oct 30 05:43:35.525644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.525559 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" Oct 30 05:43:35.821319 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:35.821297 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr"] Oct 30 05:43:35.822967 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:43:35.822940 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3e63a2_bef9_492a_939d_37d408ddf38b.slice/crio-4fa5c7971da58bfcf1a244bf6c5a3d7ad439519d444743a329279507ba573071 WatchSource:0}: Error finding container 4fa5c7971da58bfcf1a244bf6c5a3d7ad439519d444743a329279507ba573071: Status 404 returned error can't find the container with id 4fa5c7971da58bfcf1a244bf6c5a3d7ad439519d444743a329279507ba573071 Oct 30 05:43:36.028198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:36.028159 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf"] Oct 30 05:43:36.030103 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:36.030077 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb45835-8bnqf"] Oct 30 05:43:36.032235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:36.032212 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5"] Oct 30 05:43:36.040403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:36.040381 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9276e572-9cwg5"] Oct 30 05:43:36.735028 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:36.734999 6587 generic.go:356] "Generic (PLEG): container finished" podID="5e3e63a2-bef9-492a-939d-37d408ddf38b" containerID="a9a543f423977d7f7e225a9aff4624e2c317902b0584dc6b78606b876158cd27" exitCode=0 Oct 30 05:43:36.735383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:36.735065 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" event={"ID":"5e3e63a2-bef9-492a-939d-37d408ddf38b","Type":"ContainerDied","Data":"a9a543f423977d7f7e225a9aff4624e2c317902b0584dc6b78606b876158cd27"} Oct 30 05:43:36.735383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:36.735101 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" event={"ID":"5e3e63a2-bef9-492a-939d-37d408ddf38b","Type":"ContainerStarted","Data":"4fa5c7971da58bfcf1a244bf6c5a3d7ad439519d444743a329279507ba573071"} Oct 30 05:43:37.579410 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:37.579376 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2deccbff-8088-44e8-9c67-10f1165213b5" path="/var/lib/kubelet/pods/2deccbff-8088-44e8-9c67-10f1165213b5/volumes" Oct 30 05:43:37.579814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:37.579801 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f59094d-26e4-486b-b338-920802b35481" path="/var/lib/kubelet/pods/3f59094d-26e4-486b-b338-920802b35481/volumes" Oct 30 05:43:37.873451 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:37.873427 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" Oct 30 05:43:37.960314 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:37.960280 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e3e63a2-bef9-492a-939d-37d408ddf38b-kube-api-access-kbpfd\" (UniqueName: \"kubernetes.io/projected/5e3e63a2-bef9-492a-939d-37d408ddf38b-kube-api-access-kbpfd\") pod \"5e3e63a2-bef9-492a-939d-37d408ddf38b\" (UID: \"5e3e63a2-bef9-492a-939d-37d408ddf38b\") " Oct 30 05:43:37.960520 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:37.960421 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-xks-ca\") pod \"5e3e63a2-bef9-492a-939d-37d408ddf38b\" (UID: \"5e3e63a2-bef9-492a-939d-37d408ddf38b\") " Oct 30 05:43:37.960589 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:37.960558 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-openshift-ca\") pod \"5e3e63a2-bef9-492a-939d-37d408ddf38b\" (UID: \"5e3e63a2-bef9-492a-939d-37d408ddf38b\") " Oct 30 05:43:37.960927 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:37.960899 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-openshift-ca" pod "5e3e63a2-bef9-492a-939d-37d408ddf38b" (UID: "5e3e63a2-bef9-492a-939d-37d408ddf38b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:43:37.962332 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:37.962309 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3e63a2-bef9-492a-939d-37d408ddf38b-kube-api-access-kbpfd" pod "5e3e63a2-bef9-492a-939d-37d408ddf38b" (UID: "5e3e63a2-bef9-492a-939d-37d408ddf38b"). InnerVolumeSpecName "kube-api-access-kbpfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:43:37.962454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:37.962423 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-xks-ca" pod "5e3e63a2-bef9-492a-939d-37d408ddf38b" (UID: "5e3e63a2-bef9-492a-939d-37d408ddf38b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:43:38.061547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:38.061514 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:43:38.061547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:38.061543 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbpfd\" (UniqueName: \"kubernetes.io/projected/5e3e63a2-bef9-492a-939d-37d408ddf38b-kube-api-access-kbpfd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:43:38.061736 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:38.061553 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3e63a2-bef9-492a-939d-37d408ddf38b-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:43:38.754234 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:38.754199 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" Oct 30 05:43:38.754459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:38.754229 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr" event={"ID":"5e3e63a2-bef9-492a-939d-37d408ddf38b","Type":"ContainerDied","Data":"4fa5c7971da58bfcf1a244bf6c5a3d7ad439519d444743a329279507ba573071"} Oct 30 05:43:38.754459 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:38.754311 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fa5c7971da58bfcf1a244bf6c5a3d7ad439519d444743a329279507ba573071" Oct 30 05:43:44.847681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:44.847648 6587 scope.go:122] "RemoveContainer" containerID="25ed54c60798cc140d0ce5e9e8145c8dbdecbd08d128762c7ac2142b2d50b17c" Oct 30 05:43:44.856797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:44.856779 6587 scope.go:122] "RemoveContainer" containerID="77b40e88bc8985e5218ac1fa5dd5d5aee644bc8162e56472d1e0793017fc6157" Oct 30 05:43:44.868095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:44.868073 6587 scope.go:122] "RemoveContainer" containerID="b7ab73ae8f17fc4d39b7cb872de8a046de4c72fbf9d04d12a3aea10beecc2dc8" Oct 30 05:43:44.876258 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:44.876237 6587 scope.go:122] "RemoveContainer" containerID="55e8a0277be3b5edbc785ed0a740a8be4e15f5e440cf246d22f8a34a4beb0298" Oct 30 05:43:53.837970 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:53.837931 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc"] Oct 30 05:43:53.838579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:53.838548 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5e3e63a2-bef9-492a-939d-37d408ddf38b" containerName="revoke-keys" Oct 30 05:43:53.838579 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:53.838575 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3e63a2-bef9-492a-939d-37d408ddf38b" containerName="revoke-keys" Oct 30 05:43:53.838841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:53.838742 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3e63a2-bef9-492a-939d-37d408ddf38b" containerName="revoke-keys" Oct 30 05:43:53.843528 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:53.843511 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" Oct 30 05:43:53.861261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:53.861222 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc"] Oct 30 05:43:53.872203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:53.872173 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:43:53.872203 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:53.872171 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:43:53.872511 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:53.872182 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:43:53.874609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:53.874590 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:43:53.931312 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:53.931273 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v2qt\" (UniqueName: \"kubernetes.io/projected/7c447b67-fa0e-415d-a412-54ac74499dbe-kube-api-access-5v2qt\") pod \"maas-api-revoke-sub-a4aef7d7-8lpbc\" (UID: \"7c447b67-fa0e-415d-a412-54ac74499dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" Oct 30 05:43:53.931531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:53.931336 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8lpbc\" (UID: \"7c447b67-fa0e-415d-a412-54ac74499dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" Oct 30 05:43:53.931531 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:53.931473 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8lpbc\" (UID: \"7c447b67-fa0e-415d-a412-54ac74499dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" Oct 30 05:43:54.032180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:54.032138 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v2qt\" (UniqueName: \"kubernetes.io/projected/7c447b67-fa0e-415d-a412-54ac74499dbe-kube-api-access-5v2qt\") pod \"maas-api-revoke-sub-a4aef7d7-8lpbc\" (UID: \"7c447b67-fa0e-415d-a412-54ac74499dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" Oct 30 05:43:54.032404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:54.032205 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8lpbc\" (UID: \"7c447b67-fa0e-415d-a412-54ac74499dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" Oct 30 05:43:54.032404 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:54.032257 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8lpbc\" (UID: \"7c447b67-fa0e-415d-a412-54ac74499dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" Oct 30 05:43:54.033169 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:54.033133 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8lpbc\" (UID: \"7c447b67-fa0e-415d-a412-54ac74499dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" Oct 30 05:43:54.034861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:54.034835 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8lpbc\" (UID: \"7c447b67-fa0e-415d-a412-54ac74499dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" Oct 30 05:43:54.053981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:54.053949 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v2qt\" (UniqueName: \"kubernetes.io/projected/7c447b67-fa0e-415d-a412-54ac74499dbe-kube-api-access-5v2qt\") pod \"maas-api-revoke-sub-a4aef7d7-8lpbc\" (UID: \"7c447b67-fa0e-415d-a412-54ac74499dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" Oct 30 05:43:54.159989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:54.159902 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" Oct 30 05:43:54.464688 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:43:54.464654 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c447b67_fa0e_415d_a412_54ac74499dbe.slice/crio-1279661db0212d5d4feb31306e927cf4f6cd3c152ca63c22a3e8208287b72bac WatchSource:0}: Error finding container 1279661db0212d5d4feb31306e927cf4f6cd3c152ca63c22a3e8208287b72bac: Status 404 returned error can't find the container with id 1279661db0212d5d4feb31306e927cf4f6cd3c152ca63c22a3e8208287b72bac Oct 30 05:43:54.464842 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:54.464788 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc"] Oct 30 05:43:54.860190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:54.860150 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" event={"ID":"7c447b67-fa0e-415d-a412-54ac74499dbe","Type":"ContainerStarted","Data":"2767906cf4389c89b3f60d301152bfb696b76ba32de9f5d27bc06d93a0bdce0c"} Oct 30 05:43:54.860602 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:54.860217 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" event={"ID":"7c447b67-fa0e-415d-a412-54ac74499dbe","Type":"ContainerStarted","Data":"1279661db0212d5d4feb31306e927cf4f6cd3c152ca63c22a3e8208287b72bac"} Oct 30 05:43:54.909209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:54.909149 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" podStartSLOduration=1.909135486 podStartE2EDuration="1.909135486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:54.908291566 +0000 UTC m=+3559.436058351" watchObservedRunningTime="2026-10-30 05:43:54.909135486 +0000 UTC m=+3559.436902271" Oct 30 05:43:55.867507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:55.867476 6587 generic.go:356] "Generic (PLEG): container finished" podID="7c447b67-fa0e-415d-a412-54ac74499dbe" containerID="2767906cf4389c89b3f60d301152bfb696b76ba32de9f5d27bc06d93a0bdce0c" exitCode=0 Oct 30 05:43:55.867910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:55.867569 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" event={"ID":"7c447b67-fa0e-415d-a412-54ac74499dbe","Type":"ContainerDied","Data":"2767906cf4389c89b3f60d301152bfb696b76ba32de9f5d27bc06d93a0bdce0c"} Oct 30 05:43:55.984899 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:43:55.984851 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:43:57.009112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:57.009087 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" Oct 30 05:43:57.063002 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:57.062974 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-xks-ca\") pod \"7c447b67-fa0e-415d-a412-54ac74499dbe\" (UID: \"7c447b67-fa0e-415d-a412-54ac74499dbe\") " Oct 30 05:43:57.063146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:57.063080 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-openshift-ca\") pod \"7c447b67-fa0e-415d-a412-54ac74499dbe\" (UID: \"7c447b67-fa0e-415d-a412-54ac74499dbe\") " Oct 30 05:43:57.063146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:57.063132 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c447b67-fa0e-415d-a412-54ac74499dbe-kube-api-access-5v2qt\" (UniqueName: \"kubernetes.io/projected/7c447b67-fa0e-415d-a412-54ac74499dbe-kube-api-access-5v2qt\") pod \"7c447b67-fa0e-415d-a412-54ac74499dbe\" (UID: \"7c447b67-fa0e-415d-a412-54ac74499dbe\") " Oct 30 05:43:57.063460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:57.063431 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-openshift-ca" pod "7c447b67-fa0e-415d-a412-54ac74499dbe" (UID: "7c447b67-fa0e-415d-a412-54ac74499dbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:43:57.065146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:57.065117 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-xks-ca" pod "7c447b67-fa0e-415d-a412-54ac74499dbe" (UID: "7c447b67-fa0e-415d-a412-54ac74499dbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:43:57.065266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:57.065206 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c447b67-fa0e-415d-a412-54ac74499dbe-kube-api-access-5v2qt" pod "7c447b67-fa0e-415d-a412-54ac74499dbe" (UID: "7c447b67-fa0e-415d-a412-54ac74499dbe"). InnerVolumeSpecName "kube-api-access-5v2qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:43:57.164950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:57.164881 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v2qt\" (UniqueName: \"kubernetes.io/projected/7c447b67-fa0e-415d-a412-54ac74499dbe-kube-api-access-5v2qt\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:43:57.164950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:57.164909 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:43:57.164950 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:57.164919 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c447b67-fa0e-415d-a412-54ac74499dbe-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:43:57.879873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:57.879790 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" Oct 30 05:43:57.879873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:57.879824 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc" event={"ID":"7c447b67-fa0e-415d-a412-54ac74499dbe","Type":"ContainerDied","Data":"1279661db0212d5d4feb31306e927cf4f6cd3c152ca63c22a3e8208287b72bac"} Oct 30 05:43:57.879873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:57.879873 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1279661db0212d5d4feb31306e927cf4f6cd3c152ca63c22a3e8208287b72bac" Oct 30 05:43:59.030137 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:59.030090 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7"] Oct 30 05:43:59.043978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:59.043942 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f9e1c2-6rst7"] Oct 30 05:43:59.579693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:43:59.579658 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d677d710-645b-40d3-9222-24ca3d7047e5" path="/var/lib/kubelet/pods/d677d710-645b-40d3-9222-24ca3d7047e5/volumes" Oct 30 05:44:00.038499 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:00.038407 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8"] Oct 30 05:44:00.051644 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:00.051605 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24bb6c74-5bkx8"] Oct 30 05:44:01.579151 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:01.579118 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730e3261-6722-43fd-8509-19129cbe469f" path="/var/lib/kubelet/pods/730e3261-6722-43fd-8509-19129cbe469f/volumes" Oct 30 05:44:15.130032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.129989 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg"] Oct 30 05:44:15.130566 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.130411 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7c447b67-fa0e-415d-a412-54ac74499dbe" containerName="revoke-keys" Oct 30 05:44:15.130566 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.130423 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c447b67-fa0e-415d-a412-54ac74499dbe" containerName="revoke-keys" Oct 30 05:44:15.130566 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.130515 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c447b67-fa0e-415d-a412-54ac74499dbe" containerName="revoke-keys" Oct 30 05:44:15.133662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.133643 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" Oct 30 05:44:15.164655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.164605 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:44:15.164655 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.164628 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:44:15.164910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.164729 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:44:15.164910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.164792 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:44:15.166859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.166834 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg"] Oct 30 05:44:15.257516 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.257482 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hwkcg\" (UID: \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" Oct 30 05:44:15.257700 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.257624 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddkt6\" (UniqueName: \"kubernetes.io/projected/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-kube-api-access-ddkt6\") pod \"maas-api-revoke-sub-79805ca9-hwkcg\" (UID: \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" Oct 30 05:44:15.257753 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.257697 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hwkcg\" (UID: \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" Oct 30 05:44:15.358792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.358751 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hwkcg\" (UID: \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" Oct 30 05:44:15.358978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.358850 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddkt6\" (UniqueName: \"kubernetes.io/projected/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-kube-api-access-ddkt6\") pod \"maas-api-revoke-sub-79805ca9-hwkcg\" (UID: \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" Oct 30 05:44:15.358978 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.358880 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hwkcg\" (UID: \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" Oct 30 05:44:15.359587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.359557 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hwkcg\" (UID: \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" Oct 30 05:44:15.361098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.361079 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hwkcg\" (UID: \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" Oct 30 05:44:15.380981 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.380923 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddkt6\" (UniqueName: \"kubernetes.io/projected/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-kube-api-access-ddkt6\") pod \"maas-api-revoke-sub-79805ca9-hwkcg\" (UID: \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" Oct 30 05:44:15.446999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.446963 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" Oct 30 05:44:15.743707 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:44:15.743673 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5915e19d_7f5f_43eb_bcc3_95f18e0f1ee1.slice/crio-4bf4f73bf727e012eae757679bcc25160f3529cbf89eb63fbd1894d601c54f6d WatchSource:0}: Error finding container 4bf4f73bf727e012eae757679bcc25160f3529cbf89eb63fbd1894d601c54f6d: Status 404 returned error can't find the container with id 4bf4f73bf727e012eae757679bcc25160f3529cbf89eb63fbd1894d601c54f6d Oct 30 05:44:15.744174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:15.744149 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg"] Oct 30 05:44:16.013803 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:16.013722 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" event={"ID":"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1","Type":"ContainerStarted","Data":"4bf4f73bf727e012eae757679bcc25160f3529cbf89eb63fbd1894d601c54f6d"} Oct 30 05:44:17.021620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:17.021526 6587 generic.go:356] "Generic (PLEG): container finished" podID="5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1" containerID="5894c90154868d8ac5673c036a163fc7da8046080604d138eb73732ad12eb44f" exitCode=0 Oct 30 05:44:17.021620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:17.021609 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" event={"ID":"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1","Type":"ContainerDied","Data":"5894c90154868d8ac5673c036a163fc7da8046080604d138eb73732ad12eb44f"} Oct 30 05:44:18.163402 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:18.163378 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" Oct 30 05:44:18.293797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:18.293710 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-xks-ca\") pod \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\" (UID: \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\") " Oct 30 05:44:18.293960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:18.293836 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-kube-api-access-ddkt6\" (UniqueName: \"kubernetes.io/projected/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-kube-api-access-ddkt6\") pod \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\" (UID: \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\") " Oct 30 05:44:18.293960 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:18.293889 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-openshift-ca\") pod \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\" (UID: \"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1\") " Oct 30 05:44:18.294263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:18.294235 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-openshift-ca" pod "5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1" (UID: "5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:44:18.295861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:18.295837 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-kube-api-access-ddkt6" pod "5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1" (UID: "5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1"). InnerVolumeSpecName "kube-api-access-ddkt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:44:18.295984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:18.295865 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-xks-ca" pod "5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1" (UID: "5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:44:18.395262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:18.395211 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddkt6\" (UniqueName: \"kubernetes.io/projected/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-kube-api-access-ddkt6\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:44:18.395262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:18.395253 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:44:18.395262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:18.395267 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:44:19.034591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:19.034559 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" Oct 30 05:44:19.034781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:19.034590 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg" event={"ID":"5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1","Type":"ContainerDied","Data":"4bf4f73bf727e012eae757679bcc25160f3529cbf89eb63fbd1894d601c54f6d"} Oct 30 05:44:19.034781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:19.034636 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bf4f73bf727e012eae757679bcc25160f3529cbf89eb63fbd1894d601c54f6d" Oct 30 05:44:25.985410 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:44:25.985345 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:44:29.738507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:29.738459 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:44:29.739180 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:29.739141 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" podUID="b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:44:29.872518 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:29.872476 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:44:29.872883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:29.872846 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" podUID="35dc1c80-11e5-4647-b911-bf84b7e3336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:44:33.056768 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.056727 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc"] Oct 30 05:44:33.057384 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.057349 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1" containerName="revoke-keys" Oct 30 05:44:33.057496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.057386 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1" containerName="revoke-keys" Oct 30 05:44:33.057568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.057522 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1" containerName="revoke-keys" Oct 30 05:44:33.061230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.061209 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" Oct 30 05:44:33.080825 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.080800 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc"] Oct 30 05:44:33.091707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.091670 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:44:33.091707 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.091681 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:44:33.091900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.091705 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:44:33.091900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.091747 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:44:33.152210 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.152179 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tfgtc\" (UID: \"5a97a66e-144b-43de-8f61-481d5406fa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" Oct 30 05:44:33.152401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.152222 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9xd4\" (UniqueName: \"kubernetes.io/projected/5a97a66e-144b-43de-8f61-481d5406fa42-kube-api-access-z9xd4\") pod \"maas-api-revoke-sub-0b7e8edb-tfgtc\" (UID: \"5a97a66e-144b-43de-8f61-481d5406fa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" Oct 30 05:44:33.152401 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.152239 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tfgtc\" (UID: \"5a97a66e-144b-43de-8f61-481d5406fa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" Oct 30 05:44:33.253375 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.253320 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tfgtc\" (UID: \"5a97a66e-144b-43de-8f61-481d5406fa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" Oct 30 05:44:33.253570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.253390 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9xd4\" (UniqueName: \"kubernetes.io/projected/5a97a66e-144b-43de-8f61-481d5406fa42-kube-api-access-z9xd4\") pod \"maas-api-revoke-sub-0b7e8edb-tfgtc\" (UID: \"5a97a66e-144b-43de-8f61-481d5406fa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" Oct 30 05:44:33.253570 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.253417 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tfgtc\" (UID: \"5a97a66e-144b-43de-8f61-481d5406fa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" Oct 30 05:44:33.253990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.253955 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tfgtc\" (UID: \"5a97a66e-144b-43de-8f61-481d5406fa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" Oct 30 05:44:33.255773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.255748 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tfgtc\" (UID: \"5a97a66e-144b-43de-8f61-481d5406fa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" Oct 30 05:44:33.275351 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.275327 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9xd4\" (UniqueName: \"kubernetes.io/projected/5a97a66e-144b-43de-8f61-481d5406fa42-kube-api-access-z9xd4\") pod \"maas-api-revoke-sub-0b7e8edb-tfgtc\" (UID: \"5a97a66e-144b-43de-8f61-481d5406fa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" Oct 30 05:44:33.379290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.379212 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" Oct 30 05:44:33.676258 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:44:33.676226 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a97a66e_144b_43de_8f61_481d5406fa42.slice/crio-7a9ec6508111479b7d226f76f5e277d2f407b5342507258805da3777bca7c44e WatchSource:0}: Error finding container 7a9ec6508111479b7d226f76f5e277d2f407b5342507258805da3777bca7c44e: Status 404 returned error can't find the container with id 7a9ec6508111479b7d226f76f5e277d2f407b5342507258805da3777bca7c44e Oct 30 05:44:33.678094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:33.678068 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc"] Oct 30 05:44:34.135307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:34.135262 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" event={"ID":"5a97a66e-144b-43de-8f61-481d5406fa42","Type":"ContainerStarted","Data":"7402fafd32af299b21db77879ea4d00e1802ca5d4139d7c00dc6798a551b0d1d"} Oct 30 05:44:34.135737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:34.135331 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" event={"ID":"5a97a66e-144b-43de-8f61-481d5406fa42","Type":"ContainerStarted","Data":"7a9ec6508111479b7d226f76f5e277d2f407b5342507258805da3777bca7c44e"} Oct 30 05:44:34.178920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:34.178859 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" podStartSLOduration=1.178845363 podStartE2EDuration="1.178845363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:34.17685405 +0000 UTC m=+3598.704620849" watchObservedRunningTime="2026-10-30 05:44:34.178845363 +0000 UTC m=+3598.706612148" Oct 30 05:44:35.143222 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:35.143179 6587 generic.go:356] "Generic (PLEG): container finished" podID="5a97a66e-144b-43de-8f61-481d5406fa42" containerID="7402fafd32af299b21db77879ea4d00e1802ca5d4139d7c00dc6798a551b0d1d" exitCode=0 Oct 30 05:44:35.143669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:35.143219 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" event={"ID":"5a97a66e-144b-43de-8f61-481d5406fa42","Type":"ContainerDied","Data":"7402fafd32af299b21db77879ea4d00e1802ca5d4139d7c00dc6798a551b0d1d"} Oct 30 05:44:36.283462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:36.283437 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" Oct 30 05:44:36.380618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:36.380592 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:44:36.381341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:36.381317 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:44:36.383979 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:36.383952 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-openshift-ca\") pod \"5a97a66e-144b-43de-8f61-481d5406fa42\" (UID: \"5a97a66e-144b-43de-8f61-481d5406fa42\") " Oct 30 05:44:36.384098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:36.384041 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a97a66e-144b-43de-8f61-481d5406fa42-kube-api-access-z9xd4\" (UniqueName: \"kubernetes.io/projected/5a97a66e-144b-43de-8f61-481d5406fa42-kube-api-access-z9xd4\") pod \"5a97a66e-144b-43de-8f61-481d5406fa42\" (UID: \"5a97a66e-144b-43de-8f61-481d5406fa42\") " Oct 30 05:44:36.384098 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:36.384075 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-xks-ca\") pod \"5a97a66e-144b-43de-8f61-481d5406fa42\" (UID: \"5a97a66e-144b-43de-8f61-481d5406fa42\") " Oct 30 05:44:36.384391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:36.384349 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-openshift-ca" pod "5a97a66e-144b-43de-8f61-481d5406fa42" (UID: "5a97a66e-144b-43de-8f61-481d5406fa42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:44:36.386548 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:36.386515 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-xks-ca" pod "5a97a66e-144b-43de-8f61-481d5406fa42" (UID: "5a97a66e-144b-43de-8f61-481d5406fa42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:44:36.386673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:36.386593 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a97a66e-144b-43de-8f61-481d5406fa42-kube-api-access-z9xd4" pod "5a97a66e-144b-43de-8f61-481d5406fa42" (UID: "5a97a66e-144b-43de-8f61-481d5406fa42"). InnerVolumeSpecName "kube-api-access-z9xd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:44:36.485947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:36.485870 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:44:36.485947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:36.485895 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9xd4\" (UniqueName: \"kubernetes.io/projected/5a97a66e-144b-43de-8f61-481d5406fa42-kube-api-access-z9xd4\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:44:36.485947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:36.485907 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a97a66e-144b-43de-8f61-481d5406fa42-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:44:37.158340 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:37.158302 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" Oct 30 05:44:37.158624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:37.158371 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc" event={"ID":"5a97a66e-144b-43de-8f61-481d5406fa42","Type":"ContainerDied","Data":"7a9ec6508111479b7d226f76f5e277d2f407b5342507258805da3777bca7c44e"} Oct 30 05:44:37.158624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:37.158423 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a9ec6508111479b7d226f76f5e277d2f407b5342507258805da3777bca7c44e" Oct 30 05:44:43.778229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.778189 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25"] Oct 30 05:44:43.778911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.778890 6587 cpu_manager.go:414] "containerMap: removing container" podUID="5a97a66e-144b-43de-8f61-481d5406fa42" containerName="revoke-keys" Oct 30 05:44:43.778965 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.778916 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a97a66e-144b-43de-8f61-481d5406fa42" containerName="revoke-keys" Oct 30 05:44:43.779073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.779060 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a97a66e-144b-43de-8f61-481d5406fa42" containerName="revoke-keys" Oct 30 05:44:43.782676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.782651 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" Oct 30 05:44:43.801857 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.801821 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25"] Oct 30 05:44:43.811813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.811788 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:44:43.811971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.811837 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:44:43.811971 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.811870 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:44:43.813440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.813419 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:44:43.866062 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.866027 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kgd25\" (UID: \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" Oct 30 05:44:43.866224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.866080 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kgd25\" (UID: \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" Oct 30 05:44:43.866224 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.866189 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx886\" (UniqueName: \"kubernetes.io/projected/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-kube-api-access-rx886\") pod \"maas-api-revoke-sub-3d4156f1-kgd25\" (UID: \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" Oct 30 05:44:43.966778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.966731 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kgd25\" (UID: \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" Oct 30 05:44:43.966987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.966833 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx886\" (UniqueName: \"kubernetes.io/projected/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-kube-api-access-rx886\") pod \"maas-api-revoke-sub-3d4156f1-kgd25\" (UID: \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" Oct 30 05:44:43.966987 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.966944 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kgd25\" (UID: \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" Oct 30 05:44:43.967704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.967675 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kgd25\" (UID: \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" Oct 30 05:44:43.969442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.969419 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kgd25\" (UID: \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" Oct 30 05:44:43.988268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:43.988240 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx886\" (UniqueName: \"kubernetes.io/projected/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-kube-api-access-rx886\") pod \"maas-api-revoke-sub-3d4156f1-kgd25\" (UID: \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" Oct 30 05:44:44.095951 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:44.095918 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" Oct 30 05:44:44.395521 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:44:44.395481 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd4d1cc_f1ae_4b37_a7b4_6f3625596b56.slice/crio-0bef7d7ccb5648366c2217f44c879172d676c9a6601e591ffebfadd532e82437 WatchSource:0}: Error finding container 0bef7d7ccb5648366c2217f44c879172d676c9a6601e591ffebfadd532e82437: Status 404 returned error can't find the container with id 0bef7d7ccb5648366c2217f44c879172d676c9a6601e591ffebfadd532e82437 Oct 30 05:44:44.396129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:44.396102 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25"] Oct 30 05:44:44.909006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:44.908972 6587 scope.go:122] "RemoveContainer" containerID="98bd08e4b65b3e15154c0fcff11752a5c87eb87cf21e008989ec8ec5d288014a" Oct 30 05:44:44.918237 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:44.918215 6587 scope.go:122] "RemoveContainer" containerID="6b74a2473060c9141f69741c60eafdeec102a705f6869f756b8bd1adab28438f" Oct 30 05:44:44.927042 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:44.927019 6587 scope.go:122] "RemoveContainer" containerID="254c6480b7931c694268bec3f0f3d2b3b92624f66a634c423cd8e9f5098e0b92" Oct 30 05:44:44.936287 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:44.936267 6587 scope.go:122] "RemoveContainer" containerID="b46ccc8c32421f7397cd9de7d6cdc73c926fcf9d10b004722ad8e24ee159dfe9" Oct 30 05:44:44.946328 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:44.946307 6587 scope.go:122] "RemoveContainer" containerID="520e0f3fc412ba204cf2b53d25bd4bd08ce6e9b5fd921799407f3cc13fd42e6f" Oct 30 05:44:44.956419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:44.956401 6587 scope.go:122] "RemoveContainer" containerID="ca3631566ced55011fb45e0b96c4714618a1b30e989bdc336e2fecd92acea575" Oct 30 05:44:44.966423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:44.966402 6587 scope.go:122] "RemoveContainer" containerID="47bac404da756ec776adb9993488d446ed2e11f453a92eca6bb8b5f4c2efa29d" Oct 30 05:44:45.206036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:45.205998 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" event={"ID":"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56","Type":"ContainerStarted","Data":"99819d997a72972ea8e4a3596fd4e4bb17c771938be2cb1ef1e83e268721fceb"} Oct 30 05:44:45.206271 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:45.206053 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" event={"ID":"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56","Type":"ContainerStarted","Data":"0bef7d7ccb5648366c2217f44c879172d676c9a6601e591ffebfadd532e82437"} Oct 30 05:44:46.213116 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:46.213081 6587 generic.go:356] "Generic (PLEG): container finished" podID="2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56" containerID="99819d997a72972ea8e4a3596fd4e4bb17c771938be2cb1ef1e83e268721fceb" exitCode=0 Oct 30 05:44:46.213566 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:46.213158 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" event={"ID":"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56","Type":"ContainerDied","Data":"99819d997a72972ea8e4a3596fd4e4bb17c771938be2cb1ef1e83e268721fceb"} Oct 30 05:44:47.361125 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:47.361096 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" Oct 30 05:44:47.402856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:47.402828 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-openshift-ca\") pod \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\" (UID: \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\") " Oct 30 05:44:47.403045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:47.402934 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-kube-api-access-rx886\" (UniqueName: \"kubernetes.io/projected/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-kube-api-access-rx886\") pod \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\" (UID: \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\") " Oct 30 05:44:47.403045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:47.403001 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-xks-ca\") pod \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\" (UID: \"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56\") " Oct 30 05:44:47.403221 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:47.403197 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-openshift-ca" pod "2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56" (UID: "2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:44:47.403483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:47.403461 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:44:47.405652 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:47.405622 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-xks-ca" pod "2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56" (UID: "2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:44:47.405797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:47.405654 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-kube-api-access-rx886" pod "2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56" (UID: "2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56"). InnerVolumeSpecName "kube-api-access-rx886". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:44:47.504751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:47.504656 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx886\" (UniqueName: \"kubernetes.io/projected/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-kube-api-access-rx886\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:44:47.504751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:47.504692 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:44:48.225826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:48.225794 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" Oct 30 05:44:48.226045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:48.225824 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25" event={"ID":"2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56","Type":"ContainerDied","Data":"0bef7d7ccb5648366c2217f44c879172d676c9a6601e591ffebfadd532e82437"} Oct 30 05:44:48.226045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:44:48.225877 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bef7d7ccb5648366c2217f44c879172d676c9a6601e591ffebfadd532e82437" Oct 30 05:44:55.986127 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:44:55.986086 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:45:00.137721 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.137678 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh"] Oct 30 05:45:00.138151 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.138116 6587 cpu_manager.go:414] "containerMap: removing container" podUID="2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56" containerName="revoke-keys" Oct 30 05:45:00.138151 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.138128 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56" containerName="revoke-keys" Oct 30 05:45:00.138250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.138215 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56" containerName="revoke-keys" Oct 30 05:45:00.141309 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.141281 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" Oct 30 05:45:00.164129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.164091 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh"] Oct 30 05:45:00.170812 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.170779 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:45:00.170972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.170919 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:45:00.171085 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.170999 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:45:00.272109 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.272069 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3d20eda-eba0-426f-8041-6dfa3bd4de93-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888985-gzjbh\" (UID: \"c3d20eda-eba0-426f-8041-6dfa3bd4de93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" Oct 30 05:45:00.272326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.272209 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dhjn\" (UniqueName: \"kubernetes.io/projected/c3d20eda-eba0-426f-8041-6dfa3bd4de93-kube-api-access-5dhjn\") pod \"maas-api-key-cleanup-29888985-gzjbh\" (UID: \"c3d20eda-eba0-426f-8041-6dfa3bd4de93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" Oct 30 05:45:00.373568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.373522 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3d20eda-eba0-426f-8041-6dfa3bd4de93-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888985-gzjbh\" (UID: \"c3d20eda-eba0-426f-8041-6dfa3bd4de93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" Oct 30 05:45:00.373767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.373630 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dhjn\" (UniqueName: \"kubernetes.io/projected/c3d20eda-eba0-426f-8041-6dfa3bd4de93-kube-api-access-5dhjn\") pod \"maas-api-key-cleanup-29888985-gzjbh\" (UID: \"c3d20eda-eba0-426f-8041-6dfa3bd4de93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" Oct 30 05:45:00.374151 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.374127 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3d20eda-eba0-426f-8041-6dfa3bd4de93-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888985-gzjbh\" (UID: \"c3d20eda-eba0-426f-8041-6dfa3bd4de93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" Oct 30 05:45:00.395051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.394977 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dhjn\" (UniqueName: \"kubernetes.io/projected/c3d20eda-eba0-426f-8041-6dfa3bd4de93-kube-api-access-5dhjn\") pod \"maas-api-key-cleanup-29888985-gzjbh\" (UID: \"c3d20eda-eba0-426f-8041-6dfa3bd4de93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" Oct 30 05:45:00.456343 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.456297 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" Oct 30 05:45:00.755928 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:45:00.755887 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d20eda_eba0_426f_8041_6dfa3bd4de93.slice/crio-88625c6e590bbc3de8ba9f7cd69cf098b584dc59351d964e644f6c43c5cb2aba WatchSource:0}: Error finding container 88625c6e590bbc3de8ba9f7cd69cf098b584dc59351d964e644f6c43c5cb2aba: Status 404 returned error can't find the container with id 88625c6e590bbc3de8ba9f7cd69cf098b584dc59351d964e644f6c43c5cb2aba Oct 30 05:45:00.756344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:00.756322 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh"] Oct 30 05:45:01.320710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:01.320674 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" event={"ID":"c3d20eda-eba0-426f-8041-6dfa3bd4de93","Type":"ContainerStarted","Data":"9c0b0ccb476fff364d15bbf2c226ece60b11e0f371722482293480a5d96e86f4"} Oct 30 05:45:01.321130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:01.320732 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" event={"ID":"c3d20eda-eba0-426f-8041-6dfa3bd4de93","Type":"ContainerStarted","Data":"88625c6e590bbc3de8ba9f7cd69cf098b584dc59351d964e644f6c43c5cb2aba"} Oct 30 05:45:01.362793 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:01.362735 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" podStartSLOduration=1.36272 podStartE2EDuration="1.36272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:01.361740206 +0000 UTC m=+3625.889506992" watchObservedRunningTime="2026-10-30 05:45:01.36272 +0000 UTC m=+3625.890486819" Oct 30 05:45:02.327787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:02.327751 6587 generic.go:356] "Generic (PLEG): container finished" podID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" containerID="9c0b0ccb476fff364d15bbf2c226ece60b11e0f371722482293480a5d96e86f4" exitCode=60 Oct 30 05:45:02.328308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:02.327809 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" event={"ID":"c3d20eda-eba0-426f-8041-6dfa3bd4de93","Type":"ContainerDied","Data":"9c0b0ccb476fff364d15bbf2c226ece60b11e0f371722482293480a5d96e86f4"} Oct 30 05:45:02.328308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:02.328143 6587 scope.go:122] "RemoveContainer" containerID="9c0b0ccb476fff364d15bbf2c226ece60b11e0f371722482293480a5d96e86f4" Oct 30 05:45:03.038911 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:03.038832 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp"] Oct 30 05:45:03.043208 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:03.043184 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f697f9f9-7q9rp"] Oct 30 05:45:03.334938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:03.334897 6587 generic.go:356] "Generic (PLEG): container finished" podID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" containerID="be25a496afa73b941e9ca1a1aa95d22475e8703a5cb9380051e7a28e30206400" exitCode=60 Oct 30 05:45:03.335381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:03.334980 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" event={"ID":"c3d20eda-eba0-426f-8041-6dfa3bd4de93","Type":"ContainerDied","Data":"be25a496afa73b941e9ca1a1aa95d22475e8703a5cb9380051e7a28e30206400"} Oct 30 05:45:03.335381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:03.335036 6587 scope.go:122] "RemoveContainer" containerID="9c0b0ccb476fff364d15bbf2c226ece60b11e0f371722482293480a5d96e86f4" Oct 30 05:45:03.335381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:03.335278 6587 scope.go:122] "RemoveContainer" containerID="be25a496afa73b941e9ca1a1aa95d22475e8703a5cb9380051e7a28e30206400" Oct 30 05:45:03.335606 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:45:03.335581 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888985-gzjbh_odh-ai-gateway-infra(c3d20eda-eba0-426f-8041-6dfa3bd4de93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" podUID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" Oct 30 05:45:03.579523 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:03.579488 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e28d932-4790-4f6b-880c-338534347104" path="/var/lib/kubelet/pods/6e28d932-4790-4f6b-880c-338534347104/volumes" Oct 30 05:45:04.050789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:04.050751 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25"] Oct 30 05:45:04.064391 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:04.064339 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgd25"] Oct 30 05:45:04.342472 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:04.342395 6587 scope.go:122] "RemoveContainer" containerID="be25a496afa73b941e9ca1a1aa95d22475e8703a5cb9380051e7a28e30206400" Oct 30 05:45:04.342877 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:45:04.342632 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888985-gzjbh_odh-ai-gateway-infra(c3d20eda-eba0-426f-8041-6dfa3bd4de93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" podUID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" Oct 30 05:45:05.579473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:05.579437 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56" path="/var/lib/kubelet/pods/2bd4d1cc-f1ae-4b37-a7b4-6f3625596b56/volumes" Oct 30 05:45:15.577431 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:15.577404 6587 scope.go:122] "RemoveContainer" containerID="be25a496afa73b941e9ca1a1aa95d22475e8703a5cb9380051e7a28e30206400" Oct 30 05:45:16.407995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:16.407950 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" event={"ID":"c3d20eda-eba0-426f-8041-6dfa3bd4de93","Type":"ContainerStarted","Data":"86816a496c70e0626e662fb64670ecdfd845c5678e6e65937ebb364fd20ef756"} Oct 30 05:45:16.574010 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:45:16.573960 6587 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d20eda_eba0_426f_8041_6dfa3bd4de93.slice/crio-conmon-86816a496c70e0626e662fb64670ecdfd845c5678e6e65937ebb364fd20ef756.scope\": RecentStats: unable to find data in memory cache]" Oct 30 05:45:17.415969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:17.415934 6587 generic.go:356] "Generic (PLEG): container finished" podID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" containerID="86816a496c70e0626e662fb64670ecdfd845c5678e6e65937ebb364fd20ef756" exitCode=60 Oct 30 05:45:17.416437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:17.416008 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" event={"ID":"c3d20eda-eba0-426f-8041-6dfa3bd4de93","Type":"ContainerDied","Data":"86816a496c70e0626e662fb64670ecdfd845c5678e6e65937ebb364fd20ef756"} Oct 30 05:45:17.416437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:17.416072 6587 scope.go:122] "RemoveContainer" containerID="be25a496afa73b941e9ca1a1aa95d22475e8703a5cb9380051e7a28e30206400" Oct 30 05:45:17.416437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:17.416425 6587 scope.go:122] "RemoveContainer" containerID="86816a496c70e0626e662fb64670ecdfd845c5678e6e65937ebb364fd20ef756" Oct 30 05:45:17.416726 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:45:17.416707 6587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888985-gzjbh_odh-ai-gateway-infra(c3d20eda-eba0-426f-8041-6dfa3bd4de93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" podUID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" Oct 30 05:45:17.466388 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:17.466342 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh"] Oct 30 05:45:18.574287 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:18.574253 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" Oct 30 05:45:18.768656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:18.768572 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d20eda-eba0-426f-8041-6dfa3bd4de93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3d20eda-eba0-426f-8041-6dfa3bd4de93-maas-api-service-ca\") pod \"c3d20eda-eba0-426f-8041-6dfa3bd4de93\" (UID: \"c3d20eda-eba0-426f-8041-6dfa3bd4de93\") " Oct 30 05:45:18.768656 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:18.768624 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d20eda-eba0-426f-8041-6dfa3bd4de93-kube-api-access-5dhjn\" (UniqueName: \"kubernetes.io/projected/c3d20eda-eba0-426f-8041-6dfa3bd4de93-kube-api-access-5dhjn\") pod \"c3d20eda-eba0-426f-8041-6dfa3bd4de93\" (UID: \"c3d20eda-eba0-426f-8041-6dfa3bd4de93\") " Oct 30 05:45:18.769016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:18.768994 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d20eda-eba0-426f-8041-6dfa3bd4de93-maas-api-service-ca" pod "c3d20eda-eba0-426f-8041-6dfa3bd4de93" (UID: "c3d20eda-eba0-426f-8041-6dfa3bd4de93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:45:18.770829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:18.770797 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d20eda-eba0-426f-8041-6dfa3bd4de93-kube-api-access-5dhjn" pod "c3d20eda-eba0-426f-8041-6dfa3bd4de93" (UID: "c3d20eda-eba0-426f-8041-6dfa3bd4de93"). InnerVolumeSpecName "kube-api-access-5dhjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:45:18.869608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:18.869569 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3d20eda-eba0-426f-8041-6dfa3bd4de93-maas-api-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:45:18.869608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:18.869602 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dhjn\" (UniqueName: \"kubernetes.io/projected/c3d20eda-eba0-426f-8041-6dfa3bd4de93-kube-api-access-5dhjn\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:45:19.431844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:19.431811 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" Oct 30 05:45:19.431844 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:19.431824 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh" event={"ID":"c3d20eda-eba0-426f-8041-6dfa3bd4de93","Type":"ContainerDied","Data":"88625c6e590bbc3de8ba9f7cd69cf098b584dc59351d964e644f6c43c5cb2aba"} Oct 30 05:45:19.432116 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:19.431893 6587 scope.go:122] "RemoveContainer" containerID="86816a496c70e0626e662fb64670ecdfd845c5678e6e65937ebb364fd20ef756" Oct 30 05:45:19.488048 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:19.488020 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh"] Oct 30 05:45:19.501510 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:19.501481 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gzjbh"] Oct 30 05:45:19.579781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:19.579747 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" path="/var/lib/kubelet/pods/c3d20eda-eba0-426f-8041-6dfa3bd4de93/volumes" Oct 30 05:45:25.031323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:25.031285 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26"] Oct 30 05:45:25.044759 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:25.044726 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2bc3343-9lz26"] Oct 30 05:45:25.579846 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:25.579813 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d44fd769-67e4-411a-8a2e-ee50329ec6a0" path="/var/lib/kubelet/pods/d44fd769-67e4-411a-8a2e-ee50329ec6a0/volumes" Oct 30 05:45:25.985463 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:45:25.985380 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:45:26.752337 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.752299 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw"] Oct 30 05:45:26.752810 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.752730 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" containerName="cleanup" Oct 30 05:45:26.752810 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.752743 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" containerName="cleanup" Oct 30 05:45:26.752810 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.752758 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" containerName="cleanup" Oct 30 05:45:26.752810 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.752763 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" containerName="cleanup" Oct 30 05:45:26.753006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.752867 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" containerName="cleanup" Oct 30 05:45:26.753006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.752881 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" containerName="cleanup" Oct 30 05:45:26.757354 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.757334 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" Oct 30 05:45:26.776437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.776412 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw"] Oct 30 05:45:26.787716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.787678 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:45:26.787716 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.787693 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:45:26.787882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.787720 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:45:26.787882 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.787700 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:45:26.854279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.854245 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fskrw\" (UID: \"f423465e-1673-45ef-8992-e3c7f5684e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" Oct 30 05:45:26.854496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.854452 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fskrw\" (UID: \"f423465e-1673-45ef-8992-e3c7f5684e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" Oct 30 05:45:26.854557 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.854493 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtn47\" (UniqueName: \"kubernetes.io/projected/f423465e-1673-45ef-8992-e3c7f5684e87-kube-api-access-dtn47\") pod \"maas-api-revoke-sub-3d4156f1-fskrw\" (UID: \"f423465e-1673-45ef-8992-e3c7f5684e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" Oct 30 05:45:26.955290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.955241 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fskrw\" (UID: \"f423465e-1673-45ef-8992-e3c7f5684e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" Oct 30 05:45:26.955290 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.955295 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtn47\" (UniqueName: \"kubernetes.io/projected/f423465e-1673-45ef-8992-e3c7f5684e87-kube-api-access-dtn47\") pod \"maas-api-revoke-sub-3d4156f1-fskrw\" (UID: \"f423465e-1673-45ef-8992-e3c7f5684e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" Oct 30 05:45:26.955592 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.955332 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fskrw\" (UID: \"f423465e-1673-45ef-8992-e3c7f5684e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" Oct 30 05:45:26.955944 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.955920 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fskrw\" (UID: \"f423465e-1673-45ef-8992-e3c7f5684e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" Oct 30 05:45:26.957702 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.957676 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fskrw\" (UID: \"f423465e-1673-45ef-8992-e3c7f5684e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" Oct 30 05:45:26.976118 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:26.976091 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtn47\" (UniqueName: \"kubernetes.io/projected/f423465e-1673-45ef-8992-e3c7f5684e87-kube-api-access-dtn47\") pod \"maas-api-revoke-sub-3d4156f1-fskrw\" (UID: \"f423465e-1673-45ef-8992-e3c7f5684e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" Oct 30 05:45:27.070445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:27.070414 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" Oct 30 05:45:27.366929 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:45:27.366900 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf423465e_1673_45ef_8992_e3c7f5684e87.slice/crio-f7091ad137672532398bf3738e575f4baafacd0c29c7d379e2b04b1143ee1a00 WatchSource:0}: Error finding container f7091ad137672532398bf3738e575f4baafacd0c29c7d379e2b04b1143ee1a00: Status 404 returned error can't find the container with id f7091ad137672532398bf3738e575f4baafacd0c29c7d379e2b04b1143ee1a00 Oct 30 05:45:27.367381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:27.367334 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw"] Oct 30 05:45:27.489468 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:27.489436 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" event={"ID":"f423465e-1673-45ef-8992-e3c7f5684e87","Type":"ContainerStarted","Data":"f7091ad137672532398bf3738e575f4baafacd0c29c7d379e2b04b1143ee1a00"} Oct 30 05:45:28.497393 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:28.497271 6587 generic.go:356] "Generic (PLEG): container finished" podID="f423465e-1673-45ef-8992-e3c7f5684e87" containerID="7080dd6ba891736110a911d09e27df0943bcadb27dd7319a24278f5db848ba89" exitCode=0 Oct 30 05:45:28.497785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:28.497384 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" event={"ID":"f423465e-1673-45ef-8992-e3c7f5684e87","Type":"ContainerDied","Data":"7080dd6ba891736110a911d09e27df0943bcadb27dd7319a24278f5db848ba89"} Oct 30 05:45:29.635704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:29.635674 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" Oct 30 05:45:29.684073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:29.684038 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-openshift-ca\") pod \"f423465e-1673-45ef-8992-e3c7f5684e87\" (UID: \"f423465e-1673-45ef-8992-e3c7f5684e87\") " Oct 30 05:45:29.684281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:29.684101 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f423465e-1673-45ef-8992-e3c7f5684e87-kube-api-access-dtn47\" (UniqueName: \"kubernetes.io/projected/f423465e-1673-45ef-8992-e3c7f5684e87-kube-api-access-dtn47\") pod \"f423465e-1673-45ef-8992-e3c7f5684e87\" (UID: \"f423465e-1673-45ef-8992-e3c7f5684e87\") " Oct 30 05:45:29.684281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:29.684203 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-xks-ca\") pod \"f423465e-1673-45ef-8992-e3c7f5684e87\" (UID: \"f423465e-1673-45ef-8992-e3c7f5684e87\") " Oct 30 05:45:29.685242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:29.685196 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-openshift-ca" pod "f423465e-1673-45ef-8992-e3c7f5684e87" (UID: "f423465e-1673-45ef-8992-e3c7f5684e87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:45:29.687648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:29.687618 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-xks-ca" pod "f423465e-1673-45ef-8992-e3c7f5684e87" (UID: "f423465e-1673-45ef-8992-e3c7f5684e87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:45:29.691130 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:29.691105 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f423465e-1673-45ef-8992-e3c7f5684e87-kube-api-access-dtn47" pod "f423465e-1673-45ef-8992-e3c7f5684e87" (UID: "f423465e-1673-45ef-8992-e3c7f5684e87"). InnerVolumeSpecName "kube-api-access-dtn47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:45:29.785462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:29.785383 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:45:29.785462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:29.785410 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtn47\" (UniqueName: \"kubernetes.io/projected/f423465e-1673-45ef-8992-e3c7f5684e87-kube-api-access-dtn47\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:45:29.785462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:29.785422 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f423465e-1673-45ef-8992-e3c7f5684e87-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:45:30.516819 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:30.516782 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" event={"ID":"f423465e-1673-45ef-8992-e3c7f5684e87","Type":"ContainerDied","Data":"f7091ad137672532398bf3738e575f4baafacd0c29c7d379e2b04b1143ee1a00"} Oct 30 05:45:30.517014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:30.516862 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7091ad137672532398bf3738e575f4baafacd0c29c7d379e2b04b1143ee1a00" Oct 30 05:45:30.517014 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:30.516791 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw" Oct 30 05:45:45.025649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:45.025610 6587 scope.go:122] "RemoveContainer" containerID="d6db06b8c30720d08fc49ad8eaa1f3869b896bd349864ea5080a312c65863034" Oct 30 05:45:45.036764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:45.036739 6587 scope.go:122] "RemoveContainer" containerID="d16ea14e4a943c3eb5e63620243b581345e681f20fd38a4454a0fa4776a2b5ec" Oct 30 05:45:45.046576 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:45.046555 6587 scope.go:122] "RemoveContainer" containerID="ade5c365169b0ad6ecb2522146adcbe80dacaff3dcf02c25bfd5b0015e4613f2" Oct 30 05:45:45.055815 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:45.055790 6587 scope.go:122] "RemoveContainer" containerID="8d5eb35b07dd0a554b9dea768ea418a2335c9a5ac69c76e89256ef41ee0d1f07" Oct 30 05:45:45.065187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:45.065156 6587 scope.go:122] "RemoveContainer" containerID="e1ee558e4e1a95327e3d12f841420de57018dbdc21dce616b0cf3df12c365a69" Oct 30 05:45:45.073152 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:45.073128 6587 scope.go:122] "RemoveContainer" containerID="6f8c54d932475db788d8c67f91ad904fd9f3502a8ac0c9de1ec6d1dedb43f2e3" Oct 30 05:45:45.080475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:45.080460 6587 scope.go:122] "RemoveContainer" containerID="d3bd2c847f8fd6d76501e7a623f37e303f37c64583986c015c637f3b7c5380e3" Oct 30 05:45:48.007067 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.007031 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj"] Oct 30 05:45:48.007551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.007473 6587 cpu_manager.go:414] "containerMap: removing container" podUID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" containerName="cleanup" Oct 30 05:45:48.007551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.007485 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" containerName="cleanup" Oct 30 05:45:48.007551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.007509 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f423465e-1673-45ef-8992-e3c7f5684e87" containerName="revoke-keys" Oct 30 05:45:48.007551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.007514 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f423465e-1673-45ef-8992-e3c7f5684e87" containerName="revoke-keys" Oct 30 05:45:48.007718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.007583 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d20eda-eba0-426f-8041-6dfa3bd4de93" containerName="cleanup" Oct 30 05:45:48.007718 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.007593 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f423465e-1673-45ef-8992-e3c7f5684e87" containerName="revoke-keys" Oct 30 05:45:48.012953 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.012932 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" Oct 30 05:45:48.032185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.032160 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj"] Oct 30 05:45:48.043313 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.043293 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f97a49\"" Oct 30 05:45:48.043443 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.043430 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-f4041bb3-dockercfg-xv5hv\"" Oct 30 05:45:48.078521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.078479 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx"] Oct 30 05:45:48.082235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.082211 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" Oct 30 05:45:48.104838 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.104805 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx"] Oct 30 05:45:48.179416 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.179369 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jp9v\" (UniqueName: \"kubernetes.io/projected/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-kube-api-access-5jp9v\") pod \"payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj\" (UID: \"54eb51b0-1a0f-4797-9b48-4b31b8eaf783\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" Oct 30 05:45:48.179608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.179448 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9t5j\" (UniqueName: \"kubernetes.io/projected/56105d2f-9c30-4638-8b77-05af2f3c7e39-kube-api-access-h9t5j\") pod \"payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx\" (UID: \"56105d2f-9c30-4638-8b77-05af2f3c7e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" Oct 30 05:45:48.179608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.179510 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj\" (UID: \"54eb51b0-1a0f-4797-9b48-4b31b8eaf783\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" Oct 30 05:45:48.179692 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.179610 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56105d2f-9c30-4638-8b77-05af2f3c7e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx\" (UID: \"56105d2f-9c30-4638-8b77-05af2f3c7e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" Oct 30 05:45:48.280669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.280582 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jp9v\" (UniqueName: \"kubernetes.io/projected/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-kube-api-access-5jp9v\") pod \"payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj\" (UID: \"54eb51b0-1a0f-4797-9b48-4b31b8eaf783\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" Oct 30 05:45:48.280669 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.280628 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9t5j\" (UniqueName: \"kubernetes.io/projected/56105d2f-9c30-4638-8b77-05af2f3c7e39-kube-api-access-h9t5j\") pod \"payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx\" (UID: \"56105d2f-9c30-4638-8b77-05af2f3c7e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" Oct 30 05:45:48.280879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.280682 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj\" (UID: \"54eb51b0-1a0f-4797-9b48-4b31b8eaf783\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" Oct 30 05:45:48.280879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.280747 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56105d2f-9c30-4638-8b77-05af2f3c7e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx\" (UID: \"56105d2f-9c30-4638-8b77-05af2f3c7e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" Oct 30 05:45:48.281536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.281513 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56105d2f-9c30-4638-8b77-05af2f3c7e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx\" (UID: \"56105d2f-9c30-4638-8b77-05af2f3c7e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" Oct 30 05:45:48.281596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.281524 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj\" (UID: \"54eb51b0-1a0f-4797-9b48-4b31b8eaf783\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" Oct 30 05:45:48.303820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.303783 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jp9v\" (UniqueName: \"kubernetes.io/projected/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-kube-api-access-5jp9v\") pod \"payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj\" (UID: \"54eb51b0-1a0f-4797-9b48-4b31b8eaf783\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" Oct 30 05:45:48.303988 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.303870 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9t5j\" (UniqueName: \"kubernetes.io/projected/56105d2f-9c30-4638-8b77-05af2f3c7e39-kube-api-access-h9t5j\") pod \"payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx\" (UID: \"56105d2f-9c30-4638-8b77-05af2f3c7e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" Oct 30 05:45:48.325837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.325807 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" Oct 30 05:45:48.402640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.402605 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" Oct 30 05:45:48.665726 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:45:48.665694 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54eb51b0_1a0f_4797_9b48_4b31b8eaf783.slice/crio-3705aecd637ec4649b97eb288d3efb00fa46479cdfa5ccf2832c411faa986e99 WatchSource:0}: Error finding container 3705aecd637ec4649b97eb288d3efb00fa46479cdfa5ccf2832c411faa986e99: Status 404 returned error can't find the container with id 3705aecd637ec4649b97eb288d3efb00fa46479cdfa5ccf2832c411faa986e99 Oct 30 05:45:48.678848 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.678821 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj"] Oct 30 05:45:48.720058 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:45:48.720022 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56105d2f_9c30_4638_8b77_05af2f3c7e39.slice/crio-d066145ec94403ad14143c9a8c450d56a6a970571826439c33a9ca0e4fc95f3e WatchSource:0}: Error finding container d066145ec94403ad14143c9a8c450d56a6a970571826439c33a9ca0e4fc95f3e: Status 404 returned error can't find the container with id d066145ec94403ad14143c9a8c450d56a6a970571826439c33a9ca0e4fc95f3e Oct 30 05:45:48.721909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:48.721821 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx"] Oct 30 05:45:49.623077 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:49.623035 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" event={"ID":"54eb51b0-1a0f-4797-9b48-4b31b8eaf783","Type":"ContainerStarted","Data":"7b3218343c7cb3a03752471af279bbdb26c47b94bb973593009c33683322179b"} Oct 30 05:45:49.623662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:49.623100 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" event={"ID":"54eb51b0-1a0f-4797-9b48-4b31b8eaf783","Type":"ContainerStarted","Data":"3705aecd637ec4649b97eb288d3efb00fa46479cdfa5ccf2832c411faa986e99"} Oct 30 05:45:49.623662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:49.623215 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" Oct 30 05:45:49.624526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:49.624496 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" event={"ID":"56105d2f-9c30-4638-8b77-05af2f3c7e39","Type":"ContainerStarted","Data":"4f8db0246277bd58add046b6d95c45a25cd43e8efa3406dd6e3aecfb35607481"} Oct 30 05:45:49.624658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:49.624548 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" event={"ID":"56105d2f-9c30-4638-8b77-05af2f3c7e39","Type":"ContainerStarted","Data":"d066145ec94403ad14143c9a8c450d56a6a970571826439c33a9ca0e4fc95f3e"} Oct 30 05:45:49.624658 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:49.624577 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" Oct 30 05:45:49.666699 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:49.666657 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" podStartSLOduration=2.666643912 podStartE2EDuration="2.666643912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:49.666156201 +0000 UTC m=+3674.193922990" watchObservedRunningTime="2026-10-30 05:45:49.666643912 +0000 UTC m=+3674.194410730" Oct 30 05:45:49.709833 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:45:49.709787 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" podStartSLOduration=1.709775144 podStartE2EDuration="1.709775144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:49.70879326 +0000 UTC m=+3674.236560046" watchObservedRunningTime="2026-10-30 05:45:49.709775144 +0000 UTC m=+3674.237541929" Oct 30 05:45:55.986870 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:45:55.986832 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:46:00.632575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:00.632543 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" Oct 30 05:46:00.633102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:00.632622 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" Oct 30 05:46:17.032507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:17.032414 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m"] Oct 30 05:46:17.046024 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:17.045978 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-h6z2m"] Oct 30 05:46:17.580350 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:17.580308 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774c4c38-642b-44a1-8424-373c0c5ab10b" path="/var/lib/kubelet/pods/774c4c38-642b-44a1-8424-373c0c5ab10b/volumes" Oct 30 05:46:25.985287 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:46:25.985242 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:46:27.079937 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.079896 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth"] Oct 30 05:46:27.084887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.084862 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" Oct 30 05:46:27.101414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.101387 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:46:27.104341 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.104309 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth"] Oct 30 05:46:27.114581 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.114554 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:46:27.116645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.116608 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:46:27.116778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.116690 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:46:27.171571 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.171541 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-031e2e8e-b6gth\" (UID: \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" Oct 30 05:46:27.171751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.171669 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-031e2e8e-b6gth\" (UID: \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" Oct 30 05:46:27.171751 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.171720 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfqr7\" (UniqueName: \"kubernetes.io/projected/6ad1d0a6-6183-4446-83b8-3d869446f1a9-kube-api-access-bfqr7\") pod \"maas-api-revoke-sub-031e2e8e-b6gth\" (UID: \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" Oct 30 05:46:27.272926 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.272889 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfqr7\" (UniqueName: \"kubernetes.io/projected/6ad1d0a6-6183-4446-83b8-3d869446f1a9-kube-api-access-bfqr7\") pod \"maas-api-revoke-sub-031e2e8e-b6gth\" (UID: \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" Oct 30 05:46:27.273149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.272998 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-031e2e8e-b6gth\" (UID: \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" Oct 30 05:46:27.273149 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.273124 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-031e2e8e-b6gth\" (UID: \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" Oct 30 05:46:27.273827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.273796 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-031e2e8e-b6gth\" (UID: \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" Oct 30 05:46:27.275551 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.275525 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-031e2e8e-b6gth\" (UID: \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" Oct 30 05:46:27.294318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.294288 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfqr7\" (UniqueName: \"kubernetes.io/projected/6ad1d0a6-6183-4446-83b8-3d869446f1a9-kube-api-access-bfqr7\") pod \"maas-api-revoke-sub-031e2e8e-b6gth\" (UID: \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" Oct 30 05:46:27.399338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.399248 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" Oct 30 05:46:27.697277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.697243 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth"] Oct 30 05:46:27.697629 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:46:27.697605 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad1d0a6_6183_4446_83b8_3d869446f1a9.slice/crio-bcba994d76b6b55f834f729d104c0cab3572137ed83c6aff4832314a9e860c64 WatchSource:0}: Error finding container bcba994d76b6b55f834f729d104c0cab3572137ed83c6aff4832314a9e860c64: Status 404 returned error can't find the container with id bcba994d76b6b55f834f729d104c0cab3572137ed83c6aff4832314a9e860c64 Oct 30 05:46:27.835331 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:27.835294 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" event={"ID":"6ad1d0a6-6183-4446-83b8-3d869446f1a9","Type":"ContainerStarted","Data":"bcba994d76b6b55f834f729d104c0cab3572137ed83c6aff4832314a9e860c64"} Oct 30 05:46:28.842936 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:28.842899 6587 generic.go:356] "Generic (PLEG): container finished" podID="6ad1d0a6-6183-4446-83b8-3d869446f1a9" containerID="c1f9905bfc9f77001b4283c64e9435ad5e61069f3fdee0e638c5323ce00c8317" exitCode=0 Oct 30 05:46:28.843352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:28.842937 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" event={"ID":"6ad1d0a6-6183-4446-83b8-3d869446f1a9","Type":"ContainerDied","Data":"c1f9905bfc9f77001b4283c64e9435ad5e61069f3fdee0e638c5323ce00c8317"} Oct 30 05:46:29.986424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:29.986398 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" Oct 30 05:46:30.102932 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:30.102828 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-xks-ca\") pod \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\" (UID: \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\") " Oct 30 05:46:30.103145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:30.102994 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad1d0a6-6183-4446-83b8-3d869446f1a9-kube-api-access-bfqr7\" (UniqueName: \"kubernetes.io/projected/6ad1d0a6-6183-4446-83b8-3d869446f1a9-kube-api-access-bfqr7\") pod \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\" (UID: \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\") " Oct 30 05:46:30.103145 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:30.103036 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-openshift-ca\") pod \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\" (UID: \"6ad1d0a6-6183-4446-83b8-3d869446f1a9\") " Oct 30 05:46:30.103432 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:30.103411 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-openshift-ca" pod "6ad1d0a6-6183-4446-83b8-3d869446f1a9" (UID: "6ad1d0a6-6183-4446-83b8-3d869446f1a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:30.105297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:30.105268 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-xks-ca" pod "6ad1d0a6-6183-4446-83b8-3d869446f1a9" (UID: "6ad1d0a6-6183-4446-83b8-3d869446f1a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:30.105493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:30.105477 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad1d0a6-6183-4446-83b8-3d869446f1a9-kube-api-access-bfqr7" pod "6ad1d0a6-6183-4446-83b8-3d869446f1a9" (UID: "6ad1d0a6-6183-4446-83b8-3d869446f1a9"). InnerVolumeSpecName "kube-api-access-bfqr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:30.204873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:30.204839 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfqr7\" (UniqueName: \"kubernetes.io/projected/6ad1d0a6-6183-4446-83b8-3d869446f1a9-kube-api-access-bfqr7\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:46:30.204873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:30.204866 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:46:30.204873 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:30.204876 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad1d0a6-6183-4446-83b8-3d869446f1a9-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:46:30.857262 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:30.857221 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" Oct 30 05:46:30.857483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:30.857256 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth" event={"ID":"6ad1d0a6-6183-4446-83b8-3d869446f1a9","Type":"ContainerDied","Data":"bcba994d76b6b55f834f729d104c0cab3572137ed83c6aff4832314a9e860c64"} Oct 30 05:46:30.857483 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:30.857317 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcba994d76b6b55f834f729d104c0cab3572137ed83c6aff4832314a9e860c64" Oct 30 05:46:38.439883 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.439842 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv"] Oct 30 05:46:38.440324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.440299 6587 cpu_manager.go:414] "containerMap: removing container" podUID="6ad1d0a6-6183-4446-83b8-3d869446f1a9" containerName="revoke-keys" Oct 30 05:46:38.440324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.440311 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad1d0a6-6183-4446-83b8-3d869446f1a9" containerName="revoke-keys" Oct 30 05:46:38.440444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.440401 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad1d0a6-6183-4446-83b8-3d869446f1a9" containerName="revoke-keys" Oct 30 05:46:38.443957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.443938 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" Oct 30 05:46:38.474010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.473981 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:46:38.474547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.474297 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:46:38.474547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.474397 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:46:38.474547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.474491 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:46:38.477329 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.477304 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv"] Oct 30 05:46:38.487560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.487532 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbp4v\" (UniqueName: \"kubernetes.io/projected/7257c144-c017-437c-b4e8-f8a37ebd8133-kube-api-access-bbp4v\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv\" (UID: \"7257c144-c017-437c-b4e8-f8a37ebd8133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" Oct 30 05:46:38.487689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.487572 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv\" (UID: \"7257c144-c017-437c-b4e8-f8a37ebd8133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" Oct 30 05:46:38.487689 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.487596 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv\" (UID: \"7257c144-c017-437c-b4e8-f8a37ebd8133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" Oct 30 05:46:38.589138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.589098 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbp4v\" (UniqueName: \"kubernetes.io/projected/7257c144-c017-437c-b4e8-f8a37ebd8133-kube-api-access-bbp4v\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv\" (UID: \"7257c144-c017-437c-b4e8-f8a37ebd8133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" Oct 30 05:46:38.589138 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.589143 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv\" (UID: \"7257c144-c017-437c-b4e8-f8a37ebd8133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" Oct 30 05:46:38.589505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.589171 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv\" (UID: \"7257c144-c017-437c-b4e8-f8a37ebd8133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" Oct 30 05:46:38.589992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.589968 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv\" (UID: \"7257c144-c017-437c-b4e8-f8a37ebd8133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" Oct 30 05:46:38.591638 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.591616 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv\" (UID: \"7257c144-c017-437c-b4e8-f8a37ebd8133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" Oct 30 05:46:38.610857 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.610832 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbp4v\" (UniqueName: \"kubernetes.io/projected/7257c144-c017-437c-b4e8-f8a37ebd8133-kube-api-access-bbp4v\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv\" (UID: \"7257c144-c017-437c-b4e8-f8a37ebd8133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" Oct 30 05:46:38.758596 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:38.758518 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" Oct 30 05:46:39.063187 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:46:39.063155 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7257c144_c017_437c_b4e8_f8a37ebd8133.slice/crio-62546be9f40083b0d5461111d8937c1ce033a4aaf1e28cdd77c2b26f6ff7db10 WatchSource:0}: Error finding container 62546be9f40083b0d5461111d8937c1ce033a4aaf1e28cdd77c2b26f6ff7db10: Status 404 returned error can't find the container with id 62546be9f40083b0d5461111d8937c1ce033a4aaf1e28cdd77c2b26f6ff7db10 Oct 30 05:46:39.063187 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:39.063165 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv"] Oct 30 05:46:39.906954 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:39.906913 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" event={"ID":"7257c144-c017-437c-b4e8-f8a37ebd8133","Type":"ContainerStarted","Data":"431a5ba6f3c33c88a0841d6cd58e3dcbc1524bb1f0449519661de15ec418f654"} Oct 30 05:46:39.907495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:39.906976 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" event={"ID":"7257c144-c017-437c-b4e8-f8a37ebd8133","Type":"ContainerStarted","Data":"62546be9f40083b0d5461111d8937c1ce033a4aaf1e28cdd77c2b26f6ff7db10"} Oct 30 05:46:39.951374 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:39.951307 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" podStartSLOduration=1.951291958 podStartE2EDuration="1.951291958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:39.949387265 +0000 UTC m=+3724.477154048" watchObservedRunningTime="2026-10-30 05:46:39.951291958 +0000 UTC m=+3724.479058743" Oct 30 05:46:40.913829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:40.913790 6587 generic.go:356] "Generic (PLEG): container finished" podID="7257c144-c017-437c-b4e8-f8a37ebd8133" containerID="431a5ba6f3c33c88a0841d6cd58e3dcbc1524bb1f0449519661de15ec418f654" exitCode=0 Oct 30 05:46:40.914249 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:40.913864 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" event={"ID":"7257c144-c017-437c-b4e8-f8a37ebd8133","Type":"ContainerDied","Data":"431a5ba6f3c33c88a0841d6cd58e3dcbc1524bb1f0449519661de15ec418f654"} Oct 30 05:46:42.057649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:42.057620 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" Oct 30 05:46:42.127477 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:42.127441 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-xks-ca\") pod \"7257c144-c017-437c-b4e8-f8a37ebd8133\" (UID: \"7257c144-c017-437c-b4e8-f8a37ebd8133\") " Oct 30 05:46:42.127710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:42.127534 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7257c144-c017-437c-b4e8-f8a37ebd8133-kube-api-access-bbp4v\" (UniqueName: \"kubernetes.io/projected/7257c144-c017-437c-b4e8-f8a37ebd8133-kube-api-access-bbp4v\") pod \"7257c144-c017-437c-b4e8-f8a37ebd8133\" (UID: \"7257c144-c017-437c-b4e8-f8a37ebd8133\") " Oct 30 05:46:42.127710 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:42.127604 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-openshift-ca\") pod \"7257c144-c017-437c-b4e8-f8a37ebd8133\" (UID: \"7257c144-c017-437c-b4e8-f8a37ebd8133\") " Oct 30 05:46:42.128238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:42.128207 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-openshift-ca" pod "7257c144-c017-437c-b4e8-f8a37ebd8133" (UID: "7257c144-c017-437c-b4e8-f8a37ebd8133"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:42.129912 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:42.129881 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-xks-ca" pod "7257c144-c017-437c-b4e8-f8a37ebd8133" (UID: "7257c144-c017-437c-b4e8-f8a37ebd8133"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:42.130294 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:42.130269 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7257c144-c017-437c-b4e8-f8a37ebd8133-kube-api-access-bbp4v" pod "7257c144-c017-437c-b4e8-f8a37ebd8133" (UID: "7257c144-c017-437c-b4e8-f8a37ebd8133"). InnerVolumeSpecName "kube-api-access-bbp4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:42.229607 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:42.229510 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbp4v\" (UniqueName: \"kubernetes.io/projected/7257c144-c017-437c-b4e8-f8a37ebd8133-kube-api-access-bbp4v\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:46:42.229607 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:42.229551 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:46:42.229607 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:42.229561 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7257c144-c017-437c-b4e8-f8a37ebd8133-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:46:42.926076 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:42.926048 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" Oct 30 05:46:42.926260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:42.926088 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv" event={"ID":"7257c144-c017-437c-b4e8-f8a37ebd8133","Type":"ContainerDied","Data":"62546be9f40083b0d5461111d8937c1ce033a4aaf1e28cdd77c2b26f6ff7db10"} Oct 30 05:46:42.926260 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:42.926139 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62546be9f40083b0d5461111d8937c1ce033a4aaf1e28cdd77c2b26f6ff7db10" Oct 30 05:46:45.131326 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:45.131296 6587 scope.go:122] "RemoveContainer" containerID="f9ac45b56c69f4d9a7b3770da2df80fd52f00bcab3892d0b4d23d91bf97c42f8" Oct 30 05:46:45.140507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:45.140487 6587 scope.go:122] "RemoveContainer" containerID="b54ef70f7a713f535267aea340d64f1dfeb7476fddf2f9f8f8c88b633830c5e6" Oct 30 05:46:45.149474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:45.149454 6587 scope.go:122] "RemoveContainer" containerID="25f11695e8d936f1b098da0ab27888853c0f9c3558c060bd7c9c5e37be74e3f4" Oct 30 05:46:45.157837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:45.157817 6587 scope.go:122] "RemoveContainer" containerID="65bde700dcc49ac47e19599a45d7c2a18140f68f53be0ca5b8a6d8db7d803644" Oct 30 05:46:49.005900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:49.005856 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj"] Oct 30 05:46:49.038880 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:49.038844 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx"] Oct 30 05:46:52.448139 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:52.448098 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv"] Oct 30 05:46:52.461892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:52.461851 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f97a49-9dnwv"] Oct 30 05:46:53.579546 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:53.579514 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7257c144-c017-437c-b4e8-f8a37ebd8133" path="/var/lib/kubelet/pods/7257c144-c017-437c-b4e8-f8a37ebd8133/volumes" Oct 30 05:46:54.007615 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:54.007526 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" podUID="54eb51b0-1a0f-4797-9b48-4b31b8eaf783" containerName="payload-processing" containerID="cri-o://7b3218343c7cb3a03752471af279bbdb26c47b94bb973593009c33683322179b" gracePeriod=25 Oct 30 05:46:54.039300 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:46:54.039259 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" podUID="56105d2f-9c30-4638-8b77-05af2f3c7e39" containerName="payload-pre-processing" containerID="cri-o://4f8db0246277bd58add046b6d95c45a25cd43e8efa3406dd6e3aecfb35607481" gracePeriod=25 Oct 30 05:46:55.986051 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:46:55.986004 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:47:00.630711 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:00.630670 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" podUID="54eb51b0-1a0f-4797-9b48-4b31b8eaf783" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:47:00.631148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:00.630789 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" podUID="56105d2f-9c30-4638-8b77-05af2f3c7e39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:47:01.164200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.164166 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw"] Oct 30 05:47:01.177874 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.177845 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fskrw"] Oct 30 05:47:01.217741 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.217705 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd"] Oct 30 05:47:01.218144 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.218131 6587 cpu_manager.go:414] "containerMap: removing container" podUID="7257c144-c017-437c-b4e8-f8a37ebd8133" containerName="revoke-keys" Oct 30 05:47:01.218206 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.218145 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7257c144-c017-437c-b4e8-f8a37ebd8133" containerName="revoke-keys" Oct 30 05:47:01.218257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.218240 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="7257c144-c017-437c-b4e8-f8a37ebd8133" containerName="revoke-keys" Oct 30 05:47:01.223214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.223195 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" Oct 30 05:47:01.239889 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.239866 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:47:01.240073 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.240012 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:47:01.240189 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.240111 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:47:01.240292 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.240152 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:47:01.242625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.242601 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd"] Oct 30 05:47:01.336352 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.336313 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t4hsd\" (UID: \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" Oct 30 05:47:01.336537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.336482 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfqdv\" (UniqueName: \"kubernetes.io/projected/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-kube-api-access-nfqdv\") pod \"maas-api-revoke-sub-3d4156f1-t4hsd\" (UID: \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" Oct 30 05:47:01.336611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.336592 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t4hsd\" (UID: \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" Oct 30 05:47:01.437493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.437413 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfqdv\" (UniqueName: \"kubernetes.io/projected/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-kube-api-access-nfqdv\") pod \"maas-api-revoke-sub-3d4156f1-t4hsd\" (UID: \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" Oct 30 05:47:01.437493 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.437486 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t4hsd\" (UID: \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" Oct 30 05:47:01.437756 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.437521 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t4hsd\" (UID: \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" Oct 30 05:47:01.438128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.438107 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t4hsd\" (UID: \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" Oct 30 05:47:01.439999 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.439973 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t4hsd\" (UID: \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" Oct 30 05:47:01.459761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.459734 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfqdv\" (UniqueName: \"kubernetes.io/projected/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-kube-api-access-nfqdv\") pod \"maas-api-revoke-sub-3d4156f1-t4hsd\" (UID: \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" Oct 30 05:47:01.536612 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.536577 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" Oct 30 05:47:01.582620 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.582583 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f423465e-1673-45ef-8992-e3c7f5684e87" path="/var/lib/kubelet/pods/f423465e-1673-45ef-8992-e3c7f5684e87/volumes" Oct 30 05:47:01.844738 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:01.844706 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd"] Oct 30 05:47:01.845959 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:47:01.845929 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3741a835_a3bf_404b_8ec4_d3f7e53ecf19.slice/crio-a8cbfdc37ed041a668c97a60c619dc8b8be07b5a0323af098dc8c731023c7f3a WatchSource:0}: Error finding container a8cbfdc37ed041a668c97a60c619dc8b8be07b5a0323af098dc8c731023c7f3a: Status 404 returned error can't find the container with id a8cbfdc37ed041a668c97a60c619dc8b8be07b5a0323af098dc8c731023c7f3a Oct 30 05:47:02.029244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:02.029192 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" event={"ID":"3741a835-a3bf-404b-8ec4-d3f7e53ecf19","Type":"ContainerStarted","Data":"a8cbfdc37ed041a668c97a60c619dc8b8be07b5a0323af098dc8c731023c7f3a"} Oct 30 05:47:03.035903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:03.035809 6587 generic.go:356] "Generic (PLEG): container finished" podID="3741a835-a3bf-404b-8ec4-d3f7e53ecf19" containerID="e757a58a1cb2f679cda07982a051a2e33f215fa3fb04fcc5e47dc9bc4c5e7c4f" exitCode=0 Oct 30 05:47:03.035903 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:03.035850 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" event={"ID":"3741a835-a3bf-404b-8ec4-d3f7e53ecf19","Type":"ContainerDied","Data":"e757a58a1cb2f679cda07982a051a2e33f215fa3fb04fcc5e47dc9bc4c5e7c4f"} Oct 30 05:47:04.178064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:04.178038 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" Oct 30 05:47:04.260861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:04.260823 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-xks-ca\") pod \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\" (UID: \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\") " Oct 30 05:47:04.261065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:04.260885 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-openshift-ca\") pod \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\" (UID: \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\") " Oct 30 05:47:04.261065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:04.260909 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-kube-api-access-nfqdv\" (UniqueName: \"kubernetes.io/projected/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-kube-api-access-nfqdv\") pod \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\" (UID: \"3741a835-a3bf-404b-8ec4-d3f7e53ecf19\") " Oct 30 05:47:04.261293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:04.261269 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-openshift-ca" pod "3741a835-a3bf-404b-8ec4-d3f7e53ecf19" (UID: "3741a835-a3bf-404b-8ec4-d3f7e53ecf19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:04.263084 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:04.263063 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-xks-ca" pod "3741a835-a3bf-404b-8ec4-d3f7e53ecf19" (UID: "3741a835-a3bf-404b-8ec4-d3f7e53ecf19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:04.263165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:04.263115 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-kube-api-access-nfqdv" pod "3741a835-a3bf-404b-8ec4-d3f7e53ecf19" (UID: "3741a835-a3bf-404b-8ec4-d3f7e53ecf19"). InnerVolumeSpecName "kube-api-access-nfqdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:04.361845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:04.361809 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:47:04.361845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:04.361841 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfqdv\" (UniqueName: \"kubernetes.io/projected/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-kube-api-access-nfqdv\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:47:04.361845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:04.361853 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3741a835-a3bf-404b-8ec4-d3f7e53ecf19-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:47:05.047229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:05.047200 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" Oct 30 05:47:05.047229 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:05.047214 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd" event={"ID":"3741a835-a3bf-404b-8ec4-d3f7e53ecf19","Type":"ContainerDied","Data":"a8cbfdc37ed041a668c97a60c619dc8b8be07b5a0323af098dc8c731023c7f3a"} Oct 30 05:47:05.047466 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:05.047261 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8cbfdc37ed041a668c97a60c619dc8b8be07b5a0323af098dc8c731023c7f3a" Oct 30 05:47:06.029575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:06.029538 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd"] Oct 30 05:47:06.043153 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:06.043123 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qcvtd"] Oct 30 05:47:07.579870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:07.579829 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336378ec-ba3e-45d8-a954-f57cdc98c79d" path="/var/lib/kubelet/pods/336378ec-ba3e-45d8-a954-f57cdc98c79d/volumes" Oct 30 05:47:10.631177 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:10.631135 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" podUID="56105d2f-9c30-4638-8b77-05af2f3c7e39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:47:10.631683 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:10.631137 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" podUID="54eb51b0-1a0f-4797-9b48-4b31b8eaf783" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:47:14.298136 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.298107 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" Oct 30 05:47:14.324392 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.324341 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" Oct 30 05:47:14.353018 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.352989 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-kube-api-access-5jp9v\" (UniqueName: \"kubernetes.io/projected/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-kube-api-access-5jp9v\") pod \"54eb51b0-1a0f-4797-9b48-4b31b8eaf783\" (UID: \"54eb51b0-1a0f-4797-9b48-4b31b8eaf783\") " Oct 30 05:47:14.353200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.353057 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-plugins-config-volume\") pod \"54eb51b0-1a0f-4797-9b48-4b31b8eaf783\" (UID: \"54eb51b0-1a0f-4797-9b48-4b31b8eaf783\") " Oct 30 05:47:14.353200 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.353131 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56105d2f-9c30-4638-8b77-05af2f3c7e39-kube-api-access-h9t5j\" (UniqueName: \"kubernetes.io/projected/56105d2f-9c30-4638-8b77-05af2f3c7e39-kube-api-access-h9t5j\") pod \"56105d2f-9c30-4638-8b77-05af2f3c7e39\" (UID: \"56105d2f-9c30-4638-8b77-05af2f3c7e39\") " Oct 30 05:47:14.353334 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.353240 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56105d2f-9c30-4638-8b77-05af2f3c7e39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56105d2f-9c30-4638-8b77-05af2f3c7e39-plugins-config-volume\") pod \"56105d2f-9c30-4638-8b77-05af2f3c7e39\" (UID: \"56105d2f-9c30-4638-8b77-05af2f3c7e39\") " Oct 30 05:47:14.353660 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.353606 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-plugins-config-volume" pod "54eb51b0-1a0f-4797-9b48-4b31b8eaf783" (UID: "54eb51b0-1a0f-4797-9b48-4b31b8eaf783"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:14.353822 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.353657 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56105d2f-9c30-4638-8b77-05af2f3c7e39-plugins-config-volume" pod "56105d2f-9c30-4638-8b77-05af2f3c7e39" (UID: "56105d2f-9c30-4638-8b77-05af2f3c7e39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:14.355308 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.355279 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56105d2f-9c30-4638-8b77-05af2f3c7e39-kube-api-access-h9t5j" pod "56105d2f-9c30-4638-8b77-05af2f3c7e39" (UID: "56105d2f-9c30-4638-8b77-05af2f3c7e39"). InnerVolumeSpecName "kube-api-access-h9t5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:14.355418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.355330 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-kube-api-access-5jp9v" pod "54eb51b0-1a0f-4797-9b48-4b31b8eaf783" (UID: "54eb51b0-1a0f-4797-9b48-4b31b8eaf783"). InnerVolumeSpecName "kube-api-access-5jp9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:14.455006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.454973 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56105d2f-9c30-4638-8b77-05af2f3c7e39-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:47:14.455006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.455000 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jp9v\" (UniqueName: \"kubernetes.io/projected/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-kube-api-access-5jp9v\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:47:14.455006 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.455011 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54eb51b0-1a0f-4797-9b48-4b31b8eaf783-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:47:14.455259 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:14.455020 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9t5j\" (UniqueName: \"kubernetes.io/projected/56105d2f-9c30-4638-8b77-05af2f3c7e39-kube-api-access-h9t5j\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:47:15.099855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.099819 6587 generic.go:356] "Generic (PLEG): container finished" podID="56105d2f-9c30-4638-8b77-05af2f3c7e39" containerID="4f8db0246277bd58add046b6d95c45a25cd43e8efa3406dd6e3aecfb35607481" exitCode=0 Oct 30 05:47:15.100064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.099867 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" Oct 30 05:47:15.100064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.099907 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" event={"ID":"56105d2f-9c30-4638-8b77-05af2f3c7e39","Type":"ContainerDied","Data":"4f8db0246277bd58add046b6d95c45a25cd43e8efa3406dd6e3aecfb35607481"} Oct 30 05:47:15.100064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.099965 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx" event={"ID":"56105d2f-9c30-4638-8b77-05af2f3c7e39","Type":"ContainerDied","Data":"d066145ec94403ad14143c9a8c450d56a6a970571826439c33a9ca0e4fc95f3e"} Oct 30 05:47:15.100064 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.099994 6587 scope.go:122] "RemoveContainer" containerID="4f8db0246277bd58add046b6d95c45a25cd43e8efa3406dd6e3aecfb35607481" Oct 30 05:47:15.101591 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.101563 6587 generic.go:356] "Generic (PLEG): container finished" podID="54eb51b0-1a0f-4797-9b48-4b31b8eaf783" containerID="7b3218343c7cb3a03752471af279bbdb26c47b94bb973593009c33683322179b" exitCode=0 Oct 30 05:47:15.101706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.101594 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" event={"ID":"54eb51b0-1a0f-4797-9b48-4b31b8eaf783","Type":"ContainerDied","Data":"7b3218343c7cb3a03752471af279bbdb26c47b94bb973593009c33683322179b"} Oct 30 05:47:15.101706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.101656 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" event={"ID":"54eb51b0-1a0f-4797-9b48-4b31b8eaf783","Type":"ContainerDied","Data":"3705aecd637ec4649b97eb288d3efb00fa46479cdfa5ccf2832c411faa986e99"} Oct 30 05:47:15.101706 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.101603 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj" Oct 30 05:47:15.109633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.109615 6587 scope.go:122] "RemoveContainer" containerID="4f8db0246277bd58add046b6d95c45a25cd43e8efa3406dd6e3aecfb35607481" Oct 30 05:47:15.109933 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:47:15.109903 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8db0246277bd58add046b6d95c45a25cd43e8efa3406dd6e3aecfb35607481\": container with ID starting with 4f8db0246277bd58add046b6d95c45a25cd43e8efa3406dd6e3aecfb35607481 not found: ID does not exist" containerID="4f8db0246277bd58add046b6d95c45a25cd43e8efa3406dd6e3aecfb35607481" Oct 30 05:47:15.110033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.109944 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8db0246277bd58add046b6d95c45a25cd43e8efa3406dd6e3aecfb35607481"} err="failed to get container status \"4f8db0246277bd58add046b6d95c45a25cd43e8efa3406dd6e3aecfb35607481\": rpc error: code = NotFound desc = could not find container \"4f8db0246277bd58add046b6d95c45a25cd43e8efa3406dd6e3aecfb35607481\": container with ID starting with 4f8db0246277bd58add046b6d95c45a25cd43e8efa3406dd6e3aecfb35607481 not found: ID does not exist" Oct 30 05:47:15.110033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.109962 6587 scope.go:122] "RemoveContainer" containerID="7b3218343c7cb3a03752471af279bbdb26c47b94bb973593009c33683322179b" Oct 30 05:47:15.118534 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.118514 6587 scope.go:122] "RemoveContainer" containerID="7b3218343c7cb3a03752471af279bbdb26c47b94bb973593009c33683322179b" Oct 30 05:47:15.118779 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:47:15.118760 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3218343c7cb3a03752471af279bbdb26c47b94bb973593009c33683322179b\": container with ID starting with 7b3218343c7cb3a03752471af279bbdb26c47b94bb973593009c33683322179b not found: ID does not exist" containerID="7b3218343c7cb3a03752471af279bbdb26c47b94bb973593009c33683322179b" Oct 30 05:47:15.118831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.118785 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3218343c7cb3a03752471af279bbdb26c47b94bb973593009c33683322179b"} err="failed to get container status \"7b3218343c7cb3a03752471af279bbdb26c47b94bb973593009c33683322179b\": rpc error: code = NotFound desc = could not find container \"7b3218343c7cb3a03752471af279bbdb26c47b94bb973593009c33683322179b\": container with ID starting with 7b3218343c7cb3a03752471af279bbdb26c47b94bb973593009c33683322179b not found: ID does not exist" Oct 30 05:47:15.156536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.156503 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj"] Oct 30 05:47:15.168641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.168613 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj"] Oct 30 05:47:15.207181 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.207145 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx"] Oct 30 05:47:15.208878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.208856 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx"] Oct 30 05:47:15.580649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.580616 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54eb51b0-1a0f-4797-9b48-4b31b8eaf783" path="/var/lib/kubelet/pods/54eb51b0-1a0f-4797-9b48-4b31b8eaf783/volumes" Oct 30 05:47:15.581050 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:15.580982 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56105d2f-9c30-4638-8b77-05af2f3c7e39" path="/var/lib/kubelet/pods/56105d2f-9c30-4638-8b77-05af2f3c7e39/volumes" Oct 30 05:47:18.913733 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:18.913691 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95"] Oct 30 05:47:18.914250 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:18.913914 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" podUID="d296c368-73b0-4326-87bc-cd3508c843b9" containerName="manager" containerID="cri-o://bf05d17d075e966e7983d83256849e46458bc2a55dff08bfc8a5e7faaee7fc81" gracePeriod=10 Oct 30 05:47:19.134732 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:19.134700 6587 generic.go:356] "Generic (PLEG): container finished" podID="d296c368-73b0-4326-87bc-cd3508c843b9" containerID="bf05d17d075e966e7983d83256849e46458bc2a55dff08bfc8a5e7faaee7fc81" exitCode=0 Oct 30 05:47:19.134918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:19.134767 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" event={"ID":"d296c368-73b0-4326-87bc-cd3508c843b9","Type":"ContainerDied","Data":"bf05d17d075e966e7983d83256849e46458bc2a55dff08bfc8a5e7faaee7fc81"} Oct 30 05:47:19.177507 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:19.177446 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" Oct 30 05:47:19.305677 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:19.305638 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d296c368-73b0-4326-87bc-cd3508c843b9-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d296c368-73b0-4326-87bc-cd3508c843b9-extensions-socket-volume\") pod \"d296c368-73b0-4326-87bc-cd3508c843b9\" (UID: \"d296c368-73b0-4326-87bc-cd3508c843b9\") " Oct 30 05:47:19.305887 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:19.305774 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d296c368-73b0-4326-87bc-cd3508c843b9-kube-api-access-j8gr6\" (UniqueName: \"kubernetes.io/projected/d296c368-73b0-4326-87bc-cd3508c843b9-kube-api-access-j8gr6\") pod \"d296c368-73b0-4326-87bc-cd3508c843b9\" (UID: \"d296c368-73b0-4326-87bc-cd3508c843b9\") " Oct 30 05:47:19.307123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:19.307086 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d296c368-73b0-4326-87bc-cd3508c843b9-extensions-socket-volume" pod "d296c368-73b0-4326-87bc-cd3508c843b9" (UID: "d296c368-73b0-4326-87bc-cd3508c843b9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:47:19.307972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:19.307949 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d296c368-73b0-4326-87bc-cd3508c843b9-kube-api-access-j8gr6" pod "d296c368-73b0-4326-87bc-cd3508c843b9" (UID: "d296c368-73b0-4326-87bc-cd3508c843b9"). InnerVolumeSpecName "kube-api-access-j8gr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:19.406865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:19.406822 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8gr6\" (UniqueName: \"kubernetes.io/projected/d296c368-73b0-4326-87bc-cd3508c843b9-kube-api-access-j8gr6\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:47:19.406865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:19.406858 6587 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d296c368-73b0-4326-87bc-cd3508c843b9-extensions-socket-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:47:20.141890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:20.141857 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" Oct 30 05:47:20.141890 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:20.141875 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95" event={"ID":"d296c368-73b0-4326-87bc-cd3508c843b9","Type":"ContainerDied","Data":"7034af9dbdb6729d5032ef93502d2bd5b8ad981c76eb61742d859877755384d9"} Oct 30 05:47:20.142449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:20.141936 6587 scope.go:122] "RemoveContainer" containerID="bf05d17d075e966e7983d83256849e46458bc2a55dff08bfc8a5e7faaee7fc81" Oct 30 05:47:20.210969 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:20.210935 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95"] Oct 30 05:47:20.224560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:20.224534 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zdd95"] Oct 30 05:47:21.591148 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:21.591100 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d296c368-73b0-4326-87bc-cd3508c843b9" path="/var/lib/kubelet/pods/d296c368-73b0-4326-87bc-cd3508c843b9/volumes" Oct 30 05:47:25.985065 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:47:25.985024 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:47:27.029617 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:27.029578 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq"] Oct 30 05:47:27.043056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:27.043022 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-92ncq"] Oct 30 05:47:27.579763 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:27.579728 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d4fb7b8-d2cd-4eff-899a-7c559a6706cf" path="/var/lib/kubelet/pods/9d4fb7b8-d2cd-4eff-899a-7c559a6706cf/volumes" Oct 30 05:47:38.029283 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:38.029237 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l"] Oct 30 05:47:38.042790 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:38.042760 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-95d4l"] Oct 30 05:47:39.579372 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:39.579326 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7030f58-ff6c-4166-9033-4f97c6f5f2a1" path="/var/lib/kubelet/pods/c7030f58-ff6c-4166-9033-4f97c6f5f2a1/volumes" Oct 30 05:47:45.192044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:47:45.192014 6587 scope.go:122] "RemoveContainer" containerID="8549ce085a3cd882bedfd1ad204faf94f8a9b85ae63f9fa1c1ffffe449c436e6" Oct 30 05:47:55.984853 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:47:55.984776 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:48:06.027450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:06.027412 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb"] Oct 30 05:48:06.040835 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:06.040806 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6vswb"] Oct 30 05:48:07.580035 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:07.580000 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf8aee3-edcd-4277-a648-d4719042ef5e" path="/var/lib/kubelet/pods/daf8aee3-edcd-4277-a648-d4719042ef5e/volumes" Oct 30 05:48:17.029662 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:17.029628 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz"] Oct 30 05:48:17.032278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:17.032250 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4wwdz"] Oct 30 05:48:17.579992 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:17.579959 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4" path="/var/lib/kubelet/pods/b0630b7b-ed66-48bf-9aff-d7fc9ebe29d4/volumes" Oct 30 05:48:22.100371 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.100317 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn"] Oct 30 05:48:22.100773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.100721 6587 cpu_manager.go:414] "containerMap: removing container" podUID="54eb51b0-1a0f-4797-9b48-4b31b8eaf783" containerName="payload-processing" Oct 30 05:48:22.100773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.100732 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54eb51b0-1a0f-4797-9b48-4b31b8eaf783" containerName="payload-processing" Oct 30 05:48:22.100773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.100740 6587 cpu_manager.go:414] "containerMap: removing container" podUID="d296c368-73b0-4326-87bc-cd3508c843b9" containerName="manager" Oct 30 05:48:22.100773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.100745 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d296c368-73b0-4326-87bc-cd3508c843b9" containerName="manager" Oct 30 05:48:22.100773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.100762 6587 cpu_manager.go:414] "containerMap: removing container" podUID="56105d2f-9c30-4638-8b77-05af2f3c7e39" containerName="payload-pre-processing" Oct 30 05:48:22.100773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.100768 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56105d2f-9c30-4638-8b77-05af2f3c7e39" containerName="payload-pre-processing" Oct 30 05:48:22.100990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.100792 6587 cpu_manager.go:414] "containerMap: removing container" podUID="3741a835-a3bf-404b-8ec4-d3f7e53ecf19" containerName="revoke-keys" Oct 30 05:48:22.100990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.100798 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3741a835-a3bf-404b-8ec4-d3f7e53ecf19" containerName="revoke-keys" Oct 30 05:48:22.100990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.100853 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="56105d2f-9c30-4638-8b77-05af2f3c7e39" containerName="payload-pre-processing" Oct 30 05:48:22.100990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.100866 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="3741a835-a3bf-404b-8ec4-d3f7e53ecf19" containerName="revoke-keys" Oct 30 05:48:22.100990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.100874 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="54eb51b0-1a0f-4797-9b48-4b31b8eaf783" containerName="payload-processing" Oct 30 05:48:22.100990 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.100880 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="d296c368-73b0-4326-87bc-cd3508c843b9" containerName="manager" Oct 30 05:48:22.104173 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.104153 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn" Oct 30 05:48:22.114661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.114633 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn"] Oct 30 05:48:22.126213 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.126191 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4tst5\"" Oct 30 05:48:22.130563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.130536 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptntb\" (UniqueName: \"kubernetes.io/projected/9fed05f3-ad75-4044-ad92-938bcdea9fcc-kube-api-access-ptntb\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mswsn\" (UID: \"9fed05f3-ad75-4044-ad92-938bcdea9fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn" Oct 30 05:48:22.130708 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.130659 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fed05f3-ad75-4044-ad92-938bcdea9fcc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mswsn\" (UID: \"9fed05f3-ad75-4044-ad92-938bcdea9fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn" Oct 30 05:48:22.231179 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.231144 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fed05f3-ad75-4044-ad92-938bcdea9fcc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mswsn\" (UID: \"9fed05f3-ad75-4044-ad92-938bcdea9fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn" Oct 30 05:48:22.231378 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.231235 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptntb\" (UniqueName: \"kubernetes.io/projected/9fed05f3-ad75-4044-ad92-938bcdea9fcc-kube-api-access-ptntb\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mswsn\" (UID: \"9fed05f3-ad75-4044-ad92-938bcdea9fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn" Oct 30 05:48:22.231560 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.231541 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fed05f3-ad75-4044-ad92-938bcdea9fcc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mswsn\" (UID: \"9fed05f3-ad75-4044-ad92-938bcdea9fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn" Oct 30 05:48:22.252542 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.252521 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptntb\" (UniqueName: \"kubernetes.io/projected/9fed05f3-ad75-4044-ad92-938bcdea9fcc-kube-api-access-ptntb\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mswsn\" (UID: \"9fed05f3-ad75-4044-ad92-938bcdea9fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn" Oct 30 05:48:22.418387 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.418250 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn" Oct 30 05:48:22.740776 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:22.740747 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn"] Oct 30 05:48:22.741125 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:48:22.741100 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fed05f3_ad75_4044_ad92_938bcdea9fcc.slice/crio-337d31c6dce743f09d42dfbed9bfd0df15878260fbbe2fbec88bfe7bcd460b05 WatchSource:0}: Error finding container 337d31c6dce743f09d42dfbed9bfd0df15878260fbbe2fbec88bfe7bcd460b05: Status 404 returned error can't find the container with id 337d31c6dce743f09d42dfbed9bfd0df15878260fbbe2fbec88bfe7bcd460b05 Oct 30 05:48:23.480745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:23.480701 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn" event={"ID":"9fed05f3-ad75-4044-ad92-938bcdea9fcc","Type":"ContainerStarted","Data":"e0f889efaa8d1c11536470e19a0cec59db1b88108dc418ca1be9f4f9039c60ba"} Oct 30 05:48:23.481142 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:23.480775 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn" event={"ID":"9fed05f3-ad75-4044-ad92-938bcdea9fcc","Type":"ContainerStarted","Data":"337d31c6dce743f09d42dfbed9bfd0df15878260fbbe2fbec88bfe7bcd460b05"} Oct 30 05:48:23.481142 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:23.480805 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn" Oct 30 05:48:23.558789 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:23.558740 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn" podStartSLOduration=1.55872677 podStartE2EDuration="1.55872677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:23.557708964 +0000 UTC m=+3828.085475750" watchObservedRunningTime="2026-10-30 05:48:23.55872677 +0000 UTC m=+3828.086493556" Oct 30 05:48:25.985133 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:48:25.985097 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:48:34.489284 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:34.489238 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mswsn" Oct 30 05:48:38.027575 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:38.027539 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr"] Oct 30 05:48:38.029931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:38.029909 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-whddr"] Oct 30 05:48:39.579336 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:39.579296 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e3e63a2-bef9-492a-939d-37d408ddf38b" path="/var/lib/kubelet/pods/5e3e63a2-bef9-492a-939d-37d408ddf38b/volumes" Oct 30 05:48:44.738011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:44.737475 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:48:44.738011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:44.737547 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92" podUID="b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:44.876381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:44.875493 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:48:44.876381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:44.875597 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s" podUID="35dc1c80-11e5-4647-b911-bf84b7e3336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:45.236279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:45.236241 6587 scope.go:122] "RemoveContainer" containerID="583fa83b5882193857a754e5c7a5bfb5b70e167c8633f5fd46470604cc256624" Oct 30 05:48:45.250463 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:45.250431 6587 scope.go:122] "RemoveContainer" containerID="7176c5be6a960e1f40e0224a76ab44dfab8c757e69875e25777b3fd99f109e7c" Oct 30 05:48:45.262036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:45.262006 6587 scope.go:122] "RemoveContainer" containerID="2eceafad4caf206a4f3c611629e0cf2f33aa1db49880ce96c323e988ac16f686" Oct 30 05:48:46.382405 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.382368 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf"] Oct 30 05:48:46.387608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.387587 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" Oct 30 05:48:46.405679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.405651 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf"] Oct 30 05:48:46.417325 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.417298 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:48:46.417475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.417326 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:48:46.417475 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.417385 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:48:46.418074 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.418056 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:48:46.511886 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.511851 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6qzhf\" (UID: \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" Oct 30 05:48:46.512065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.511921 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k8q8\" (UniqueName: \"kubernetes.io/projected/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-kube-api-access-4k8q8\") pod \"maas-api-revoke-sub-d576e68b-6qzhf\" (UID: \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" Oct 30 05:48:46.512065 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.512017 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6qzhf\" (UID: \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" Oct 30 05:48:46.612788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.612748 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6qzhf\" (UID: \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" Oct 30 05:48:46.612962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.612871 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k8q8\" (UniqueName: \"kubernetes.io/projected/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-kube-api-access-4k8q8\") pod \"maas-api-revoke-sub-d576e68b-6qzhf\" (UID: \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" Oct 30 05:48:46.612962 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.612924 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6qzhf\" (UID: \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" Oct 30 05:48:46.613490 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.613467 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6qzhf\" (UID: \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" Oct 30 05:48:46.615264 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.615241 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6qzhf\" (UID: \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" Oct 30 05:48:46.634537 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.634484 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k8q8\" (UniqueName: \"kubernetes.io/projected/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-kube-api-access-4k8q8\") pod \"maas-api-revoke-sub-d576e68b-6qzhf\" (UID: \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" Oct 30 05:48:46.700389 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.700352 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" Oct 30 05:48:46.997931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:46.997899 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf"] Oct 30 05:48:46.998453 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:48:46.998424 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75e5980_a585_499c_9cde_2c3bf5c8a3d0.slice/crio-5b45e681423746db5f33cada925daf1aee1532e9a4eb5d04571bc47ce9af982f WatchSource:0}: Error finding container 5b45e681423746db5f33cada925daf1aee1532e9a4eb5d04571bc47ce9af982f: Status 404 returned error can't find the container with id 5b45e681423746db5f33cada925daf1aee1532e9a4eb5d04571bc47ce9af982f Oct 30 05:48:47.621563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:47.621469 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" event={"ID":"e75e5980-a585-499c-9cde-2c3bf5c8a3d0","Type":"ContainerStarted","Data":"911b8e12019671cf0c6d428ccbfbff485a7446586b11822bf77c6add449e363c"} Oct 30 05:48:47.621563 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:47.621536 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" event={"ID":"e75e5980-a585-499c-9cde-2c3bf5c8a3d0","Type":"ContainerStarted","Data":"5b45e681423746db5f33cada925daf1aee1532e9a4eb5d04571bc47ce9af982f"} Oct 30 05:48:47.664095 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:47.664045 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" podStartSLOduration=1.664033417 podStartE2EDuration="1.664033417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:47.663871291 +0000 UTC m=+3852.191638081" watchObservedRunningTime="2026-10-30 05:48:47.664033417 +0000 UTC m=+3852.191800203" Oct 30 05:48:48.628867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:48.628832 6587 generic.go:356] "Generic (PLEG): container finished" podID="e75e5980-a585-499c-9cde-2c3bf5c8a3d0" containerID="911b8e12019671cf0c6d428ccbfbff485a7446586b11822bf77c6add449e363c" exitCode=0 Oct 30 05:48:48.629295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:48.628914 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" event={"ID":"e75e5980-a585-499c-9cde-2c3bf5c8a3d0","Type":"ContainerDied","Data":"911b8e12019671cf0c6d428ccbfbff485a7446586b11822bf77c6add449e363c"} Oct 30 05:48:49.770830 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:49.770805 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" Oct 30 05:48:49.841737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:49.841705 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-xks-ca\") pod \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\" (UID: \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\") " Oct 30 05:48:49.843865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:49.843830 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-xks-ca" pod "e75e5980-a585-499c-9cde-2c3bf5c8a3d0" (UID: "e75e5980-a585-499c-9cde-2c3bf5c8a3d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:48:49.942572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:49.942503 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-openshift-ca\") pod \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\" (UID: \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\") " Oct 30 05:48:49.942572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:49.942553 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-kube-api-access-4k8q8\" (UniqueName: \"kubernetes.io/projected/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-kube-api-access-4k8q8\") pod \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\" (UID: \"e75e5980-a585-499c-9cde-2c3bf5c8a3d0\") " Oct 30 05:48:49.942899 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:49.942877 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-openshift-ca" pod "e75e5980-a585-499c-9cde-2c3bf5c8a3d0" (UID: "e75e5980-a585-499c-9cde-2c3bf5c8a3d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:48:49.942972 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:49.942959 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:48:49.944479 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:49.944461 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-kube-api-access-4k8q8" pod "e75e5980-a585-499c-9cde-2c3bf5c8a3d0" (UID: "e75e5980-a585-499c-9cde-2c3bf5c8a3d0"). InnerVolumeSpecName "kube-api-access-4k8q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:48:50.044261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:50.044231 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:48:50.044261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:50.044257 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k8q8\" (UniqueName: \"kubernetes.io/projected/e75e5980-a585-499c-9cde-2c3bf5c8a3d0-kube-api-access-4k8q8\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:48:50.640856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:50.640824 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" Oct 30 05:48:50.641031 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:50.640824 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf" event={"ID":"e75e5980-a585-499c-9cde-2c3bf5c8a3d0","Type":"ContainerDied","Data":"5b45e681423746db5f33cada925daf1aee1532e9a4eb5d04571bc47ce9af982f"} Oct 30 05:48:50.641031 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:50.640951 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b45e681423746db5f33cada925daf1aee1532e9a4eb5d04571bc47ce9af982f" Oct 30 05:48:55.985259 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:48:55.985209 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:48:57.027102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:57.027066 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc"] Oct 30 05:48:57.040773 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:57.040738 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8lpbc"] Oct 30 05:48:57.579650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:48:57.579618 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c447b67-fa0e-415d-a412-54ac74499dbe" path="/var/lib/kubelet/pods/7c447b67-fa0e-415d-a412-54ac74499dbe/volumes" Oct 30 05:49:02.911878 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:02.911842 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw"] Oct 30 05:49:02.912494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:02.912476 6587 cpu_manager.go:414] "containerMap: removing container" podUID="e75e5980-a585-499c-9cde-2c3bf5c8a3d0" containerName="revoke-keys" Oct 30 05:49:02.912552 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:02.912497 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e75e5980-a585-499c-9cde-2c3bf5c8a3d0" containerName="revoke-keys" Oct 30 05:49:02.912629 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:02.912618 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="e75e5980-a585-499c-9cde-2c3bf5c8a3d0" containerName="revoke-keys" Oct 30 05:49:02.917460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:02.917437 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" Oct 30 05:49:02.934026 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:02.934002 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw"] Oct 30 05:49:02.945646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:02.945622 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:02.945646 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:02.945630 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:02.945814 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:02.945630 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:02.947434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:02.947417 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:49:03.005552 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:03.005526 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vmdzw\" (UID: \"b4045253-968d-46c8-89e1-62bcefb3c222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" Oct 30 05:49:03.005698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:03.005565 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87k98\" (UniqueName: \"kubernetes.io/projected/b4045253-968d-46c8-89e1-62bcefb3c222-kube-api-access-87k98\") pod \"maas-api-revoke-sub-d2bc93bc-vmdzw\" (UID: \"b4045253-968d-46c8-89e1-62bcefb3c222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" Oct 30 05:49:03.005698 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:03.005588 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vmdzw\" (UID: \"b4045253-968d-46c8-89e1-62bcefb3c222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" Oct 30 05:49:03.106399 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:03.106338 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vmdzw\" (UID: \"b4045253-968d-46c8-89e1-62bcefb3c222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" Oct 30 05:49:03.106568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:03.106425 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87k98\" (UniqueName: \"kubernetes.io/projected/b4045253-968d-46c8-89e1-62bcefb3c222-kube-api-access-87k98\") pod \"maas-api-revoke-sub-d2bc93bc-vmdzw\" (UID: \"b4045253-968d-46c8-89e1-62bcefb3c222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" Oct 30 05:49:03.106568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:03.106462 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vmdzw\" (UID: \"b4045253-968d-46c8-89e1-62bcefb3c222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" Oct 30 05:49:03.107033 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:03.107012 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vmdzw\" (UID: \"b4045253-968d-46c8-89e1-62bcefb3c222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" Oct 30 05:49:03.108901 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:03.108873 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vmdzw\" (UID: \"b4045253-968d-46c8-89e1-62bcefb3c222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" Oct 30 05:49:03.127859 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:03.127832 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87k98\" (UniqueName: \"kubernetes.io/projected/b4045253-968d-46c8-89e1-62bcefb3c222-kube-api-access-87k98\") pod \"maas-api-revoke-sub-d2bc93bc-vmdzw\" (UID: \"b4045253-968d-46c8-89e1-62bcefb3c222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" Oct 30 05:49:03.230722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:03.230636 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" Oct 30 05:49:03.534001 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:03.533969 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw"] Oct 30 05:49:03.534891 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:49:03.534862 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4045253_968d_46c8_89e1_62bcefb3c222.slice/crio-2e5c01423372c74cd353cbdb8258f6c48de287411c8c4996f3dd7d8d5a15fe50 WatchSource:0}: Error finding container 2e5c01423372c74cd353cbdb8258f6c48de287411c8c4996f3dd7d8d5a15fe50: Status 404 returned error can't find the container with id 2e5c01423372c74cd353cbdb8258f6c48de287411c8c4996f3dd7d8d5a15fe50 Oct 30 05:49:03.715478 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:03.715441 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" event={"ID":"b4045253-968d-46c8-89e1-62bcefb3c222","Type":"ContainerStarted","Data":"2e5c01423372c74cd353cbdb8258f6c48de287411c8c4996f3dd7d8d5a15fe50"} Oct 30 05:49:04.723727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:04.723119 6587 generic.go:356] "Generic (PLEG): container finished" podID="b4045253-968d-46c8-89e1-62bcefb3c222" containerID="3d4949ec6ffb351759cdb450a065ebec5fb2129f9ee94bd37b3fa6501107fe34" exitCode=0 Oct 30 05:49:04.723727 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:04.723256 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" event={"ID":"b4045253-968d-46c8-89e1-62bcefb3c222","Type":"ContainerDied","Data":"3d4949ec6ffb351759cdb450a065ebec5fb2129f9ee94bd37b3fa6501107fe34"} Oct 30 05:49:05.863739 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:05.863712 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" Oct 30 05:49:05.933175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:05.933141 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-openshift-ca\") pod \"b4045253-968d-46c8-89e1-62bcefb3c222\" (UID: \"b4045253-968d-46c8-89e1-62bcefb3c222\") " Oct 30 05:49:05.933407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:05.933200 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-xks-ca\") pod \"b4045253-968d-46c8-89e1-62bcefb3c222\" (UID: \"b4045253-968d-46c8-89e1-62bcefb3c222\") " Oct 30 05:49:05.933407 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:05.933249 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4045253-968d-46c8-89e1-62bcefb3c222-kube-api-access-87k98\" (UniqueName: \"kubernetes.io/projected/b4045253-968d-46c8-89e1-62bcefb3c222-kube-api-access-87k98\") pod \"b4045253-968d-46c8-89e1-62bcefb3c222\" (UID: \"b4045253-968d-46c8-89e1-62bcefb3c222\") " Oct 30 05:49:05.933580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:05.933553 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-openshift-ca" pod "b4045253-968d-46c8-89e1-62bcefb3c222" (UID: "b4045253-968d-46c8-89e1-62bcefb3c222"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:05.935449 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:05.935413 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4045253-968d-46c8-89e1-62bcefb3c222-kube-api-access-87k98" pod "b4045253-968d-46c8-89e1-62bcefb3c222" (UID: "b4045253-968d-46c8-89e1-62bcefb3c222"). InnerVolumeSpecName "kube-api-access-87k98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:05.935574 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:05.935518 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-xks-ca" pod "b4045253-968d-46c8-89e1-62bcefb3c222" (UID: "b4045253-968d-46c8-89e1-62bcefb3c222"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:06.034843 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:06.034777 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87k98\" (UniqueName: \"kubernetes.io/projected/b4045253-968d-46c8-89e1-62bcefb3c222-kube-api-access-87k98\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:49:06.034843 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:06.034804 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:49:06.034843 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:06.034815 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4045253-968d-46c8-89e1-62bcefb3c222-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:49:06.735549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:06.735519 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" Oct 30 05:49:06.735549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:06.735540 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw" event={"ID":"b4045253-968d-46c8-89e1-62bcefb3c222","Type":"ContainerDied","Data":"2e5c01423372c74cd353cbdb8258f6c48de287411c8c4996f3dd7d8d5a15fe50"} Oct 30 05:49:06.735770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:06.735585 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e5c01423372c74cd353cbdb8258f6c48de287411c8c4996f3dd7d8d5a15fe50" Oct 30 05:49:19.035058 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:19.034983 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg"] Oct 30 05:49:19.049578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:19.049549 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hwkcg"] Oct 30 05:49:19.581791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:19.581756 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1" path="/var/lib/kubelet/pods/5915e19d-7f5f-43eb-bcc3-95f18e0f1ee1/volumes" Oct 30 05:49:20.238041 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.237998 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn"] Oct 30 05:49:20.238665 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.238643 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b4045253-968d-46c8-89e1-62bcefb3c222" containerName="revoke-keys" Oct 30 05:49:20.238761 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.238667 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4045253-968d-46c8-89e1-62bcefb3c222" containerName="revoke-keys" Oct 30 05:49:20.238828 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.238776 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4045253-968d-46c8-89e1-62bcefb3c222" containerName="revoke-keys" Oct 30 05:49:20.242185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.242163 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" Oct 30 05:49:20.263227 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.263202 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn"] Oct 30 05:49:20.274587 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.274562 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-dcbb35\"" Oct 30 05:49:20.274701 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.274609 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-6c526358-dockercfg-nkrpq\"" Oct 30 05:49:20.332774 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.332741 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655"] Oct 30 05:49:20.337349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.337326 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" Oct 30 05:49:20.356307 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.356279 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655"] Oct 30 05:49:20.391270 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.391237 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ac1d9d-065a-4871-a410-3c3488dee3e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn\" (UID: \"98ac1d9d-065a-4871-a410-3c3488dee3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" Oct 30 05:49:20.391494 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.391308 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrgbk\" (UniqueName: \"kubernetes.io/projected/98ac1d9d-065a-4871-a410-3c3488dee3e6-kube-api-access-xrgbk\") pod \"payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn\" (UID: \"98ac1d9d-065a-4871-a410-3c3488dee3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" Oct 30 05:49:20.492688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.492602 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40adfc11-d6a9-4829-869b-78b98769298c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655\" (UID: \"40adfc11-d6a9-4829-869b-78b98769298c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" Oct 30 05:49:20.492688 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.492665 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ac1d9d-065a-4871-a410-3c3488dee3e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn\" (UID: \"98ac1d9d-065a-4871-a410-3c3488dee3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" Oct 30 05:49:20.492892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.492734 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnvd\" (UniqueName: \"kubernetes.io/projected/40adfc11-d6a9-4829-869b-78b98769298c-kube-api-access-bxnvd\") pod \"payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655\" (UID: \"40adfc11-d6a9-4829-869b-78b98769298c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" Oct 30 05:49:20.492892 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.492809 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrgbk\" (UniqueName: \"kubernetes.io/projected/98ac1d9d-065a-4871-a410-3c3488dee3e6-kube-api-access-xrgbk\") pod \"payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn\" (UID: \"98ac1d9d-065a-4871-a410-3c3488dee3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" Oct 30 05:49:20.493509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.493488 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ac1d9d-065a-4871-a410-3c3488dee3e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn\" (UID: \"98ac1d9d-065a-4871-a410-3c3488dee3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" Oct 30 05:49:20.513648 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.513620 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrgbk\" (UniqueName: \"kubernetes.io/projected/98ac1d9d-065a-4871-a410-3c3488dee3e6-kube-api-access-xrgbk\") pod \"payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn\" (UID: \"98ac1d9d-065a-4871-a410-3c3488dee3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" Oct 30 05:49:20.555394 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.555342 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" Oct 30 05:49:20.593573 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.593519 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnvd\" (UniqueName: \"kubernetes.io/projected/40adfc11-d6a9-4829-869b-78b98769298c-kube-api-access-bxnvd\") pod \"payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655\" (UID: \"40adfc11-d6a9-4829-869b-78b98769298c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" Oct 30 05:49:20.593758 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.593740 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40adfc11-d6a9-4829-869b-78b98769298c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655\" (UID: \"40adfc11-d6a9-4829-869b-78b98769298c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" Oct 30 05:49:20.594661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.594620 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40adfc11-d6a9-4829-869b-78b98769298c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655\" (UID: \"40adfc11-d6a9-4829-869b-78b98769298c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" Oct 30 05:49:20.627633 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.627599 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnvd\" (UniqueName: \"kubernetes.io/projected/40adfc11-d6a9-4829-869b-78b98769298c-kube-api-access-bxnvd\") pod \"payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655\" (UID: \"40adfc11-d6a9-4829-869b-78b98769298c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" Oct 30 05:49:20.654544 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.654322 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" Oct 30 05:49:20.856975 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.856941 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn"] Oct 30 05:49:20.857710 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:49:20.857680 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ac1d9d_065a_4871_a410_3c3488dee3e6.slice/crio-afddf5a2c2721ac206554792917c049545f10a0381deb22d706d8cfa53251c72 WatchSource:0}: Error finding container afddf5a2c2721ac206554792917c049545f10a0381deb22d706d8cfa53251c72: Status 404 returned error can't find the container with id afddf5a2c2721ac206554792917c049545f10a0381deb22d706d8cfa53251c72 Oct 30 05:49:20.954845 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:49:20.954815 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40adfc11_d6a9_4829_869b_78b98769298c.slice/crio-54689877ce7e365c1a74721e79c4fb4bf0ee3bb9eea67d49a9fd23f9a9941b0c WatchSource:0}: Error finding container 54689877ce7e365c1a74721e79c4fb4bf0ee3bb9eea67d49a9fd23f9a9941b0c: Status 404 returned error can't find the container with id 54689877ce7e365c1a74721e79c4fb4bf0ee3bb9eea67d49a9fd23f9a9941b0c Oct 30 05:49:20.955831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:20.955804 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655"] Oct 30 05:49:21.823112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:21.823075 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" event={"ID":"98ac1d9d-065a-4871-a410-3c3488dee3e6","Type":"ContainerStarted","Data":"3ac6dc45393fb9a318478ddf4446cbccfcbfb8b7d0a570b99752bd54e9bd7fea"} Oct 30 05:49:21.823650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:21.823138 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" event={"ID":"98ac1d9d-065a-4871-a410-3c3488dee3e6","Type":"ContainerStarted","Data":"afddf5a2c2721ac206554792917c049545f10a0381deb22d706d8cfa53251c72"} Oct 30 05:49:21.823650 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:21.823167 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" Oct 30 05:49:21.824917 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:21.824887 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" event={"ID":"40adfc11-d6a9-4829-869b-78b98769298c","Type":"ContainerStarted","Data":"7fcd09aa49d61b0295eb2f1029b221d7b849db657ab6566a4ad59aa17528021c"} Oct 30 05:49:21.825051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:21.824942 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" event={"ID":"40adfc11-d6a9-4829-869b-78b98769298c","Type":"ContainerStarted","Data":"54689877ce7e365c1a74721e79c4fb4bf0ee3bb9eea67d49a9fd23f9a9941b0c"} Oct 30 05:49:21.825051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:21.824981 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" Oct 30 05:49:21.864809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:21.864751 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" podStartSLOduration=1.864737637 podStartE2EDuration="1.864737637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:21.863949715 +0000 UTC m=+3886.391716514" watchObservedRunningTime="2026-10-30 05:49:21.864737637 +0000 UTC m=+3886.392504427" Oct 30 05:49:21.905418 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:21.905344 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" podStartSLOduration=1.905330422 podStartE2EDuration="1.905330422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:21.904748403 +0000 UTC m=+3886.432515211" watchObservedRunningTime="2026-10-30 05:49:21.905330422 +0000 UTC m=+3886.433097208" Oct 30 05:49:25.985503 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:49:25.985462 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:49:32.832693 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:32.832663 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" Oct 30 05:49:32.833132 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:32.832729 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" Oct 30 05:49:36.417673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:36.417643 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:49:36.419501 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:36.419474 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:49:37.035147 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:37.035111 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc"] Oct 30 05:49:37.048997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:37.048961 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfgtc"] Oct 30 05:49:37.580344 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:37.580310 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a97a66e-144b-43de-8f61-481d5406fa42" path="/var/lib/kubelet/pods/5a97a66e-144b-43de-8f61-481d5406fa42/volumes" Oct 30 05:49:45.301953 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:45.301917 6587 scope.go:122] "RemoveContainer" containerID="a9a543f423977d7f7e225a9aff4624e2c317902b0584dc6b78606b876158cd27" Oct 30 05:49:45.310801 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:45.310777 6587 scope.go:122] "RemoveContainer" containerID="74527878621b75a8c13942bc50345a845c0bf92f0846c3ffc20d0f55ef9c0482" Oct 30 05:49:45.319536 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:45.319503 6587 scope.go:122] "RemoveContainer" containerID="554a66e8f51a9168c0dfd1a548a5ef9cd6842f2e98ad64c87815dd79a220e9ca" Oct 30 05:49:54.789737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.789702 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62"] Oct 30 05:49:54.795306 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.795287 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" Oct 30 05:49:54.812800 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.812773 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62"] Oct 30 05:49:54.824349 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.824321 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:54.824568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.824322 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:54.824568 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.824321 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:54.825185 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.825162 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:49:54.853850 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.853815 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8bgs\" (UniqueName: \"kubernetes.io/projected/1fd7220c-7093-4843-b68e-e6c1d75758f4-kube-api-access-b8bgs\") pod \"maas-api-revoke-sub-9e021154-4qc62\" (UID: \"1fd7220c-7093-4843-b68e-e6c1d75758f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" Oct 30 05:49:54.854045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.853889 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e021154-4qc62\" (UID: \"1fd7220c-7093-4843-b68e-e6c1d75758f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" Oct 30 05:49:54.854045 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.853947 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e021154-4qc62\" (UID: \"1fd7220c-7093-4843-b68e-e6c1d75758f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" Oct 30 05:49:54.955016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.954981 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e021154-4qc62\" (UID: \"1fd7220c-7093-4843-b68e-e6c1d75758f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" Oct 30 05:49:54.955016 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.955019 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e021154-4qc62\" (UID: \"1fd7220c-7093-4843-b68e-e6c1d75758f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" Oct 30 05:49:54.955281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.955118 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8bgs\" (UniqueName: \"kubernetes.io/projected/1fd7220c-7093-4843-b68e-e6c1d75758f4-kube-api-access-b8bgs\") pod \"maas-api-revoke-sub-9e021154-4qc62\" (UID: \"1fd7220c-7093-4843-b68e-e6c1d75758f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" Oct 30 05:49:54.955709 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.955680 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e021154-4qc62\" (UID: \"1fd7220c-7093-4843-b68e-e6c1d75758f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" Oct 30 05:49:54.957545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.957513 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e021154-4qc62\" (UID: \"1fd7220c-7093-4843-b68e-e6c1d75758f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" Oct 30 05:49:54.975094 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:54.975063 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8bgs\" (UniqueName: \"kubernetes.io/projected/1fd7220c-7093-4843-b68e-e6c1d75758f4-kube-api-access-b8bgs\") pod \"maas-api-revoke-sub-9e021154-4qc62\" (UID: \"1fd7220c-7093-4843-b68e-e6c1d75758f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" Oct 30 05:49:55.109421 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:55.109382 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" Oct 30 05:49:55.413608 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:55.413576 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62"] Oct 30 05:49:55.414262 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:49:55.414234 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd7220c_7093_4843_b68e_e6c1d75758f4.slice/crio-516e220ecab8448f37dba056899fbf3950634bc14123d61a5ab3d31c03834c7f WatchSource:0}: Error finding container 516e220ecab8448f37dba056899fbf3950634bc14123d61a5ab3d31c03834c7f: Status 404 returned error can't find the container with id 516e220ecab8448f37dba056899fbf3950634bc14123d61a5ab3d31c03834c7f Oct 30 05:49:55.985959 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:49:55.985874 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:49:56.003034 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:56.003004 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" event={"ID":"1fd7220c-7093-4843-b68e-e6c1d75758f4","Type":"ContainerStarted","Data":"06265fc152c05462c5b6767fd36fdfeada7c1ede58f03fa3c48b345a9a48fcdc"} Oct 30 05:49:56.003182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:56.003057 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" event={"ID":"1fd7220c-7093-4843-b68e-e6c1d75758f4","Type":"ContainerStarted","Data":"516e220ecab8448f37dba056899fbf3950634bc14123d61a5ab3d31c03834c7f"} Oct 30 05:49:56.044989 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:56.044934 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" podStartSLOduration=2.044920687 podStartE2EDuration="2.044920687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:56.04360387 +0000 UTC m=+3920.571370657" watchObservedRunningTime="2026-10-30 05:49:56.044920687 +0000 UTC m=+3920.572687472" Oct 30 05:49:57.010056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:57.010021 6587 generic.go:356] "Generic (PLEG): container finished" podID="1fd7220c-7093-4843-b68e-e6c1d75758f4" containerID="06265fc152c05462c5b6767fd36fdfeada7c1ede58f03fa3c48b345a9a48fcdc" exitCode=0 Oct 30 05:49:57.010506 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:57.010103 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" event={"ID":"1fd7220c-7093-4843-b68e-e6c1d75758f4","Type":"ContainerDied","Data":"06265fc152c05462c5b6767fd36fdfeada7c1ede58f03fa3c48b345a9a48fcdc"} Oct 30 05:49:58.157247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:58.157212 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" Oct 30 05:49:58.290931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:58.290839 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-openshift-ca\") pod \"1fd7220c-7093-4843-b68e-e6c1d75758f4\" (UID: \"1fd7220c-7093-4843-b68e-e6c1d75758f4\") " Oct 30 05:49:58.290931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:58.290891 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-xks-ca\") pod \"1fd7220c-7093-4843-b68e-e6c1d75758f4\" (UID: \"1fd7220c-7093-4843-b68e-e6c1d75758f4\") " Oct 30 05:49:58.291184 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:58.290994 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd7220c-7093-4843-b68e-e6c1d75758f4-kube-api-access-b8bgs\" (UniqueName: \"kubernetes.io/projected/1fd7220c-7093-4843-b68e-e6c1d75758f4-kube-api-access-b8bgs\") pod \"1fd7220c-7093-4843-b68e-e6c1d75758f4\" (UID: \"1fd7220c-7093-4843-b68e-e6c1d75758f4\") " Oct 30 05:49:58.291277 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:58.291248 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-openshift-ca" pod "1fd7220c-7093-4843-b68e-e6c1d75758f4" (UID: "1fd7220c-7093-4843-b68e-e6c1d75758f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:58.291452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:58.291434 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:49:58.293111 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:58.293090 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-xks-ca" pod "1fd7220c-7093-4843-b68e-e6c1d75758f4" (UID: "1fd7220c-7093-4843-b68e-e6c1d75758f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:58.293194 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:58.293091 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd7220c-7093-4843-b68e-e6c1d75758f4-kube-api-access-b8bgs" pod "1fd7220c-7093-4843-b68e-e6c1d75758f4" (UID: "1fd7220c-7093-4843-b68e-e6c1d75758f4"). InnerVolumeSpecName "kube-api-access-b8bgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:58.392454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:58.392415 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd7220c-7093-4843-b68e-e6c1d75758f4-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:49:58.392454 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:58.392449 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8bgs\" (UniqueName: \"kubernetes.io/projected/1fd7220c-7093-4843-b68e-e6c1d75758f4-kube-api-access-b8bgs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:49:59.022749 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:59.022712 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" event={"ID":"1fd7220c-7093-4843-b68e-e6c1d75758f4","Type":"ContainerDied","Data":"516e220ecab8448f37dba056899fbf3950634bc14123d61a5ab3d31c03834c7f"} Oct 30 05:49:59.022973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:59.022751 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62" Oct 30 05:49:59.022973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:49:59.022764 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="516e220ecab8448f37dba056899fbf3950634bc14123d61a5ab3d31c03834c7f" Oct 30 05:50:05.759158 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.759118 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv"] Oct 30 05:50:05.759820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.759591 6587 cpu_manager.go:414] "containerMap: removing container" podUID="1fd7220c-7093-4843-b68e-e6c1d75758f4" containerName="revoke-keys" Oct 30 05:50:05.759820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.759605 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd7220c-7093-4843-b68e-e6c1d75758f4" containerName="revoke-keys" Oct 30 05:50:05.759820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.759680 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd7220c-7093-4843-b68e-e6c1d75758f4" containerName="revoke-keys" Oct 30 05:50:05.762816 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.762795 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" Oct 30 05:50:05.779473 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.779447 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:50:05.779616 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.779452 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:50:05.793206 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.793179 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:50:05.793521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.793506 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:50:05.795414 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.795388 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv"] Oct 30 05:50:05.876909 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.876876 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv\" (UID: \"f21c16e1-240e-43cc-8acf-3b0009854c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" Oct 30 05:50:05.877092 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.876923 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g9wx\" (UniqueName: \"kubernetes.io/projected/f21c16e1-240e-43cc-8acf-3b0009854c99-kube-api-access-7g9wx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv\" (UID: \"f21c16e1-240e-43cc-8acf-3b0009854c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" Oct 30 05:50:05.877092 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.877057 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv\" (UID: \"f21c16e1-240e-43cc-8acf-3b0009854c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" Oct 30 05:50:05.978268 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.978233 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv\" (UID: \"f21c16e1-240e-43cc-8acf-3b0009854c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" Oct 30 05:50:05.978492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.978294 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv\" (UID: \"f21c16e1-240e-43cc-8acf-3b0009854c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" Oct 30 05:50:05.978492 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.978341 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g9wx\" (UniqueName: \"kubernetes.io/projected/f21c16e1-240e-43cc-8acf-3b0009854c99-kube-api-access-7g9wx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv\" (UID: \"f21c16e1-240e-43cc-8acf-3b0009854c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" Oct 30 05:50:05.979036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.979011 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv\" (UID: \"f21c16e1-240e-43cc-8acf-3b0009854c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" Oct 30 05:50:05.980797 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.980779 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv\" (UID: \"f21c16e1-240e-43cc-8acf-3b0009854c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" Oct 30 05:50:05.999323 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:05.999294 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g9wx\" (UniqueName: \"kubernetes.io/projected/f21c16e1-240e-43cc-8acf-3b0009854c99-kube-api-access-7g9wx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv\" (UID: \"f21c16e1-240e-43cc-8acf-3b0009854c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" Oct 30 05:50:06.076453 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:06.076419 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" Oct 30 05:50:06.380676 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:06.380077 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv"] Oct 30 05:50:06.381567 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:50:06.381513 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21c16e1_240e_43cc_8acf_3b0009854c99.slice/crio-48140e0669e090a938e954bca3cf2bc55719528c16f358083b5a61879d7f83c1 WatchSource:0}: Error finding container 48140e0669e090a938e954bca3cf2bc55719528c16f358083b5a61879d7f83c1: Status 404 returned error can't find the container with id 48140e0669e090a938e954bca3cf2bc55719528c16f358083b5a61879d7f83c1 Oct 30 05:50:07.066968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:07.066927 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" event={"ID":"f21c16e1-240e-43cc-8acf-3b0009854c99","Type":"ContainerStarted","Data":"4c2bb2261bc3ed091d4bb12b325644579eac1f06c4b9055485e3ec0463a5088b"} Oct 30 05:50:07.067429 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:07.067019 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" event={"ID":"f21c16e1-240e-43cc-8acf-3b0009854c99","Type":"ContainerStarted","Data":"48140e0669e090a938e954bca3cf2bc55719528c16f358083b5a61879d7f83c1"} Oct 30 05:50:07.111295 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:07.111235 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" podStartSLOduration=2.111221187 podStartE2EDuration="2.111221187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:07.10960143 +0000 UTC m=+3931.637368216" watchObservedRunningTime="2026-10-30 05:50:07.111221187 +0000 UTC m=+3931.638988005" Oct 30 05:50:08.073550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:08.073515 6587 generic.go:356] "Generic (PLEG): container finished" podID="f21c16e1-240e-43cc-8acf-3b0009854c99" containerID="4c2bb2261bc3ed091d4bb12b325644579eac1f06c4b9055485e3ec0463a5088b" exitCode=0 Oct 30 05:50:08.073995 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:08.073560 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" event={"ID":"f21c16e1-240e-43cc-8acf-3b0009854c99","Type":"ContainerDied","Data":"4c2bb2261bc3ed091d4bb12b325644579eac1f06c4b9055485e3ec0463a5088b"} Oct 30 05:50:09.230661 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:09.230629 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" Oct 30 05:50:09.304072 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:09.304037 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-openshift-ca\") pod \"f21c16e1-240e-43cc-8acf-3b0009854c99\" (UID: \"f21c16e1-240e-43cc-8acf-3b0009854c99\") " Oct 30 05:50:09.304244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:09.304140 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21c16e1-240e-43cc-8acf-3b0009854c99-kube-api-access-7g9wx\" (UniqueName: \"kubernetes.io/projected/f21c16e1-240e-43cc-8acf-3b0009854c99-kube-api-access-7g9wx\") pod \"f21c16e1-240e-43cc-8acf-3b0009854c99\" (UID: \"f21c16e1-240e-43cc-8acf-3b0009854c99\") " Oct 30 05:50:09.304244 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:09.304203 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-xks-ca\") pod \"f21c16e1-240e-43cc-8acf-3b0009854c99\" (UID: \"f21c16e1-240e-43cc-8acf-3b0009854c99\") " Oct 30 05:50:09.304509 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:09.304485 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-openshift-ca" pod "f21c16e1-240e-43cc-8acf-3b0009854c99" (UID: "f21c16e1-240e-43cc-8acf-3b0009854c99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:09.304609 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:09.304591 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:50:09.306306 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:09.306271 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-xks-ca" pod "f21c16e1-240e-43cc-8acf-3b0009854c99" (UID: "f21c16e1-240e-43cc-8acf-3b0009854c99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:09.306460 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:09.306407 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21c16e1-240e-43cc-8acf-3b0009854c99-kube-api-access-7g9wx" pod "f21c16e1-240e-43cc-8acf-3b0009854c99" (UID: "f21c16e1-240e-43cc-8acf-3b0009854c99"). InnerVolumeSpecName "kube-api-access-7g9wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:09.405854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:09.405763 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21c16e1-240e-43cc-8acf-3b0009854c99-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:50:09.405854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:09.405793 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g9wx\" (UniqueName: \"kubernetes.io/projected/f21c16e1-240e-43cc-8acf-3b0009854c99-kube-api-access-7g9wx\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:50:10.084957 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:10.084921 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" Oct 30 05:50:10.085150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:10.084954 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv" event={"ID":"f21c16e1-240e-43cc-8acf-3b0009854c99","Type":"ContainerDied","Data":"48140e0669e090a938e954bca3cf2bc55719528c16f358083b5a61879d7f83c1"} Oct 30 05:50:10.085150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:10.085018 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48140e0669e090a938e954bca3cf2bc55719528c16f358083b5a61879d7f83c1" Oct 30 05:50:16.291140 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:16.291109 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655"] Oct 30 05:50:16.323111 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:16.323079 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn"] Oct 30 05:50:17.928830 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:17.928792 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv"] Oct 30 05:50:17.942540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:17.942507 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcbb35-cbvxv"] Oct 30 05:50:19.580029 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:19.579995 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21c16e1-240e-43cc-8acf-3b0009854c99" path="/var/lib/kubelet/pods/f21c16e1-240e-43cc-8acf-3b0009854c99/volumes" Oct 30 05:50:21.291444 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:21.291390 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" podUID="40adfc11-d6a9-4829-869b-78b98769298c" containerName="payload-processing" containerID="cri-o://7fcd09aa49d61b0295eb2f1029b221d7b849db657ab6566a4ad59aa17528021c" gracePeriod=25 Oct 30 05:50:21.323643 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:21.323600 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" podUID="98ac1d9d-065a-4871-a410-3c3488dee3e6" containerName="payload-pre-processing" containerID="cri-o://3ac6dc45393fb9a318478ddf4446cbccfcbfb8b7d0a570b99752bd54e9bd7fea" gracePeriod=25 Oct 30 05:50:22.831577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:22.831530 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" podUID="98ac1d9d-065a-4871-a410-3c3488dee3e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:50:22.831985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:22.831581 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" podUID="40adfc11-d6a9-4829-869b-78b98769298c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:50:25.988884 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:50:25.988840 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:50:32.853383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:32.850485 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" podUID="40adfc11-d6a9-4829-869b-78b98769298c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:50:32.853383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:32.851126 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" podUID="98ac1d9d-065a-4871-a410-3c3488dee3e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:50:41.575855 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.575831 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" Oct 30 05:50:41.604209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.604185 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" Oct 30 05:50:41.679467 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.679439 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ac1d9d-065a-4871-a410-3c3488dee3e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ac1d9d-065a-4871-a410-3c3488dee3e6-plugins-config-volume\") pod \"98ac1d9d-065a-4871-a410-3c3488dee3e6\" (UID: \"98ac1d9d-065a-4871-a410-3c3488dee3e6\") " Oct 30 05:50:41.679651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.679477 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ac1d9d-065a-4871-a410-3c3488dee3e6-kube-api-access-xrgbk\" (UniqueName: \"kubernetes.io/projected/98ac1d9d-065a-4871-a410-3c3488dee3e6-kube-api-access-xrgbk\") pod \"98ac1d9d-065a-4871-a410-3c3488dee3e6\" (UID: \"98ac1d9d-065a-4871-a410-3c3488dee3e6\") " Oct 30 05:50:41.679651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.679498 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40adfc11-d6a9-4829-869b-78b98769298c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40adfc11-d6a9-4829-869b-78b98769298c-plugins-config-volume\") pod \"40adfc11-d6a9-4829-869b-78b98769298c\" (UID: \"40adfc11-d6a9-4829-869b-78b98769298c\") " Oct 30 05:50:41.679651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.679527 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40adfc11-d6a9-4829-869b-78b98769298c-kube-api-access-bxnvd\" (UniqueName: \"kubernetes.io/projected/40adfc11-d6a9-4829-869b-78b98769298c-kube-api-access-bxnvd\") pod \"40adfc11-d6a9-4829-869b-78b98769298c\" (UID: \"40adfc11-d6a9-4829-869b-78b98769298c\") " Oct 30 05:50:41.679997 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.679965 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40adfc11-d6a9-4829-869b-78b98769298c-plugins-config-volume" pod "40adfc11-d6a9-4829-869b-78b98769298c" (UID: "40adfc11-d6a9-4829-869b-78b98769298c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:41.680141 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.680021 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ac1d9d-065a-4871-a410-3c3488dee3e6-plugins-config-volume" pod "98ac1d9d-065a-4871-a410-3c3488dee3e6" (UID: "98ac1d9d-065a-4871-a410-3c3488dee3e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:41.681684 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.681664 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ac1d9d-065a-4871-a410-3c3488dee3e6-kube-api-access-xrgbk" pod "98ac1d9d-065a-4871-a410-3c3488dee3e6" (UID: "98ac1d9d-065a-4871-a410-3c3488dee3e6"). InnerVolumeSpecName "kube-api-access-xrgbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:41.681826 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.681740 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40adfc11-d6a9-4829-869b-78b98769298c-kube-api-access-bxnvd" pod "40adfc11-d6a9-4829-869b-78b98769298c" (UID: "40adfc11-d6a9-4829-869b-78b98769298c"). InnerVolumeSpecName "kube-api-access-bxnvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:41.781423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.781393 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ac1d9d-065a-4871-a410-3c3488dee3e6-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:50:41.781423 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.781418 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrgbk\" (UniqueName: \"kubernetes.io/projected/98ac1d9d-065a-4871-a410-3c3488dee3e6-kube-api-access-xrgbk\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:50:41.781619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.781430 6587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40adfc11-d6a9-4829-869b-78b98769298c-plugins-config-volume\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:50:41.781619 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:41.781442 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxnvd\" (UniqueName: \"kubernetes.io/projected/40adfc11-d6a9-4829-869b-78b98769298c-kube-api-access-bxnvd\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:50:42.306286 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.306246 6587 generic.go:356] "Generic (PLEG): container finished" podID="40adfc11-d6a9-4829-869b-78b98769298c" containerID="7fcd09aa49d61b0295eb2f1029b221d7b849db657ab6566a4ad59aa17528021c" exitCode=0 Oct 30 05:50:42.306502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.306315 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" Oct 30 05:50:42.306502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.306397 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" event={"ID":"40adfc11-d6a9-4829-869b-78b98769298c","Type":"ContainerDied","Data":"7fcd09aa49d61b0295eb2f1029b221d7b849db657ab6566a4ad59aa17528021c"} Oct 30 05:50:42.306502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.306481 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655" event={"ID":"40adfc11-d6a9-4829-869b-78b98769298c","Type":"ContainerDied","Data":"54689877ce7e365c1a74721e79c4fb4bf0ee3bb9eea67d49a9fd23f9a9941b0c"} Oct 30 05:50:42.306714 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.306509 6587 scope.go:122] "RemoveContainer" containerID="7fcd09aa49d61b0295eb2f1029b221d7b849db657ab6566a4ad59aa17528021c" Oct 30 05:50:42.308049 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.308020 6587 generic.go:356] "Generic (PLEG): container finished" podID="98ac1d9d-065a-4871-a410-3c3488dee3e6" containerID="3ac6dc45393fb9a318478ddf4446cbccfcbfb8b7d0a570b99752bd54e9bd7fea" exitCode=0 Oct 30 05:50:42.308197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.308084 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" Oct 30 05:50:42.308197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.308091 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" event={"ID":"98ac1d9d-065a-4871-a410-3c3488dee3e6","Type":"ContainerDied","Data":"3ac6dc45393fb9a318478ddf4446cbccfcbfb8b7d0a570b99752bd54e9bd7fea"} Oct 30 05:50:42.308197 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.308129 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn" event={"ID":"98ac1d9d-065a-4871-a410-3c3488dee3e6","Type":"ContainerDied","Data":"afddf5a2c2721ac206554792917c049545f10a0381deb22d706d8cfa53251c72"} Oct 30 05:50:42.317715 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.317696 6587 scope.go:122] "RemoveContainer" containerID="7fcd09aa49d61b0295eb2f1029b221d7b849db657ab6566a4ad59aa17528021c" Oct 30 05:50:42.318011 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:50:42.317986 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fcd09aa49d61b0295eb2f1029b221d7b849db657ab6566a4ad59aa17528021c\": container with ID starting with 7fcd09aa49d61b0295eb2f1029b221d7b849db657ab6566a4ad59aa17528021c not found: ID does not exist" containerID="7fcd09aa49d61b0295eb2f1029b221d7b849db657ab6566a4ad59aa17528021c" Oct 30 05:50:42.318119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.318018 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fcd09aa49d61b0295eb2f1029b221d7b849db657ab6566a4ad59aa17528021c"} err="failed to get container status \"7fcd09aa49d61b0295eb2f1029b221d7b849db657ab6566a4ad59aa17528021c\": rpc error: code = NotFound desc = could not find container \"7fcd09aa49d61b0295eb2f1029b221d7b849db657ab6566a4ad59aa17528021c\": container with ID starting with 7fcd09aa49d61b0295eb2f1029b221d7b849db657ab6566a4ad59aa17528021c not found: ID does not exist" Oct 30 05:50:42.318119 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.318040 6587 scope.go:122] "RemoveContainer" containerID="3ac6dc45393fb9a318478ddf4446cbccfcbfb8b7d0a570b99752bd54e9bd7fea" Oct 30 05:50:42.326285 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.326265 6587 scope.go:122] "RemoveContainer" containerID="3ac6dc45393fb9a318478ddf4446cbccfcbfb8b7d0a570b99752bd54e9bd7fea" Oct 30 05:50:42.326551 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:50:42.326532 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac6dc45393fb9a318478ddf4446cbccfcbfb8b7d0a570b99752bd54e9bd7fea\": container with ID starting with 3ac6dc45393fb9a318478ddf4446cbccfcbfb8b7d0a570b99752bd54e9bd7fea not found: ID does not exist" containerID="3ac6dc45393fb9a318478ddf4446cbccfcbfb8b7d0a570b99752bd54e9bd7fea" Oct 30 05:50:42.326607 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.326559 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac6dc45393fb9a318478ddf4446cbccfcbfb8b7d0a570b99752bd54e9bd7fea"} err="failed to get container status \"3ac6dc45393fb9a318478ddf4446cbccfcbfb8b7d0a570b99752bd54e9bd7fea\": rpc error: code = NotFound desc = could not find container \"3ac6dc45393fb9a318478ddf4446cbccfcbfb8b7d0a570b99752bd54e9bd7fea\": container with ID starting with 3ac6dc45393fb9a318478ddf4446cbccfcbfb8b7d0a570b99752bd54e9bd7fea not found: ID does not exist" Oct 30 05:50:42.362159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.362134 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn"] Oct 30 05:50:42.375744 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.375721 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn"] Oct 30 05:50:42.403495 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.403469 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655"] Oct 30 05:50:42.407681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:42.407656 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655"] Oct 30 05:50:43.579580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:43.579543 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40adfc11-d6a9-4829-869b-78b98769298c" path="/var/lib/kubelet/pods/40adfc11-d6a9-4829-869b-78b98769298c/volumes" Oct 30 05:50:43.580086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:43.580068 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ac1d9d-065a-4871-a410-3c3488dee3e6" path="/var/lib/kubelet/pods/98ac1d9d-065a-4871-a410-3c3488dee3e6/volumes" Oct 30 05:50:45.347385 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:45.347331 6587 scope.go:122] "RemoveContainer" containerID="5894c90154868d8ac5673c036a163fc7da8046080604d138eb73732ad12eb44f" Oct 30 05:50:45.355991 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:45.355969 6587 scope.go:122] "RemoveContainer" containerID="2767906cf4389c89b3f60d301152bfb696b76ba32de9f5d27bc06d93a0bdce0c" Oct 30 05:50:45.363625 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:45.363606 6587 scope.go:122] "RemoveContainer" containerID="99819d997a72972ea8e4a3596fd4e4bb17c771938be2cb1ef1e83e268721fceb" Oct 30 05:50:45.370942 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:50:45.370925 6587 scope.go:122] "RemoveContainer" containerID="7402fafd32af299b21db77879ea4d00e1802ca5d4139d7c00dc6798a551b0d1d" Oct 30 05:50:55.984890 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:50:55.984805 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:51:04.840881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.840831 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84"] Oct 30 05:51:04.841556 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.841534 6587 cpu_manager.go:414] "containerMap: removing container" podUID="98ac1d9d-065a-4871-a410-3c3488dee3e6" containerName="payload-pre-processing" Oct 30 05:51:04.841624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.841559 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ac1d9d-065a-4871-a410-3c3488dee3e6" containerName="payload-pre-processing" Oct 30 05:51:04.841624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.841595 6587 cpu_manager.go:414] "containerMap: removing container" podUID="40adfc11-d6a9-4829-869b-78b98769298c" containerName="payload-processing" Oct 30 05:51:04.841624 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.841605 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40adfc11-d6a9-4829-869b-78b98769298c" containerName="payload-processing" Oct 30 05:51:04.841734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.841630 6587 cpu_manager.go:414] "containerMap: removing container" podUID="f21c16e1-240e-43cc-8acf-3b0009854c99" containerName="revoke-keys" Oct 30 05:51:04.841734 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.841639 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21c16e1-240e-43cc-8acf-3b0009854c99" containerName="revoke-keys" Oct 30 05:51:04.841799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.841742 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ac1d9d-065a-4871-a410-3c3488dee3e6" containerName="payload-pre-processing" Oct 30 05:51:04.841799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.841759 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="40adfc11-d6a9-4829-869b-78b98769298c" containerName="payload-processing" Oct 30 05:51:04.841799 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.841772 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21c16e1-240e-43cc-8acf-3b0009854c99" containerName="revoke-keys" Oct 30 05:51:04.847447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.847422 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" Oct 30 05:51:04.862137 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.862110 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84"] Oct 30 05:51:04.873831 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.873785 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:04.874069 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.873834 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:51:04.874293 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.874264 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:51:04.877231 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.877210 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:51:04.973896 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.973855 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-r4w84\" (UID: \"54764729-5ace-4cc0-a648-e35813ed2968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" Oct 30 05:51:04.974099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.973934 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkkqw\" (UniqueName: \"kubernetes.io/projected/54764729-5ace-4cc0-a648-e35813ed2968-kube-api-access-dkkqw\") pod \"maas-api-revoke-sub-452b6d7b-r4w84\" (UID: \"54764729-5ace-4cc0-a648-e35813ed2968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" Oct 30 05:51:04.974099 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:04.974082 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-r4w84\" (UID: \"54764729-5ace-4cc0-a648-e35813ed2968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" Oct 30 05:51:05.074938 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:05.074899 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-r4w84\" (UID: \"54764729-5ace-4cc0-a648-e35813ed2968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" Oct 30 05:51:05.075155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:05.074959 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkkqw\" (UniqueName: \"kubernetes.io/projected/54764729-5ace-4cc0-a648-e35813ed2968-kube-api-access-dkkqw\") pod \"maas-api-revoke-sub-452b6d7b-r4w84\" (UID: \"54764729-5ace-4cc0-a648-e35813ed2968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" Oct 30 05:51:05.075155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:05.075001 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-r4w84\" (UID: \"54764729-5ace-4cc0-a648-e35813ed2968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" Oct 30 05:51:05.075867 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:05.075835 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-r4w84\" (UID: \"54764729-5ace-4cc0-a648-e35813ed2968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" Oct 30 05:51:05.077659 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:05.077624 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-r4w84\" (UID: \"54764729-5ace-4cc0-a648-e35813ed2968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" Oct 30 05:51:05.096256 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:05.096180 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkkqw\" (UniqueName: \"kubernetes.io/projected/54764729-5ace-4cc0-a648-e35813ed2968-kube-api-access-dkkqw\") pod \"maas-api-revoke-sub-452b6d7b-r4w84\" (UID: \"54764729-5ace-4cc0-a648-e35813ed2968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" Oct 30 05:51:05.164550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:05.164509 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" Oct 30 05:51:05.513539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:05.513437 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84"] Oct 30 05:51:06.491279 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:06.491239 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" event={"ID":"54764729-5ace-4cc0-a648-e35813ed2968","Type":"ContainerStarted","Data":"13b5b3d16711221b32bfc263158ff6bbd9c501f44a604db99a8aaee89b57f5ec"} Oct 30 05:51:06.491922 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:06.491894 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" event={"ID":"54764729-5ace-4cc0-a648-e35813ed2968","Type":"ContainerStarted","Data":"4e6125c550ba6e5afaba32af4bb98662837ae0c04ab0aa92342abb3a9df278e2"} Oct 30 05:51:06.543383 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:06.539828 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" podStartSLOduration=2.539808119 podStartE2EDuration="2.539808119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:06.538759491 +0000 UTC m=+3991.066526277" watchObservedRunningTime="2026-10-30 05:51:06.539808119 +0000 UTC m=+3991.067574904" Oct 30 05:51:07.499199 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:07.499146 6587 generic.go:356] "Generic (PLEG): container finished" podID="54764729-5ace-4cc0-a648-e35813ed2968" containerID="13b5b3d16711221b32bfc263158ff6bbd9c501f44a604db99a8aaee89b57f5ec" exitCode=0 Oct 30 05:51:07.499811 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:07.499276 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" event={"ID":"54764729-5ace-4cc0-a648-e35813ed2968","Type":"ContainerDied","Data":"13b5b3d16711221b32bfc263158ff6bbd9c501f44a604db99a8aaee89b57f5ec"} Oct 30 05:51:08.655603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:08.655578 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" Oct 30 05:51:08.740994 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:08.740952 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-openshift-ca\") pod \"54764729-5ace-4cc0-a648-e35813ed2968\" (UID: \"54764729-5ace-4cc0-a648-e35813ed2968\") " Oct 30 05:51:08.741182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:08.741024 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-xks-ca\") pod \"54764729-5ace-4cc0-a648-e35813ed2968\" (UID: \"54764729-5ace-4cc0-a648-e35813ed2968\") " Oct 30 05:51:08.741182 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:08.741107 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54764729-5ace-4cc0-a648-e35813ed2968-kube-api-access-dkkqw\" (UniqueName: \"kubernetes.io/projected/54764729-5ace-4cc0-a648-e35813ed2968-kube-api-access-dkkqw\") pod \"54764729-5ace-4cc0-a648-e35813ed2968\" (UID: \"54764729-5ace-4cc0-a648-e35813ed2968\") " Oct 30 05:51:08.741400 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:08.741348 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-openshift-ca" pod "54764729-5ace-4cc0-a648-e35813ed2968" (UID: "54764729-5ace-4cc0-a648-e35813ed2968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:08.741505 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:08.741449 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:51:08.743424 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:08.743394 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-xks-ca" pod "54764729-5ace-4cc0-a648-e35813ed2968" (UID: "54764729-5ace-4cc0-a648-e35813ed2968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:08.743545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:08.743419 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54764729-5ace-4cc0-a648-e35813ed2968-kube-api-access-dkkqw" pod "54764729-5ace-4cc0-a648-e35813ed2968" (UID: "54764729-5ace-4cc0-a648-e35813ed2968"). InnerVolumeSpecName "kube-api-access-dkkqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:08.842269 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:08.842230 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkkqw\" (UniqueName: \"kubernetes.io/projected/54764729-5ace-4cc0-a648-e35813ed2968-kube-api-access-dkkqw\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:51:08.842497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:08.842274 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54764729-5ace-4cc0-a648-e35813ed2968-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:51:09.510809 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:09.510776 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" Oct 30 05:51:09.510985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:09.510812 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r4w84" event={"ID":"54764729-5ace-4cc0-a648-e35813ed2968","Type":"ContainerDied","Data":"4e6125c550ba6e5afaba32af4bb98662837ae0c04ab0aa92342abb3a9df278e2"} Oct 30 05:51:09.510985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:09.510899 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e6125c550ba6e5afaba32af4bb98662837ae0c04ab0aa92342abb3a9df278e2" Oct 30 05:51:20.905527 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:20.905471 6587 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:51:20.906198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:20.905543 6587 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt" podUID="ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:22.054263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.054229 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5"] Oct 30 05:51:22.054719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.054703 6587 cpu_manager.go:414] "containerMap: removing container" podUID="54764729-5ace-4cc0-a648-e35813ed2968" containerName="revoke-keys" Oct 30 05:51:22.054719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.054719 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54764729-5ace-4cc0-a648-e35813ed2968" containerName="revoke-keys" Oct 30 05:51:22.054829 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.054776 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="54764729-5ace-4cc0-a648-e35813ed2968" containerName="revoke-keys" Oct 30 05:51:22.058017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.057995 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" Oct 30 05:51:22.089458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.089412 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:51:22.089657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.089482 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:22.089657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.089551 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:51:22.089657 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.089614 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:51:22.091540 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.091513 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5"] Oct 30 05:51:22.140246 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.140210 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-lm6g5\" (UID: \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" Oct 30 05:51:22.140461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.140332 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfkpz\" (UniqueName: \"kubernetes.io/projected/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-kube-api-access-zfkpz\") pod \"maas-api-revoke-sub-3e31a506-lm6g5\" (UID: \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" Oct 30 05:51:22.140461 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.140397 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-lm6g5\" (UID: \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" Oct 30 05:51:22.242240 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.242170 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfkpz\" (UniqueName: \"kubernetes.io/projected/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-kube-api-access-zfkpz\") pod \"maas-api-revoke-sub-3e31a506-lm6g5\" (UID: \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" Oct 30 05:51:22.242550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.242258 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-lm6g5\" (UID: \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" Oct 30 05:51:22.242550 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.242395 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-lm6g5\" (UID: \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" Oct 30 05:51:22.243338 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.243293 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-lm6g5\" (UID: \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" Oct 30 05:51:22.245865 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.245829 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-lm6g5\" (UID: \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" Oct 30 05:51:22.273288 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.273174 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfkpz\" (UniqueName: \"kubernetes.io/projected/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-kube-api-access-zfkpz\") pod \"maas-api-revoke-sub-3e31a506-lm6g5\" (UID: \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" Oct 30 05:51:22.372214 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.372099 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" Oct 30 05:51:22.695737 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:22.695702 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5"] Oct 30 05:51:22.696899 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:51:22.696859 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f05ec3_abfc_4b5f_9099_0e1d53d7de60.slice/crio-e9d047b9e0586a975e7d5a16783014a72ad73f174d8f17b516cd3b0da8ba0526 WatchSource:0}: Error finding container e9d047b9e0586a975e7d5a16783014a72ad73f174d8f17b516cd3b0da8ba0526: Status 404 returned error can't find the container with id e9d047b9e0586a975e7d5a16783014a72ad73f174d8f17b516cd3b0da8ba0526 Oct 30 05:51:23.612547 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:23.612499 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" event={"ID":"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60","Type":"ContainerStarted","Data":"546a74fdb2418c013b5ccc64cc558afa7ca0fb15b9ade509e55aa3258f087709"} Oct 30 05:51:23.613053 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:23.612567 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" event={"ID":"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60","Type":"ContainerStarted","Data":"e9d047b9e0586a975e7d5a16783014a72ad73f174d8f17b516cd3b0da8ba0526"} Oct 30 05:51:23.667442 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:23.659928 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" podStartSLOduration=1.659906079 podStartE2EDuration="1.659906079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:23.659631905 +0000 UTC m=+4008.187398731" watchObservedRunningTime="2026-10-30 05:51:23.659906079 +0000 UTC m=+4008.187672869" Oct 30 05:51:24.622419 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:24.622351 6587 generic.go:356] "Generic (PLEG): container finished" podID="b7f05ec3-abfc-4b5f-9099-0e1d53d7de60" containerID="546a74fdb2418c013b5ccc64cc558afa7ca0fb15b9ade509e55aa3258f087709" exitCode=0 Oct 30 05:51:24.623090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:24.623045 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" event={"ID":"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60","Type":"ContainerDied","Data":"546a74fdb2418c013b5ccc64cc558afa7ca0fb15b9ade509e55aa3258f087709"} Oct 30 05:51:25.764654 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:25.764625 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" Oct 30 05:51:25.904150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:25.904066 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-openshift-ca\") pod \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\" (UID: \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\") " Oct 30 05:51:25.904150 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:25.904122 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-xks-ca\") pod \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\" (UID: \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\") " Oct 30 05:51:25.904428 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:25.904247 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-kube-api-access-zfkpz\" (UniqueName: \"kubernetes.io/projected/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-kube-api-access-zfkpz\") pod \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\" (UID: \"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60\") " Oct 30 05:51:25.904553 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:25.904526 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-openshift-ca" pod "b7f05ec3-abfc-4b5f-9099-0e1d53d7de60" (UID: "b7f05ec3-abfc-4b5f-9099-0e1d53d7de60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:25.906261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:25.906237 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-xks-ca" pod "b7f05ec3-abfc-4b5f-9099-0e1d53d7de60" (UID: "b7f05ec3-abfc-4b5f-9099-0e1d53d7de60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:25.906462 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:25.906436 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-kube-api-access-zfkpz" pod "b7f05ec3-abfc-4b5f-9099-0e1d53d7de60" (UID: "b7f05ec3-abfc-4b5f-9099-0e1d53d7de60"). InnerVolumeSpecName "kube-api-access-zfkpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:25.985085 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:51:25.985049 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:51:26.005263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:26.005237 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfkpz\" (UniqueName: \"kubernetes.io/projected/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-kube-api-access-zfkpz\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:51:26.005263 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:26.005261 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:51:26.005474 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:26.005274 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f05ec3-abfc-4b5f-9099-0e1d53d7de60-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:51:26.634078 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:26.634046 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" Oct 30 05:51:26.634297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:26.634093 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lm6g5" event={"ID":"b7f05ec3-abfc-4b5f-9099-0e1d53d7de60","Type":"ContainerDied","Data":"e9d047b9e0586a975e7d5a16783014a72ad73f174d8f17b516cd3b0da8ba0526"} Oct 30 05:51:26.634297 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:26.634138 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9d047b9e0586a975e7d5a16783014a72ad73f174d8f17b516cd3b0da8ba0526" Oct 30 05:51:31.028900 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:31.028860 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth"] Oct 30 05:51:31.042112 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:31.042078 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031e2e8e-b6gth"] Oct 30 05:51:31.579640 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:31.579607 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad1d0a6-6183-4446-83b8-3d869446f1a9" path="/var/lib/kubelet/pods/6ad1d0a6-6183-4446-83b8-3d869446f1a9/volumes" Oct 30 05:51:45.410433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.410400 6587 scope.go:122] "RemoveContainer" containerID="7080dd6ba891736110a911d09e27df0943bcadb27dd7319a24278f5db848ba89" Oct 30 05:51:45.517434 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.517382 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w"] Oct 30 05:51:45.517918 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.517895 6587 cpu_manager.go:414] "containerMap: removing container" podUID="b7f05ec3-abfc-4b5f-9099-0e1d53d7de60" containerName="revoke-keys" Oct 30 05:51:45.518026 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.517920 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f05ec3-abfc-4b5f-9099-0e1d53d7de60" containerName="revoke-keys" Oct 30 05:51:45.518088 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.518052 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f05ec3-abfc-4b5f-9099-0e1d53d7de60" containerName="revoke-keys" Oct 30 05:51:45.522545 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.522521 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" Oct 30 05:51:45.540805 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.540778 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w"] Oct 30 05:51:45.552190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.552162 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:51:45.552190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.552180 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\"" Oct 30 05:51:45.552190 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.552177 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:51:45.552437 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.552207 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:45.659983 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.659948 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-628fe216-cl64w\" (UID: \"22db4268-6b00-4424-952d-4187a0bb8879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" Oct 30 05:51:45.660154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.659991 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-628fe216-cl64w\" (UID: \"22db4268-6b00-4424-952d-4187a0bb8879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" Oct 30 05:51:45.660154 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.660074 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29hlx\" (UniqueName: \"kubernetes.io/projected/22db4268-6b00-4424-952d-4187a0bb8879-kube-api-access-29hlx\") pod \"maas-api-revoke-sub-628fe216-cl64w\" (UID: \"22db4268-6b00-4424-952d-4187a0bb8879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" Oct 30 05:51:45.761565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.761469 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-628fe216-cl64w\" (UID: \"22db4268-6b00-4424-952d-4187a0bb8879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" Oct 30 05:51:45.761565 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.761530 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-628fe216-cl64w\" (UID: \"22db4268-6b00-4424-952d-4187a0bb8879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" Oct 30 05:51:45.761791 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.761585 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29hlx\" (UniqueName: \"kubernetes.io/projected/22db4268-6b00-4424-952d-4187a0bb8879-kube-api-access-29hlx\") pod \"maas-api-revoke-sub-628fe216-cl64w\" (UID: \"22db4268-6b00-4424-952d-4187a0bb8879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" Oct 30 05:51:45.762278 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.762257 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-628fe216-cl64w\" (UID: \"22db4268-6b00-4424-952d-4187a0bb8879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" Oct 30 05:51:45.763876 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.763859 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-628fe216-cl64w\" (UID: \"22db4268-6b00-4424-952d-4187a0bb8879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" Oct 30 05:51:45.783696 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.783667 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29hlx\" (UniqueName: \"kubernetes.io/projected/22db4268-6b00-4424-952d-4187a0bb8879-kube-api-access-29hlx\") pod \"maas-api-revoke-sub-628fe216-cl64w\" (UID: \"22db4268-6b00-4424-952d-4187a0bb8879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" Oct 30 05:51:45.835425 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:45.835383 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" Oct 30 05:51:46.131120 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:51:46.131067 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22db4268_6b00_4424_952d_4187a0bb8879.slice/crio-f82ce46f5d557b08fe8344e1f78d6e58f3dbc28c97f9daf069e1af9c59cbed40 WatchSource:0}: Error finding container f82ce46f5d557b08fe8344e1f78d6e58f3dbc28c97f9daf069e1af9c59cbed40: Status 404 returned error can't find the container with id f82ce46f5d557b08fe8344e1f78d6e58f3dbc28c97f9daf069e1af9c59cbed40 Oct 30 05:51:46.131120 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:46.131109 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w"] Oct 30 05:51:46.745128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:46.745040 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" event={"ID":"22db4268-6b00-4424-952d-4187a0bb8879","Type":"ContainerStarted","Data":"dbed524f8f5b06d3bb9a56c64e75230fa5b3216ad2c7a692c34c8a13fba87f0f"} Oct 30 05:51:46.745128 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:46.745092 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" event={"ID":"22db4268-6b00-4424-952d-4187a0bb8879","Type":"ContainerStarted","Data":"f82ce46f5d557b08fe8344e1f78d6e58f3dbc28c97f9daf069e1af9c59cbed40"} Oct 30 05:51:46.787687 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:46.787634 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" podStartSLOduration=1.787621298 podStartE2EDuration="1.787621298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:46.785886272 +0000 UTC m=+4031.313653058" watchObservedRunningTime="2026-10-30 05:51:46.787621298 +0000 UTC m=+4031.315388084" Oct 30 05:51:47.751984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:47.751939 6587 generic.go:356] "Generic (PLEG): container finished" podID="22db4268-6b00-4424-952d-4187a0bb8879" containerID="dbed524f8f5b06d3bb9a56c64e75230fa5b3216ad2c7a692c34c8a13fba87f0f" exitCode=0 Oct 30 05:51:47.751984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:47.751977 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" event={"ID":"22db4268-6b00-4424-952d-4187a0bb8879","Type":"ContainerDied","Data":"dbed524f8f5b06d3bb9a56c64e75230fa5b3216ad2c7a692c34c8a13fba87f0f"} Oct 30 05:51:48.901578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:48.901551 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" Oct 30 05:51:48.993778 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:48.993746 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-openshift-ca\") pod \"22db4268-6b00-4424-952d-4187a0bb8879\" (UID: \"22db4268-6b00-4424-952d-4187a0bb8879\") " Oct 30 05:51:48.993948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:48.993838 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22db4268-6b00-4424-952d-4187a0bb8879-kube-api-access-29hlx\" (UniqueName: \"kubernetes.io/projected/22db4268-6b00-4424-952d-4187a0bb8879-kube-api-access-29hlx\") pod \"22db4268-6b00-4424-952d-4187a0bb8879\" (UID: \"22db4268-6b00-4424-952d-4187a0bb8879\") " Oct 30 05:51:48.993948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:48.993937 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-xks-ca\") pod \"22db4268-6b00-4424-952d-4187a0bb8879\" (UID: \"22db4268-6b00-4424-952d-4187a0bb8879\") " Oct 30 05:51:48.994174 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:48.994152 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-openshift-ca" pod "22db4268-6b00-4424-952d-4187a0bb8879" (UID: "22db4268-6b00-4424-952d-4187a0bb8879"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:48.994324 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:48.994298 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-openshift-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:51:48.995939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:48.995913 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-xks-ca" pod "22db4268-6b00-4424-952d-4187a0bb8879" (UID: "22db4268-6b00-4424-952d-4187a0bb8879"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:48.996044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:48.995964 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22db4268-6b00-4424-952d-4187a0bb8879-kube-api-access-29hlx" pod "22db4268-6b00-4424-952d-4187a0bb8879" (UID: "22db4268-6b00-4424-952d-4187a0bb8879"). InnerVolumeSpecName "kube-api-access-29hlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:49.095299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:49.095267 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29hlx\" (UniqueName: \"kubernetes.io/projected/22db4268-6b00-4424-952d-4187a0bb8879-kube-api-access-29hlx\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:51:49.095299 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:49.095297 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22db4268-6b00-4424-952d-4187a0bb8879-maas-api-xks-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:51:49.764618 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:49.764583 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" Oct 30 05:51:49.764787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:49.764617 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628fe216-cl64w" event={"ID":"22db4268-6b00-4424-952d-4187a0bb8879","Type":"ContainerDied","Data":"f82ce46f5d557b08fe8344e1f78d6e58f3dbc28c97f9daf069e1af9c59cbed40"} Oct 30 05:51:49.764787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:49.764670 6587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f82ce46f5d557b08fe8344e1f78d6e58f3dbc28c97f9daf069e1af9c59cbed40" Oct 30 05:51:55.985212 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:51:55.985171 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:51:57.987242 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:57.987204 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-94hk5"] Oct 30 05:51:57.987898 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:57.987871 6587 cpu_manager.go:414] "containerMap: removing container" podUID="22db4268-6b00-4424-952d-4187a0bb8879" containerName="revoke-keys" Oct 30 05:51:57.988021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:57.987898 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22db4268-6b00-4424-952d-4187a0bb8879" containerName="revoke-keys" Oct 30 05:51:57.988021 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:57.988004 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="22db4268-6b00-4424-952d-4187a0bb8879" containerName="revoke-keys" Oct 30 05:51:57.990980 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:57.990958 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.030549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.023779 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:51:58.030549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.023803 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-grc2l\"" Oct 30 05:51:58.030549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.023822 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:58.030549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.023877 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 05:51:58.030549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.023939 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 05:51:58.030549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.026154 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-94hk5"] Oct 30 05:51:58.095440 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.095396 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c9a279-0047-40ec-8985-f2c5c25bbb22-openshift-service-ca\") pod \"maas-api-5c48877c75-94hk5\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.095641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.095518 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-maas-api-tls\") pod \"maas-api-5c48877c75-94hk5\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.095641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.095564 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-metrics-certs\") pod \"maas-api-5c48877c75-94hk5\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.095641 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.095604 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kptsf\" (UniqueName: \"kubernetes.io/projected/97c9a279-0047-40ec-8985-f2c5c25bbb22-kube-api-access-kptsf\") pod \"maas-api-5c48877c75-94hk5\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.196910 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.196872 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c9a279-0047-40ec-8985-f2c5c25bbb22-openshift-service-ca\") pod \"maas-api-5c48877c75-94hk5\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.197102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.196931 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-maas-api-tls\") pod \"maas-api-5c48877c75-94hk5\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.197102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.196963 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-metrics-certs\") pod \"maas-api-5c48877c75-94hk5\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.197102 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.196996 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kptsf\" (UniqueName: \"kubernetes.io/projected/97c9a279-0047-40ec-8985-f2c5c25bbb22-kube-api-access-kptsf\") pod \"maas-api-5c48877c75-94hk5\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.197586 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.197564 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c9a279-0047-40ec-8985-f2c5c25bbb22-openshift-service-ca\") pod \"maas-api-5c48877c75-94hk5\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.199403 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.199379 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-maas-api-tls\") pod \"maas-api-5c48877c75-94hk5\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.199594 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.199570 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-metrics-certs\") pod \"maas-api-5c48877c75-94hk5\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.218012 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.217981 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kptsf\" (UniqueName: \"kubernetes.io/projected/97c9a279-0047-40ec-8985-f2c5c25bbb22-kube-api-access-kptsf\") pod \"maas-api-5c48877c75-94hk5\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.309413 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.309384 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:58.606808 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.606778 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-94hk5"] Oct 30 05:51:58.607399 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:51:58.607345 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c9a279_0047_40ec_8985_f2c5c25bbb22.slice/crio-fa2e053678501ab81508c917d6b6aa9326310a55890344efb4f7029c841707aa WatchSource:0}: Error finding container fa2e053678501ab81508c917d6b6aa9326310a55890344efb4f7029c841707aa: Status 404 returned error can't find the container with id fa2e053678501ab81508c917d6b6aa9326310a55890344efb4f7029c841707aa Oct 30 05:51:58.608653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.608632 6587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:51:58.809813 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:58.809780 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" event={"ID":"97c9a279-0047-40ec-8985-f2c5c25bbb22","Type":"ContainerStarted","Data":"fa2e053678501ab81508c917d6b6aa9326310a55890344efb4f7029c841707aa"} Oct 30 05:51:59.817672 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:59.817629 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" event={"ID":"97c9a279-0047-40ec-8985-f2c5c25bbb22","Type":"ContainerStarted","Data":"e16e281d9f92d6232b904bfd87d9fd3722282f60bfc967066b92bb3b93e0f65f"} Oct 30 05:51:59.818165 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:59.817699 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:51:59.863257 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:51:59.863187 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" podStartSLOduration=2.566852633 podStartE2EDuration="2.863157952s" totalImagesPullingTime="296.305319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:59.861699352 +0000 UTC m=+4044.389466133" watchObservedRunningTime="2026-10-30 05:51:59.863157952 +0000 UTC m=+4044.390924737" Oct 30 05:52:05.030488 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.030451 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd"] Oct 30 05:52:05.043956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.043921 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t4hsd"] Oct 30 05:52:05.137296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.137239 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-94hk5"] Oct 30 05:52:05.137539 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.137488 6587 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" podUID="97c9a279-0047-40ec-8985-f2c5c25bbb22" containerName="maas-api" containerID="cri-o://e16e281d9f92d6232b904bfd87d9fd3722282f60bfc967066b92bb3b93e0f65f" gracePeriod=30 Oct 30 05:52:05.143015 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.142989 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:52:05.169066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.169038 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-jm72k"] Oct 30 05:52:05.172653 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.172636 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.193700 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.193661 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-jm72k"] Oct 30 05:52:05.269685 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.269649 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aa73fe6-99f9-41df-85da-4cb4aa561edd-openshift-service-ca\") pod \"maas-api-595cc648c-jm72k\" (UID: \"6aa73fe6-99f9-41df-85da-4cb4aa561edd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.269841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.269691 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aa73fe6-99f9-41df-85da-4cb4aa561edd-metrics-certs\") pod \"maas-api-595cc648c-jm72k\" (UID: \"6aa73fe6-99f9-41df-85da-4cb4aa561edd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.269841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.269786 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aa73fe6-99f9-41df-85da-4cb4aa561edd-maas-api-tls\") pod \"maas-api-595cc648c-jm72k\" (UID: \"6aa73fe6-99f9-41df-85da-4cb4aa561edd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.270037 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.270006 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g5gn\" (UniqueName: \"kubernetes.io/projected/6aa73fe6-99f9-41df-85da-4cb4aa561edd-kube-api-access-6g5gn\") pod \"maas-api-595cc648c-jm72k\" (UID: \"6aa73fe6-99f9-41df-85da-4cb4aa561edd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.371261 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.371232 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g5gn\" (UniqueName: \"kubernetes.io/projected/6aa73fe6-99f9-41df-85da-4cb4aa561edd-kube-api-access-6g5gn\") pod \"maas-api-595cc648c-jm72k\" (UID: \"6aa73fe6-99f9-41df-85da-4cb4aa561edd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.371427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.371274 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aa73fe6-99f9-41df-85da-4cb4aa561edd-openshift-service-ca\") pod \"maas-api-595cc648c-jm72k\" (UID: \"6aa73fe6-99f9-41df-85da-4cb4aa561edd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.371427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.371291 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aa73fe6-99f9-41df-85da-4cb4aa561edd-metrics-certs\") pod \"maas-api-595cc648c-jm72k\" (UID: \"6aa73fe6-99f9-41df-85da-4cb4aa561edd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.371427 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.371316 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aa73fe6-99f9-41df-85da-4cb4aa561edd-maas-api-tls\") pod \"maas-api-595cc648c-jm72k\" (UID: \"6aa73fe6-99f9-41df-85da-4cb4aa561edd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.371984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.371959 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aa73fe6-99f9-41df-85da-4cb4aa561edd-openshift-service-ca\") pod \"maas-api-595cc648c-jm72k\" (UID: \"6aa73fe6-99f9-41df-85da-4cb4aa561edd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.373862 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.373843 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aa73fe6-99f9-41df-85da-4cb4aa561edd-maas-api-tls\") pod \"maas-api-595cc648c-jm72k\" (UID: \"6aa73fe6-99f9-41df-85da-4cb4aa561edd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.373956 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.373908 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aa73fe6-99f9-41df-85da-4cb4aa561edd-metrics-certs\") pod \"maas-api-595cc648c-jm72k\" (UID: \"6aa73fe6-99f9-41df-85da-4cb4aa561edd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.391877 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.391843 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g5gn\" (UniqueName: \"kubernetes.io/projected/6aa73fe6-99f9-41df-85da-4cb4aa561edd-kube-api-access-6g5gn\") pod \"maas-api-595cc648c-jm72k\" (UID: \"6aa73fe6-99f9-41df-85da-4cb4aa561edd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.394664 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.394645 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:52:05.471973 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.471943 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c9a279-0047-40ec-8985-f2c5c25bbb22-kube-api-access-kptsf\" (UniqueName: \"kubernetes.io/projected/97c9a279-0047-40ec-8985-f2c5c25bbb22-kube-api-access-kptsf\") pod \"97c9a279-0047-40ec-8985-f2c5c25bbb22\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " Oct 30 05:52:05.472143 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.472000 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-maas-api-tls\") pod \"97c9a279-0047-40ec-8985-f2c5c25bbb22\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " Oct 30 05:52:05.472143 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.472034 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-metrics-certs\") pod \"97c9a279-0047-40ec-8985-f2c5c25bbb22\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " Oct 30 05:52:05.472143 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.472113 6587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c9a279-0047-40ec-8985-f2c5c25bbb22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c9a279-0047-40ec-8985-f2c5c25bbb22-openshift-service-ca\") pod \"97c9a279-0047-40ec-8985-f2c5c25bbb22\" (UID: \"97c9a279-0047-40ec-8985-f2c5c25bbb22\") " Oct 30 05:52:05.472605 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.472568 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c9a279-0047-40ec-8985-f2c5c25bbb22-openshift-service-ca" pod "97c9a279-0047-40ec-8985-f2c5c25bbb22" (UID: "97c9a279-0047-40ec-8985-f2c5c25bbb22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:05.474447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.474335 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-maas-api-tls" pod "97c9a279-0047-40ec-8985-f2c5c25bbb22" (UID: "97c9a279-0047-40ec-8985-f2c5c25bbb22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:52:05.474447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.474348 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-metrics-certs" pod "97c9a279-0047-40ec-8985-f2c5c25bbb22" (UID: "97c9a279-0047-40ec-8985-f2c5c25bbb22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:52:05.474447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.474409 6587 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c9a279-0047-40ec-8985-f2c5c25bbb22-kube-api-access-kptsf" pod "97c9a279-0047-40ec-8985-f2c5c25bbb22" (UID: "97c9a279-0047-40ec-8985-f2c5c25bbb22"). InnerVolumeSpecName "kube-api-access-kptsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:05.538017 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.537982 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:05.575057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.573703 6587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kptsf\" (UniqueName: \"kubernetes.io/projected/97c9a279-0047-40ec-8985-f2c5c25bbb22-kube-api-access-kptsf\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:52:05.575057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.573739 6587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-maas-api-tls\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:52:05.575057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.573751 6587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c9a279-0047-40ec-8985-f2c5c25bbb22-metrics-certs\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:52:05.575057 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.573761 6587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c9a279-0047-40ec-8985-f2c5c25bbb22-openshift-service-ca\") on node \"ip-10-0-134-105.ec2.internal\" DevicePath \"\"" Oct 30 05:52:05.587500 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.587455 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3741a835-a3bf-404b-8ec4-d3f7e53ecf19" path="/var/lib/kubelet/pods/3741a835-a3bf-404b-8ec4-d3f7e53ecf19/volumes" Oct 30 05:52:05.847726 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.847694 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-jm72k"] Oct 30 05:52:05.849682 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:52:05.849650 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa73fe6_99f9_41df_85da_4cb4aa561edd.slice/crio-f08cd3ad73c2256aa792dcf9b5c9df0d4f0377e4e0dfa50c39d08ada950c4926 WatchSource:0}: Error finding container f08cd3ad73c2256aa792dcf9b5c9df0d4f0377e4e0dfa50c39d08ada950c4926: Status 404 returned error can't find the container with id f08cd3ad73c2256aa792dcf9b5c9df0d4f0377e4e0dfa50c39d08ada950c4926 Oct 30 05:52:05.850673 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.850645 6587 generic.go:356] "Generic (PLEG): container finished" podID="97c9a279-0047-40ec-8985-f2c5c25bbb22" containerID="e16e281d9f92d6232b904bfd87d9fd3722282f60bfc967066b92bb3b93e0f65f" exitCode=0 Oct 30 05:52:05.850765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.850711 6587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" Oct 30 05:52:05.850765 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.850731 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" event={"ID":"97c9a279-0047-40ec-8985-f2c5c25bbb22","Type":"ContainerDied","Data":"e16e281d9f92d6232b904bfd87d9fd3722282f60bfc967066b92bb3b93e0f65f"} Oct 30 05:52:05.850845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.850793 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-94hk5" event={"ID":"97c9a279-0047-40ec-8985-f2c5c25bbb22","Type":"ContainerDied","Data":"fa2e053678501ab81508c917d6b6aa9326310a55890344efb4f7029c841707aa"} Oct 30 05:52:05.850845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.850826 6587 scope.go:122] "RemoveContainer" containerID="e16e281d9f92d6232b904bfd87d9fd3722282f60bfc967066b92bb3b93e0f65f" Oct 30 05:52:05.861497 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.861476 6587 scope.go:122] "RemoveContainer" containerID="e16e281d9f92d6232b904bfd87d9fd3722282f60bfc967066b92bb3b93e0f65f" Oct 30 05:52:05.861779 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:52:05.861755 6587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16e281d9f92d6232b904bfd87d9fd3722282f60bfc967066b92bb3b93e0f65f\": container with ID starting with e16e281d9f92d6232b904bfd87d9fd3722282f60bfc967066b92bb3b93e0f65f not found: ID does not exist" containerID="e16e281d9f92d6232b904bfd87d9fd3722282f60bfc967066b92bb3b93e0f65f" Oct 30 05:52:05.861861 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.861789 6587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16e281d9f92d6232b904bfd87d9fd3722282f60bfc967066b92bb3b93e0f65f"} err="failed to get container status \"e16e281d9f92d6232b904bfd87d9fd3722282f60bfc967066b92bb3b93e0f65f\": rpc error: code = NotFound desc = could not find container \"e16e281d9f92d6232b904bfd87d9fd3722282f60bfc967066b92bb3b93e0f65f\": container with ID starting with e16e281d9f92d6232b904bfd87d9fd3722282f60bfc967066b92bb3b93e0f65f not found: ID does not exist" Oct 30 05:52:05.906802 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.906774 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-94hk5"] Oct 30 05:52:05.920235 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:05.920208 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-94hk5"] Oct 30 05:52:06.858159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:06.858122 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" event={"ID":"6aa73fe6-99f9-41df-85da-4cb4aa561edd","Type":"ContainerStarted","Data":"57698b2cb234ecf0f095afa6f665dca20702e31b4a6bb2f2b6a73f1c8f2e6f4b"} Oct 30 05:52:06.858603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:06.858181 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" event={"ID":"6aa73fe6-99f9-41df-85da-4cb4aa561edd","Type":"ContainerStarted","Data":"f08cd3ad73c2256aa792dcf9b5c9df0d4f0377e4e0dfa50c39d08ada950c4926"} Oct 30 05:52:06.858603 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:06.858211 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:06.903458 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:06.903393 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" podStartSLOduration=1.5778157670000001 podStartE2EDuration="1.903374462s" totalImagesPullingTime="325.558695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:06.903248863 +0000 UTC m=+4051.431015651" watchObservedRunningTime="2026-10-30 05:52:06.903374462 +0000 UTC m=+4051.431141250" Oct 30 05:52:07.583032 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:07.582995 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c9a279-0047-40ec-8985-f2c5c25bbb22" path="/var/lib/kubelet/pods/97c9a279-0047-40ec-8985-f2c5c25bbb22/volumes" Oct 30 05:52:12.875729 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:12.875685 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-jm72k" Oct 30 05:52:25.984990 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:52:25.984947 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:52:45.434577 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:45.434539 6587 scope.go:122] "RemoveContainer" containerID="c1f9905bfc9f77001b4283c64e9435ad5e61069f3fdee0e638c5323ce00c8317" Oct 30 05:52:45.443198 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:52:45.443174 6587 scope.go:122] "RemoveContainer" containerID="431a5ba6f3c33c88a0841d6cd58e3dcbc1524bb1f0449519661de15ec418f654" Oct 30 05:52:55.984550 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:52:55.984507 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:53:00.668985 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:00.668953 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59945dd659-9f5g7_a8adbcc0-729e-4774-af4f-43d362118031/authorino/0.log" Oct 30 05:53:11.099984 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:11.099950 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-77449b8677-8jpjr_e7d449b9-933c-4431-bbf5-5376c47eb048/manager/0.log" Oct 30 05:53:12.570209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:12.570177 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-dp85z_5c2574b8-389f-46fa-bb0c-d144b95455c1/manager/0.log" Oct 30 05:53:13.053155 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:13.053127 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8484dd48c9-2v6v6_ac919fe3-ac98-44cd-8d95-6df70a406fa4/copy-manifests/0.log" Oct 30 05:53:13.104939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:13.104908 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8484dd48c9-2v6v6_ac919fe3-ac98-44cd-8d95-6df70a406fa4/manager/0.log" Oct 30 05:53:13.772745 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:13.772718 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-tw5cx_d02fa1f2-298d-4a23-a302-6a59e1cbd188/proxy/0.log" Oct 30 05:53:14.739441 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:14.739409 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-jm72k_6aa73fe6-99f9-41df-85da-4cb4aa561edd/maas-api/0.log" Oct 30 05:53:15.123010 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:15.122983 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-lm6g5_b7f05ec3-abfc-4b5f-9099-0e1d53d7de60/revoke-keys/0.log" Oct 30 05:53:15.304820 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:15.304790 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-r4w84_54764729-5ace-4cc0-a648-e35813ed2968/revoke-keys/0.log" Oct 30 05:53:15.495175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:15.495102 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-628fe216-cl64w_22db4268-6b00-4424-952d-4187a0bb8879/revoke-keys/0.log" Oct 30 05:53:15.865931 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:15.865907 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9e021154-4qc62_1fd7220c-7093-4843-b68e-e6c1d75758f4/revoke-keys/0.log" Oct 30 05:53:16.055036 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:16.055008 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-vmdzw_b4045253-968d-46c8-89e1-62bcefb3c222/revoke-keys/0.log" Oct 30 05:53:16.238947 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:16.238874 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-6qzhf_e75e5980-a585-499c-9cde-2c3bf5c8a3d0/revoke-keys/0.log" Oct 30 05:53:18.179982 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:18.179952 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59945dd659-9f5g7_a8adbcc0-729e-4774-af4f-43d362118031/authorino/0.log" Oct 30 05:53:19.039755 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:19.039722 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-mswsn_9fed05f3-ad75-4044-ad92-938bcdea9fcc/manager/0.log" Oct 30 05:53:20.633146 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:20.633115 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54668d58ff-2xllz_ddc306dd-f974-4fd8-abb7-067320d7ca83/kube-auth-proxy/0.log" Oct 30 05:53:21.224968 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:21.224941 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-52stt_ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd/istio-proxy/0.log" Oct 30 05:53:21.814318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:21.814286 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-db25s_35dc1c80-11e5-4647-b911-bf84b7e3336f/istio-proxy/0.log" Oct 30 05:53:22.187159 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:22.187088 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-jbc92_b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa/istio-proxy/0.log" Oct 30 05:53:22.978704 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:22.978667 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-p6s8d_d2ed078a-da76-4628-a77f-5b56163461d4/payload-pre-processing/0.log" Oct 30 05:53:23.172086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:23.172004 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vqsmz_a8739f61-fe1e-4c58-b535-4815d55e05f1/payload-processing/0.log" Oct 30 05:53:24.785636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:24.785605 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-dmbs8_c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7/main/0.log" Oct 30 05:53:24.987353 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:24.987321 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-cnqvb_cd43eb0e-0dca-47cd-9fda-bb4a0dbce266/main/0.log" Oct 30 05:53:25.371796 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:25.371766 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n_da31dd7c-961b-46e8-9f28-1e7e6780fa10/main/0.log" Oct 30 05:53:25.562163 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:25.562130 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59_e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b/main/0.log" Oct 30 05:53:25.984588 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:53:25.984548 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:53:26.530781 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:26.530749 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59945dd659-9f5g7_a8adbcc0-729e-4774-af4f-43d362118031/authorino/0.log" Oct 30 05:53:35.732108 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:35.732071 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-77449b8677-8jpjr_e7d449b9-933c-4431-bbf5-5376c47eb048/manager/0.log" Oct 30 05:53:37.052651 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:37.052616 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-dp85z_5c2574b8-389f-46fa-bb0c-d144b95455c1/manager/0.log" Oct 30 05:53:37.488090 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:37.488014 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8484dd48c9-2v6v6_ac919fe3-ac98-44cd-8d95-6df70a406fa4/copy-manifests/0.log" Oct 30 05:53:37.521845 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:37.521821 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8484dd48c9-2v6v6_ac919fe3-ac98-44cd-8d95-6df70a406fa4/manager/0.log" Oct 30 05:53:38.173044 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:38.173009 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-tw5cx_d02fa1f2-298d-4a23-a302-6a59e1cbd188/proxy/0.log" Oct 30 05:53:38.936524 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:38.936495 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-jm72k_6aa73fe6-99f9-41df-85da-4cb4aa561edd/maas-api/0.log" Oct 30 05:53:39.330578 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:39.330552 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-lm6g5_b7f05ec3-abfc-4b5f-9099-0e1d53d7de60/revoke-keys/0.log" Oct 30 05:53:39.522787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:39.522756 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-r4w84_54764729-5ace-4cc0-a648-e35813ed2968/revoke-keys/0.log" Oct 30 05:53:39.709767 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:39.709685 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-628fe216-cl64w_22db4268-6b00-4424-952d-4187a0bb8879/revoke-keys/0.log" Oct 30 05:53:40.071381 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:40.071335 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9e021154-4qc62_1fd7220c-7093-4843-b68e-e6c1d75758f4/revoke-keys/0.log" Oct 30 05:53:40.247296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:40.247267 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-vmdzw_b4045253-968d-46c8-89e1-62bcefb3c222/revoke-keys/0.log" Oct 30 05:53:40.436681 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:40.436598 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-6qzhf_e75e5980-a585-499c-9cde-2c3bf5c8a3d0/revoke-keys/0.log" Oct 30 05:53:41.966948 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:41.966914 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59945dd659-9f5g7_a8adbcc0-729e-4774-af4f-43d362118031/authorino/0.log" Oct 30 05:53:42.861209 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:42.861180 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-mswsn_9fed05f3-ad75-4044-ad92-938bcdea9fcc/manager/0.log" Oct 30 05:53:44.268450 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:44.268421 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54668d58ff-2xllz_ddc306dd-f974-4fd8-abb7-067320d7ca83/kube-auth-proxy/0.log" Oct 30 05:53:44.844939 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:44.844910 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-52stt_ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd/istio-proxy/0.log" Oct 30 05:53:45.452827 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:45.452801 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-db25s_35dc1c80-11e5-4647-b911-bf84b7e3336f/istio-proxy/0.log" Oct 30 05:53:45.469770 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:45.469741 6587 scope.go:122] "RemoveContainer" containerID="e757a58a1cb2f679cda07982a051a2e33f215fa3fb04fcc5e47dc9bc4c5e7c4f" Oct 30 05:53:45.851645 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:45.851609 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-jbc92_b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa/istio-proxy/0.log" Oct 30 05:53:46.610126 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:46.610095 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-p6s8d_d2ed078a-da76-4628-a77f-5b56163461d4/payload-pre-processing/0.log" Oct 30 05:53:46.797856 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:46.797775 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vqsmz_a8739f61-fe1e-4c58-b535-4815d55e05f1/payload-processing/0.log" Oct 30 05:53:47.938870 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:47.938838 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-dmbs8_c04c6cd0-60c3-4a63-8115-d2af2c1cd1a7/main/0.log" Oct 30 05:53:48.140680 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:48.140647 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-cnqvb_cd43eb0e-0dca-47cd-9fda-bb4a0dbce266/main/0.log" Oct 30 05:53:48.515068 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:48.515042 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-9vb6n_da31dd7c-961b-46e8-9f28-1e7e6780fa10/main/0.log" Oct 30 05:53:48.721116 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:48.721090 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-qjd59_e1c66fe9-17ee-4ae2-a1d4-3788a1a4937b/main/0.log" Oct 30 05:53:50.029302 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:50.029257 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf"] Oct 30 05:53:50.043011 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:50.042981 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6qzhf"] Oct 30 05:53:51.579225 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:53:51.579189 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e75e5980-a585-499c-9cde-2c3bf5c8a3d0" path="/var/lib/kubelet/pods/e75e5980-a585-499c-9cde-2c3bf5c8a3d0/volumes" Oct 30 05:53:55.985009 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:53:55.984969 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:54:06.028170 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:54:06.028137 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw"] Oct 30 05:54:06.041526 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:54:06.041496 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vmdzw"] Oct 30 05:54:07.579792 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:54:07.579755 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4045253-968d-46c8-89e1-62bcefb3c222" path="/var/lib/kubelet/pods/b4045253-968d-46c8-89e1-62bcefb3c222/volumes" Oct 30 05:54:20.443188 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:54:20.443160 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vqsmz_a8739f61-fe1e-4c58-b535-4815d55e05f1/payload-processing/0.log" Oct 30 05:54:20.637788 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:54:20.637758 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vqsmz_a8739f61-fe1e-4c58-b535-4815d55e05f1/payload-processing/0.log" Oct 30 05:54:25.985564 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:54:25.985524 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:54:36.459447 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:54:36.459415 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:54:36.460920 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:54:36.460897 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tk9mv_09846780-4fb0-474a-bd15-9b272dc00772/ovn-acl-logging/0.log" Oct 30 05:54:55.984993 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:54:55.984952 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:54:59.026854 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:54:59.026819 6587 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62"] Oct 30 05:54:59.040071 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:54:59.040039 6587 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e021154-4qc62"] Oct 30 05:54:59.579679 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:54:59.579638 6587 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd7220c-7093-4843-b68e-e6c1d75758f4" path="/var/lib/kubelet/pods/1fd7220c-7093-4843-b68e-e6c1d75758f4/volumes" Oct 30 05:55:00.966433 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:00.966405 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vqsmz_a8739f61-fe1e-4c58-b535-4815d55e05f1/payload-processing/0.log" Oct 30 05:55:01.156020 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:01.155988 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vqsmz_a8739f61-fe1e-4c58-b535-4815d55e05f1/payload-processing/0.log" Oct 30 05:55:06.315081 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:06.315049 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hcgqp_0b9fc38f-1ef9-4fbd-bcc2-3172ed84e48c/global-pull-secret-syncer/0.log" Oct 30 05:55:07.259521 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:07.259488 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cqv2d_7115245a-cbc9-4180-9709-2a6e590bb170/konnectivity-agent/0.log" Oct 30 05:55:07.534735 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:07.534654 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-105.ec2.internal_624730a70c2b0e3a128e1c5af9dbe16d/haproxy/0.log" Oct 30 05:55:13.973283 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:13.973245 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59945dd659-9f5g7_a8adbcc0-729e-4774-af4f-43d362118031/authorino/0.log" Oct 30 05:55:14.476572 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:14.476544 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-mswsn_9fed05f3-ad75-4044-ad92-938bcdea9fcc/manager/0.log" Oct 30 05:55:16.191766 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.191730 6587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj"] Oct 30 05:55:16.192247 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.192229 6587 cpu_manager.go:414] "containerMap: removing container" podUID="97c9a279-0047-40ec-8985-f2c5c25bbb22" containerName="maas-api" Oct 30 05:55:16.192296 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.192249 6587 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c9a279-0047-40ec-8985-f2c5c25bbb22" containerName="maas-api" Oct 30 05:55:16.192380 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.192320 6587 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c9a279-0047-40ec-8985-f2c5c25bbb22" containerName="maas-api" Oct 30 05:55:16.195941 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.195921 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.215056 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.215030 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj"] Oct 30 05:55:16.222079 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.222060 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e3a61c90-c5ce-42ad-80ed-f94d548ef369/alertmanager/0.log" Oct 30 05:55:16.227066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.227005 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzzpg\"/\"kube-root-ca.crt\"" Oct 30 05:55:16.227066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.227033 6587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzzpg\"/\"openshift-service-ca.crt\"" Oct 30 05:55:16.227066 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.227054 6587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rzzpg\"/\"default-dockercfg-jv58p\"" Oct 30 05:55:16.256686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.256654 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4b027f9-803a-4318-9ac1-3ba963b658e0-lib-modules\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.256821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.256698 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4b027f9-803a-4318-9ac1-3ba963b658e0-proc\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.256821 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.256720 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4b027f9-803a-4318-9ac1-3ba963b658e0-podres\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.256946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.256868 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6tpv\" (UniqueName: \"kubernetes.io/projected/d4b027f9-803a-4318-9ac1-3ba963b658e0-kube-api-access-h6tpv\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.256946 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.256901 6587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4b027f9-803a-4318-9ac1-3ba963b658e0-sys\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.340502 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.340475 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e3a61c90-c5ce-42ad-80ed-f94d548ef369/config-reloader/0.log" Oct 30 05:55:16.357530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.357504 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4b027f9-803a-4318-9ac1-3ba963b658e0-lib-modules\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.357631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.357543 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4b027f9-803a-4318-9ac1-3ba963b658e0-proc\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.357631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.357561 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4b027f9-803a-4318-9ac1-3ba963b658e0-podres\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.357631 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.357623 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tpv\" (UniqueName: \"kubernetes.io/projected/d4b027f9-803a-4318-9ac1-3ba963b658e0-kube-api-access-h6tpv\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.357764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.357646 6587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4b027f9-803a-4318-9ac1-3ba963b658e0-sys\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.357764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.357643 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4b027f9-803a-4318-9ac1-3ba963b658e0-lib-modules\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.357764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.357643 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4b027f9-803a-4318-9ac1-3ba963b658e0-proc\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.357764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.357680 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4b027f9-803a-4318-9ac1-3ba963b658e0-podres\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.357764 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.357739 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4b027f9-803a-4318-9ac1-3ba963b658e0-sys\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.377719 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.377695 6587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6tpv\" (UniqueName: \"kubernetes.io/projected/d4b027f9-803a-4318-9ac1-3ba963b658e0-kube-api-access-h6tpv\") pod \"perf-node-gather-daemonset-ff7hj\" (UID: \"d4b027f9-803a-4318-9ac1-3ba963b658e0\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.457107 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.457032 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e3a61c90-c5ce-42ad-80ed-f94d548ef369/kube-rbac-proxy-web/0.log" Oct 30 05:55:16.509086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.509053 6587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:16.572265 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.572176 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e3a61c90-c5ce-42ad-80ed-f94d548ef369/kube-rbac-proxy/0.log" Oct 30 05:55:16.687593 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.687512 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e3a61c90-c5ce-42ad-80ed-f94d548ef369/kube-rbac-proxy-metric/0.log" Oct 30 05:55:16.810607 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.810581 6587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj"] Oct 30 05:55:16.811138 ip-10-0-134-105 kubenswrapper[6587]: W1030 05:55:16.811117 6587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd4b027f9_803a_4318_9ac1_3ba963b658e0.slice/crio-f4288ca3ef03377c376969b428169577042cef6e6b5e13627c10db2598ee7f07 WatchSource:0}: Error finding container f4288ca3ef03377c376969b428169577042cef6e6b5e13627c10db2598ee7f07: Status 404 returned error can't find the container with id f4288ca3ef03377c376969b428169577042cef6e6b5e13627c10db2598ee7f07 Oct 30 05:55:16.815445 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.815415 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e3a61c90-c5ce-42ad-80ed-f94d548ef369/prom-label-proxy/0.log" Oct 30 05:55:16.895156 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.895128 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" event={"ID":"d4b027f9-803a-4318-9ac1-3ba963b658e0","Type":"ContainerStarted","Data":"f4288ca3ef03377c376969b428169577042cef6e6b5e13627c10db2598ee7f07"} Oct 30 05:55:16.931841 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:16.931818 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e3a61c90-c5ce-42ad-80ed-f94d548ef369/init-config-reloader/0.log" Oct 30 05:55:17.196266 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:17.196162 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8bgh9_32ed1e66-99e0-4714-92d1-1ef6213fed2c/kube-state-metrics/0.log" Oct 30 05:55:17.307636 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:17.307603 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8bgh9_32ed1e66-99e0-4714-92d1-1ef6213fed2c/kube-rbac-proxy-main/0.log" Oct 30 05:55:17.421777 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:17.421722 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8bgh9_32ed1e66-99e0-4714-92d1-1ef6213fed2c/kube-rbac-proxy-self/0.log" Oct 30 05:55:17.541129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:17.541055 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d6768cb7d-ssqrt_d9fd93f4-68f8-4711-ad54-410d10af6c35/metrics-server/0.log" Oct 30 05:55:17.901530 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:17.901496 6587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" event={"ID":"d4b027f9-803a-4318-9ac1-3ba963b658e0","Type":"ContainerStarted","Data":"35b585ce6afbd5f046895dc22421cb938900eeaaffb078343e4d48920edaf882"} Oct 30 05:55:17.901723 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:17.901631 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:17.943015 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:17.942970 6587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" podStartSLOduration=1.942957522 podStartE2EDuration="1.942957522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:17.942634824 +0000 UTC m=+4242.470401610" watchObservedRunningTime="2026-10-30 05:55:17.942957522 +0000 UTC m=+4242.470724325" Oct 30 05:55:18.506611 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:18.506580 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4r7p_b1e09016-1f64-4c16-be6b-ab60b33a13e8/node-exporter/0.log" Oct 30 05:55:18.619496 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:18.619468 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4r7p_b1e09016-1f64-4c16-be6b-ab60b33a13e8/kube-rbac-proxy/0.log" Oct 30 05:55:18.733051 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:18.733025 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4r7p_b1e09016-1f64-4c16-be6b-ab60b33a13e8/init-textfile/0.log" Oct 30 05:55:18.852238 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:18.852211 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-s8lx6_30011b22-59f1-4361-b961-c5be6de91e88/kube-rbac-proxy-main/0.log" Oct 30 05:55:18.966580 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:18.966550 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-s8lx6_30011b22-59f1-4361-b961-c5be6de91e88/kube-rbac-proxy-self/0.log" Oct 30 05:55:19.081649 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:19.081620 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-s8lx6_30011b22-59f1-4361-b961-c5be6de91e88/openshift-state-metrics/0.log" Oct 30 05:55:19.212452 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:19.212374 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8aeae53-ab5d-4890-9d61-f6402edc264d/prometheus/0.log" Oct 30 05:55:19.327837 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:19.327810 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8aeae53-ab5d-4890-9d61-f6402edc264d/config-reloader/0.log" Oct 30 05:55:19.441881 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:19.441851 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8aeae53-ab5d-4890-9d61-f6402edc264d/thanos-sidecar/0.log" Oct 30 05:55:19.556549 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:19.556522 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8aeae53-ab5d-4890-9d61-f6402edc264d/kube-rbac-proxy-web/0.log" Oct 30 05:55:19.670686 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:19.670663 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8aeae53-ab5d-4890-9d61-f6402edc264d/kube-rbac-proxy/0.log" Oct 30 05:55:19.785008 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:19.784984 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8aeae53-ab5d-4890-9d61-f6402edc264d/kube-rbac-proxy-thanos/0.log" Oct 30 05:55:19.901123 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:19.901050 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8aeae53-ab5d-4890-9d61-f6402edc264d/init-config-reloader/0.log" Oct 30 05:55:20.382175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:20.382147 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cd9d8f774-qwxql_dabdaf17-12d9-48bc-94e3-fac2c28cc3dc/telemeter-client/0.log" Oct 30 05:55:20.495512 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:20.495482 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cd9d8f774-qwxql_dabdaf17-12d9-48bc-94e3-fac2c28cc3dc/reload/0.log" Oct 30 05:55:20.610722 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:20.610695 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cd9d8f774-qwxql_dabdaf17-12d9-48bc-94e3-fac2c28cc3dc/kube-rbac-proxy/0.log" Oct 30 05:55:23.917129 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:23.917103 6587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-ff7hj" Oct 30 05:55:25.984757 ip-10-0-134-105 kubenswrapper[6587]: E1030 05:55:25.984719 6587 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h72cf" containerName="extractor" Oct 30 05:55:28.629318 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:28.629289 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8jf62_c669d496-909b-428e-9b02-1ba9b05fd718/dns/0.log" Oct 30 05:55:28.740089 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:28.740059 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8jf62_c669d496-909b-428e-9b02-1ba9b05fd718/kube-rbac-proxy/0.log" Oct 30 05:55:29.585785 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:29.585759 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xhn9f_59632489-f9ac-44e1-ae2a-7a119c0ece48/dns-node-resolver/0.log" Oct 30 05:55:30.925787 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:30.925743 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dl5rx_9c1d23d7-96c3-4406-9354-860812b8d2bb/node-ca/0.log" Oct 30 05:55:31.613175 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:31.613145 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54668d58ff-2xllz_ddc306dd-f974-4fd8-abb7-067320d7ca83/kube-auth-proxy/0.log" Oct 30 05:55:31.813086 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:31.813036 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-52stt_ae8662ab-ab3e-4271-b8a9-0a0ee4ff6bdd/istio-proxy/0.log" Oct 30 05:55:32.012230 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:32.012121 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-db25s_35dc1c80-11e5-4647-b911-bf84b7e3336f/istio-proxy/0.log" Oct 30 05:55:32.145281 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:32.145242 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-jbc92_b41789fb-0a8e-48dd-8cb0-8b9af7d70ffa/istio-proxy/0.log" Oct 30 05:55:32.372879 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:32.372840 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-p6s8d_d2ed078a-da76-4628-a77f-5b56163461d4/payload-pre-processing/0.log" Oct 30 05:55:32.435122 ip-10-0-134-105 kubenswrapper[6587]: I1030 05:55:32.435086 6587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-vqsmz_a8739f61-fe1e-4c58-b535-4815d55e05f1/payload-processing/0.log"